Hi All,

I am running a web farm of IIS7 on Server 2008 Web edition. We max out at 150 hits/sec at peak times load balanced across 4 servers.

One of our customers is complaining they are getting time outs and slow responses to one of our web services so I have been investigating.

I started by looking through the logs and found that most requests are serviced in sub 120ms but some hang on for seconds!

I followed up by running my own load tests from an external linux VPS using httping. This confirmed the behavior our client and the logs were showing.

To track down the issue I used powershell to write a quick test script * so I could test behind the firewall; all of the following figures are based upon script looping through 10,000 requests. I got the same behavior but less pronounced (some requests taking 1000+ms with an average of 18ms).

I thought maybe it was the Load balancer, so I ran the script on the web servers (one by one); Same behavior, less pronounced again ( long requests taking 25+ms avg 1ms) .

I reduced complexity and ran it against localhost with the default website; same behavior same times as above .

I took it to the test lab, clean (ish) install of 2k8 default website in IIS7; same results.

I am now perplexed? I have looked online for idea's on why I am getting seemingly random high latency spikes. I am sure that other things in my network such as Load balancers ect are amplifying the issue externally but to begin with I aim to get my max requests in my lab to a same order time as my average.

*SCRIPT WARNING

$client = new-object System.Net.WebClient
$slow=0
for($i=1; $i -le 10000; $i++)
{
$time = (measure-command { $client.DownloadFile( "http://localhost/", "C:\data\index.tmp")}).TotalMilliseconds
 if($time -gt 10)
 {
  echo "Time over 10 ms: " $time
 }
}

/SCRIPT

Obviously if you Tee-Object -file the output of the measure-command to a text file you can put it in a spreadsheet and Analise .

I've been getting a quick break down out of Excel that looks like this.

Any idea's on things I could monitor, tune, ect?

Regards

*PS I vote for an upgrade of the WYSIWYG editior so that it works in Opera.

Hi,

we are having a bit of a problem with content delivery from the server (IIS7 on Windows Server 2008) that's running a couple PHP web sites. Quite often, small batches of requests end up taking a long time to complete for no apparent reason. This mostly happens with static content, but some php page requests are also affected similarly. They show up in the request queue as being stuck in "SendResponse" stage:

REQUEST "f70000438000116a" (url:GET /gfx/public/upload/45849.JPG, time:11934 msec, client:87.59.109.41, stage:SendResponse, module:IIS Web Core)
REQUEST "fb00000180002198" (url:GET /gfx/public/upload/45844.JPG, time:11825 msec, client:87.59.109.41, stage:SendResponse, module:IIS Web Core)
REQUEST "5b0000428000080a" (url:GET /gfx/public/upload/45845.JPG, time:11809 msec, client:87.59.109.41, stage:SendResponse, module:IIS Web Core)
REQUEST "d100004280000e40" (url:GET /gfx/public/upload/47222.JPG, time:56722 msec, client:110.36.20.149, stage:SendResponse, module:IIS Web Core)

Some of such stuck requests end up in the logs with "sc-win32-status" codes of 64/121/1236, which would seem to suggest the requests hitting some sort of time limit before being finally killed.

I also tried gathering some failed request logs, but all request steps there are shown as taking 0ms, with the last one being "GENERAL_RESPONSE_ENTITY_FILE" or "GENERAL_RESPONSE_ENTITY_BUFFER". Is this normal, or should all requests end up with the "GENERAL_REQUEST_END" step instead?

Anyway, this is all rather puzzling to me and I can't spot any clues that would let me move forward. Is there perhaps a way to look deeper into what's happening in the "SendResponse" stage? I.e.: whether data is being sent at all, if the connection is stuck waiting for packets from server/client side or something else entirely?

I have an mvc4 web api that services on average (W3SVC_W3WP) 50req/sec (per each server) on 4 physical servers (xeon 2.5Ghz 6 core 12 logical processors with 64G memory running win2008R2 64bit).

Recently we started to get flooded with new request due with poor capacity planning and we are seeing lots of queuing (asp.net v4.0.30319 Requests Queued = 2k-5k) and total req/sec up to about 200 (avg before was 50 that we could handle).

What can I do do help scale my application on each of the four servers?

Here is a link to some perf counters -> http://bit.ly/WHFNv0

**Given**:

- IIs 7.5 - 2 web sites (both expose web rest api) running on one 64bit integrated app pool
- cpu per server ~45-55%
- One worker process (w3wp.exe)
- Queue Length = 1000

**MachineConfig:**

<system.web>
<processModel autoConfig="false" maxWorkerThreads="4095" maxIoThreads="4095" minIoThreads="2047"
requestQueueLimit="5000" responseDeadlockInterval="00:03:00" />
<httpRuntime minFreeThreads="704" minLocalRequestFreeThreads="608" />
</system.web>

Setting max worker processes in IIS Advanced Pool Settings from 1 -> 2 didn't seem like it helped much.

Environment: windows server 2008 R2 64bit, IIS7.5, ASP.NET 4.0 WebSite.

Problem Description:

When my websit's current connections is below 500 ( from the performance monitor in Advanced settings), everything is well.

When it reaches 600 and higher , I request any aspx page even an empty page without any logic or Database connectionm, it doesn't response, the process bar goes very slow, after 1-2 minute is finally comes out. I tried on several machines and all the same result.  It looks very likely that my request is being queued because there are too many concurrent connections on IIS. 

But if I request a static file like htm or jpg , it comes out very quickly without any delay. So I think only .aspx files are blocked .

I checked my IIS settings, all default settings, I didn't change anything.

So can anyone put some light on this problem? Do I need any special settings for such a level of concurrency visits?

Last week we updated our (cluster) hosting environment to Windows Server 2012 and IIS8. Since the update we experiencing some ‘server to busy’ warnings on aspx sites (asp.net).

We have multiple load balances (hardware) webservers with 24GB memory and XEON processors each. De servers are not (physical) busy at all during the warnings... All the servers in our cluster are not using more then 10GB physical memory.

Here’s what we figured out so far. When we set the “Virtual Memory Limit” to zero in the “Application Pool Defaults”, the servers will work for a couple of minutes. But after some minutes, suddenly all (or some) asp.net websites are giving a “server to busy” warning. When we refresh the sites, sometimes they work and sometimes they don’t.

We are now using  the following “Application Pool Defaults” to let the asp.net sites work. However, some asp.net sites are not working properly with these settings at all.

Private Memory Limit: 100mb
Virtual Memory Limit: 100mb

Can someone tell us what we doing wrong? Why are the asp.net sites giving a “server to busy” warning while the servers aren’t busy at all?

Thanks in advance, if you need any more information about settings etc please let us know.

We're running IIS7 on WS08.  I have experienced SEVERAL times when I'm accessing a page on this server where it will simply stop responding.  The load progress bar will show no progress at all, and after a minute or two, I finally stop it.  I can either refresh the browser (which usually immediately brings the page up), or go directly to the page or a different page on the same web site.  I have seen this happen on standard html pages as well as asp and asp.net (2.x) pages.  It happens most of the time on pages that I've just changed, but I've seen it happen on pages that I have not changed in quite a while.

I have checked the event log when this happens, and there are no entries in there indicating any errors of any kind.

I usually access this server across the network, so instead of a standard "www.mysite.com", I usually do a "http://myserver/myfolder/mypage.htm" sort of thing.  But, I believe I've seen this problem when accessing it both ways.

We have some of the same apps running on a WS03 server perfectly.

1) Any idea what is causing this?

2) If there are no ideas,  are there any testing methods anyone can suggest to help me figure this out?

Thanks,
Jesse

When I'm using web garden to start 5 w3wp processes, I found that one of the processs always takes a lot of memory and reachs about 2G, but the others 4 remain 800m or so, seems the load is mainly taken by 1 process only and others server very little.

Is this a by-defalut design? Is there a way to make the visits shared evenly by all processes?

 Hello,

We have a strange problem on our Windows 2008 R2, IIS 7.5 webserver. Recently we've started on updating some of our website's to .NET 4.0, but we noticed that some of them load very very slow at first requests after the recycling of the application pool, it takes 1-2 minutes for the page to load if that happens. Any requests after that are normal.

After some testing and researching, we noticed that not all website's have that problem. First of all, none of our old .NET 1.1 website's have the problem (which run on the same server). At first i thought that this problem existed with all .NET 4.0 website's we have, but that is not the case. Today i made a simple ASP.NET project with nothing extra in it, just the sample Web Forms project of Visual Studio. When i uploaded it to our webserver, it didn't have that problem. So i thought that the problem was in our code, so i started adding things to the test website, like data retrieval from a database, etc. But the website kept being fast, even at first requests.

After a while i deleted all files of the test website and copied all files of a website which has the slow performance problem, into the folder where the test website was. I didn't change any settings in IIS of the test website. The copied website was fast at first requests, just like the test website. So i started thinking that it had to be an application pool problem and i compared the settings of the 2 application pools. I noticed that the settings were identical.

Then i changed the application pool of the fast test website, to that of the slow website and tried again. The test website then also was as slow as the other website, at first request. Then i changed it back to it's own application pool and it was fast again. After that, i changed the application pool of the slow website, to that of the fast test website. Now the slow website was fast again.

So my question is, how can this happen? How can one application pool become slower than the other application pool with identical settings, even after restarting both of them and is there a way to fix/prevent it? Once an application pool becomes slow, it does not become fast again, even with stopping and starting it.

I made sure to recycle the application pools before every test i did, so that all tests were 'first requests'.

Hello,

can anybody please point me to some usefull documentation or give me detailed explanation ?

Situation:

IIS receives request to ashx handler (querystring based) that is setup to have output cache (frequentHitTimePeriod=10sec., frequentHitThreshold=1).

request usually takes some 5-15sec to respond back (it is maptile 256x256px generated from huge database)

after 1sec, another request (one or more) hits the server (exactly same query string, with same result map).

At this moment, original (first) request is still in progress.....

Will IIS output cache engine recognize this situation properly and keep wait these 2 extra requests until first one finish.

AND THEN:

if first request is http:200 puts result on cache and returns these 2 extra request from cache (no ashx call)

if first request fails, sends first response to client (with some errcode) and retry with second request (call ashx again), keep wait third

etc. until no more identical req.

thnx

A.

We have configured a 32 bit application pool running on IIS 7.5 64 bit.   We have a 3rd party ISAPI filter and there is only 64 bit version available.   Is there anyway in which we can have the filter work with the 32 bit application pool?

-Javed 

Guys,

Can anyone think of a product that would allow me to publish performance counters for running web services?

This is what my boss envisions:

1. Deploy the product to the web servers
2. Enable the product by modding the web.config files for the target web services
3. Specify which  SOAP methods or WCF operations will get monitored
4. Rinse and repeat

The gyst is, we want to inject the logging without having to explicitly instrument the apps.

Does that make sense to you guys? Is there a product out there that we can use? Should I have drank the BLUE pill?

Thanks in advance for any help you can provide.

A<M

Hi,

I have an ASP.NET 4.5 (Web API) web application hosted in IIS on Windows Server 2008 R2. This web application is hosted in it's own application pool using the Integrated pipeline.

I am trying to use Performance Monitor to capture the ammount of memory and IO pressure of the application.

eg:

• How much memory (managed or otherwise) is the application consuming
• How much IO work (request data in / response data out - not writing to disk) is the application performing at any given second.

I with to capture these metrics so that I can gauge performance benefits of various enhancements we have to our system.

I am not interested in requests per second style load testing - this will be handled separately.

Problem: I am struggling to identify the correct performance counters to use to capture the above metrics.

There are many different counters that relate to memory usage and all of the ones I have tried do not seem change... even when I upload a 1GB file to my application.

For IO pressure I am using IO Read / Write (sec) counters under the Process object... but if I'm honest I'm only guessing.

Can anyone point me in the right direction?

Hi experts,

Is there any way to recycle the specific Application Pools in my IIS? I'm using ASP.NET and I want to recycle the pool when it crush. Thus it possible?

Regards,

We are experiencing some problems in production (IIS 7.5 integrated managed pipeline mode and .NET 4.5 application) when to may of our incoming IIS requests trigger asynchronous web service requests. There is a quite nice MSDN blog post here by Thomas Marquardt that gives a very good picture of the potential problems that can arise. I believe our problem is due to the ServicePointManager.DefaultConnectionLimit low default value and we are about to deploy this fix (requires code change as can be read about in the blog.

However, I have quite a few remaining questions from this blog post. I have posted these questions yesterday in the blog post (quite miserably since there was a cap on the word lenght...), but since I am guessing it may take some time before Thomas may response, I repost the questions here as well in hope that someone in the community knows some of the answers. You may need to read the blog post first to be in tune with what I write below, but I can promise it is good reading.  If someone knows one or several of the answers to the questions, please just refer to the what question it is in the answer (eg 1.c)

Here is my comment(s) in the blog post with nice WYSIWYG markup:

Thanks [Thomas Marquardt] for some great explanations and for taking the time to answer all these questions. It really clarifies things. In our case, we have a problem in production with an .NET 4.5 / IIS 7.5 web server running integrated [managed pipeline] mode when we get a high load of requests that utilises a high degree of outgoingasynchronous web service requests (stack trace reveals HttpWebRequest usage beneath the hood of the HttpClient class usage). Based on my reading here, it seems the solution is to set theServicePointManager.DefaultConnectionLimit programmatically in production code as you suggest (since we use autoConfig set to true). However before I try this in our production environment (initiating a redeploy) I have some remaining categorised questions with regards to what I’ve read (sublabeled (a), (b) etc to make your answering easier):

1. appRequestQueueLimit config:
   (a) Do I understand it correctly that the httpRuntime setting appRequestQueueLimit configuration only is applicable for IIS 6 and 7 when running in classic mode? I.e it sets the limit for the old application scoped queue (with “miserable performance” as you write)?
2. (questions a-f below) 
   When our servers gets over loaded and don’t scale (CPU only at 20%), we get 503 error code in response to the original request + that the async. web service requests ends up with quite a few exceptions: “SocketException: An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full”. A problem with this type of error message hint is of course that you are not sure which queue may be full (if it is the queue). 
   
   (a) So how many queues exists on the server in the (total) request pipeline? From you description, in our scope, there is both a (1) process-wide native queue as well as the (2) CLR ThreadPool that naturally can be regarded as a queue. But you also mention the HTTP.sys kernal queue. 
   (b) Is the HTTP.sys kernal queue the same queue as (1) process-wide queue? 
   (c) If not, could you perhaps explain breifly what the purpose of this HTTP.sys queue is? 
   (d) Are there any other lower level queues that can become problematic? (eg network card driver) 
   
   However, in our case, if the problem is actually due to connectionManagement/MaxConnections, I guess the web service error message is quite logical. If there is no possibility for the web service to get access to the connection (keep-alive), the number of requests waiting to execute in the ThreadPool should increase of course until the maxium number of threads is hit. A poor solution would be to increase the number of maxThreads in the ThreadPool... 
   (e) Is there a performance monitor for showing the current number of threads in the ThreadPool? 
   
   Also, there seems to be a performance counter called [Web Service.Current Connections] that should really hit a max pegged value (12*core count) if this is the problem. 
   (f) Is this correct (below the (e) question)?
    
3. From you question response 8 Dec 2011, I understand it that “Requests Queued” is use both to measure the number of requests that are waiting as well as the number of executing requests in the ThreadPool + those requests that possible (unlikely if correctly configured) hit the native queue. It seems to me then, that it is quite difficult to distinguish the scenario when there actually are requests in the native queue. 
   (a) Would that be possible in .NET 4.5 / IIS 7 integrated mode? (how if so? :)
   
4. You mentioned in comment 22 april, 2010 and 28 oct, 2010 that if you run out of (ephemeral) ports it can be good to decrease the TcpTimedWaitDelay registry setting (as well as MaxUserPort). For us I don’t think this is a problem since we use TCP keep-alive (persisted) connection for the web service requests. 
   (a) However, in order to find out if this was a problem, would it not be enough to check the performance counter [TCPv4.Connection Established] and see if this value is close to the maximum amount of available ports (which should be quite high on Windows 2003/2008 by default). 
   
   Further more, I think decreasing TcpTimedWaitDelay is a good option although perhaps one need to consider other types of TCP reliant I/O high latency requests running on the server (eg web services and even database calls) if lowering this number. 
   (b) Do you agree?

 I can of course follow up on this thread with answers from our internal investigation to my questions. 

I have enbaled kernel cache..

I am using "CacheForTimePeriod" policy..

response from .aspx page is cached..

But how do I access that cached data programmatically..???

e.g- Some records from table is cached. for next request instead of querying database I want to search cached data in ...

I have an ASP.Net 2.0 Website running in a production environment on IIS 7.5.

We have recently started getting serious performance issues with the site. I have elminated most causes i.e. network and code and are left in a strange situation.

If I run the website on the Virtual Server (Windows 2008 R2) under which IIS is running the website responds resonable quickly 30 seconds to complete a complicated operation. If I run the website from the Virtual Machine housing the Database that is on the same physical box the operation takes 2 minutes rather than 30 seconds?

I have used some logging software (fiddler) and this shows me that when on the same box as IIS the IIS server receives the request, waits 30 seconds then sends the response. It also shows that when on the DB Box the IIS server receives the request, waits 2 minutes and sends the response.

I have a feeling this may have something to do with authentication or security of somekind as the delay only ever happens when accessing the site from outside of the IIS box? Can anybody help me narrow down how and why this might be happening?

Hello, I have a problem with the IIS 7.5 , and I don't know how to resolve this problem.

When I open the IIS it's normal, but when I click in HOST ( name of machine) in IIS , There is a big delay for show the sites, the delay takes a long time, I need help for this, please i need a solution, because the delay it's soo long, if need more information please call me, Tks.. Regards  

Here's my environment:
IIS7 on Win 7, .NET 4, App Pool Integrated

web.config

<?xml version="1.0" encoding="UTF-8"?><configuration></configuration>

Test.aspx

<%@ Page Language="C#" %><!DOCTYPE html><script runat="server">
        protected void OnAction(object sender, EventArgs e)
        {
            int count;
            status.Text = (int.TryParse(status.Text, out count) ? count + 1 : 0).ToString();
            Session["test"] = count;
        }</script><html xmlns="http://www.w3.org/1999/xhtml"><head runat="server"><title>IIS Session Hang Test</title><script>
            var mutiPostback = function () {
                var e = document.getElementById('LinkButton1');
                e.click();
                e.click();
            };</script></head><body><form id="Form1" method="post" runat="server"><asp:ScriptManager runat="server" ID="SM"></asp:ScriptManager><asp:UpdatePanel ID="UpdatePanel1" runat="server"><Triggers><asp:AsyncPostBackTrigger ControlID="LinkButton1"/></Triggers><ContentTemplate><asp:Label runat="server" ID="status" /></ContentTemplate></asp:UpdatePanel><input type="button" id="button1" onclick="mutiPostback();" value="MultiPostback"/><div style="display: none"><asp:Button ID="LinkButton1" runat="server" OnClick="OnAction" Text="Click" /></div></form></body></html>

Yes, the multiple postback is intentional, we notice this behavoir will cause many request stuck in RequestAcquireState and ultimately prevent any new requests being accepted by the server. However, this problem is only observable under IE (IE6 ~ 9) and not on Chrome or FF.

Here's a screen shot of how the hanged request look like in request list for worker process.

Now we have found a few ways to get around this problem, but none are acceptable in our situation:

1. Remove/Comment out Session usage.
2. Change app pool to Classic mode.

NOTE: we also found that even if we don't directly use Session as shown in the example, the problem still occurs. IE: if we add a Global.asax.cs and add an empty Session_Start event handler, the request will still hang in RequestAcquireState.

Does anyone have a clue why this is happening and how can we resolve this issue that only seem to happen in Integrated managed pipeline mode?

We have a IIS 7.5 (.net 4.0) web service that is running slowly.  I ran wireshark and found that it is making at least 6 DNS calls looking for wpad.<domain>.net.  Each one times out as we don't use WPAD.  Each WPAD call is taking about 3/4 of a second before timing out, so the total delay is 4.5 seconds.  After the 6 requests finally timeout, it tries DNS with the url it is trying to resolve and that works.  So the service finally continues on.

Platform is windows server 2008 r2.   I want to completely disable WPAD. We don't use a proxy and as far as I know we don't need WPAD. We don't use DHCP.  I want to stop WPAD but I haven't been successful. I have tried the following:

1.  disable "automatically detect settings" in IE      2.  disable "use automatic configuration script" in IE     3.  Checked that WinHTTP Web Proxy Auto-Discovery Service is not running automatically, it is set to run manual so I think that should be ok.     4.   Executed "Netsh winhttp show proxy" which tells me Direct access (no proxy server).

But it still tries to use WPAD. 

I just found this, wondering if anyone has used <defaultProxy> or <proxy> settings in web.config to turn off the proxy?  Wondering if that is what I need to do?

Hi,

we are having a bit of a problem with content delivery from the server (IIS7 on Windows Server 2008) that's running a couple PHP web sites. Quite often, small batches of requests end up taking a long time to complete for no apparent reason. This mostly happens with static content, but some php page requests are also affected similarly. They show up in the request queue as being stuck in "SendResponse" stage:

REQUEST "f70000438000116a" (url:GET /gfx/public/upload/45849.JPG, time:11934 msec, client:87.59.109.41, stage:SendResponse, module:IIS Web Core)
REQUEST "fb00000180002198" (url:GET /gfx/public/upload/45844.JPG, time:11825 msec, client:87.59.109.41, stage:SendResponse, module:IIS Web Core)
REQUEST "5b0000428000080a" (url:GET /gfx/public/upload/45845.JPG, time:11809 msec, client:87.59.109.41, stage:SendResponse, module:IIS Web Core)
REQUEST "d100004280000e40" (url:GET /gfx/public/upload/47222.JPG, time:56722 msec, client:110.36.20.149, stage:SendResponse, module:IIS Web Core)

Some of such stuck requests end up in the logs with "sc-win32-status" codes of 64/121/1236, which would seem to suggest the requests hitting some sort of time limit before being finally killed.

I also tried gathering some failed request logs, but all request steps there are shown as taking 0ms, with the last one being "GENERAL_RESPONSE_ENTITY_FILE" or "GENERAL_RESPONSE_ENTITY_BUFFER". Is this normal, or should all requests end up with the "GENERAL_REQUEST_END" step instead?

Anyway, this is all rather puzzling to me and I can't spot any clues that would let me move forward. Is there perhaps a way to look deeper into what's happening in the "SendResponse" stage? I.e.: whether data is being sent at all, if the connection is stuck waiting for packets from server/client side or something else entirely?