Hey everyone!
I'm currently experiencing some odd behavior with an IIS7 (+ASP.NET) setup. For some reason, one or more random (frequent) web requests gets stuck in "ExecuteRequestHandler" for about 6s up to 1m. Afterwards, the request continues as if nothing happened. Content is delivered and no exception occurs.
I'm running:
- IIS 7 (Most configurations are default)
- Windows 2008 R2
- .NET 4.5
- Custom Web Application running one regular ASP.NET web site and once WCF back-end.
Until now I have tried:
- Going through the Event Viewer but nothing showed up.
- Shutting down the antivirus solution.
- Disabled all performance counters I could disable.
- Reinstalled IIS, .NET and WCF.
- Reassured that the code wasn't faulty. In addition, this issue has only occurred on 2 out of ~20 server installations.
- Monitoring the w3wp.exe processes using Process Explorer and Resource Monitor.
- I noticed here that when I popped Analyze Wait Chain during a hang, it showed one process which was itself and it stated "One or more threads of w3wp.exe are waiting to finish network I/O" as seen in Evidence 1. When I took a look in Process Explorer for the call Stack it showed a bunch of cryptic information. You can see this below as Evidence 2.
- I activated fail tracing in IIS and that resulted in seeing that the execution took the amount of time I described in the symptoms but when I look at the time stamps in the logs, I can't see any entry that exceeded 0ms.
- I have disabled drive caching, drive indexing and turned off CPU Parking in pure desperation.
- I have even attempted to use Diagnostics Tool 2 but once it completed the analysis, it crashed for me and that happened every time it completed. So no help from that tool.
- I monitored the Requests live as well from within an IIS worker process and found them listed there for 6s to about 1min peaked then released and successfully completed.
I have searched everywhere for an answer but I can't seem to find anything on the topic at hand. I've gone through alot of performance troubleshooting guides and I'm well experienced in IIS and .NET. Should you have any more questions regarding the setup, I'll be glad to share it so that you can hopefully assist me. So without further ado...
Evidence 1
This shows an image from within Resource Monitor showing a wait is ongoing as supported by the evidences below.
http://i.imgur.com/kbQ2yWu.jpg
Evidence 2
This is a direct pull-out from the Stack of a Thread that froze the current request. I pulled it from Process Explorer.
ntoskrnl.exe!KeWaitForMultipleObjects+0xc0a
ntoskrnl.exe!KeAcquireSpinLockAtDpcLevel+0x732
ntoskrnl.exe!KeWaitForMutexObject+0x19f
ntoskrnl.exe!PoStartNextPowerIrp+0xba4
ntoskrnl.exe!PoStartNextPowerIrp+0x1821
ntoskrnl.exe!KeAcquireSpinLockAtDpcLevel+0x93d
ntoskrnl.exe!KeWaitForMutexObject+0x19f
ntoskrnl.exe!NtWaitForSingleObject+0xde
ntoskrnl.exe!KeSynchronizeExecution+0x3a23
wow64cpu.dll!TurboDispatchJumpAddressEnd+0x6c0
wow64cpu.dll!TurboDispatchJumpAddressEnd+0x4a8
wow64.dll!Wow64SystemServiceEx+0x1ce
wow64.dll!Wow64LdrpInitialize+0x42a
ntdll.dll!RtlIsDosDeviceName_U+0x23a27
ntdll.dll!LdrInitializeThunk+0xe
ntdll.dll!ZwWaitForSingleObject+0x15
mswsock.dll+0x76b6
WS2_32.dll!recv+0x79
System.ni.dll+0x20b087
System.ni.dll+0x1e313d
System.ni.dll+0x1e305d
System.ni.dll+0x1e2f83
System.ni.dll+0x1e2ef4
System.ni.dll+0x1e2a3c
System.ni.dll+0x1bb559
System.ni.dll+0x80c92e
System.ni.dll+0x1e23df
System.ni.dll+0x606f1c
System.ni.dll+0x6007ae
System.ni.dll+0x602eeb
System.ni.dll+0x605f80
System.ni.dll+0x606073
System.Web.ni.dll+0x1f9716
System.Web.ni.dll+0x1d4986
System.Web.ni.dll+0x1e78d7
System.Web.ni.dll+0x1d4c40
System.Web.ni.dll+0x1d140b
System.Web.ni.dll+0x1d6d27
System.Web.ni.dll+0x1d6a1f
webengine4.dll!MgdGetCurrentNotificationInfo+0x89
webengine4.dll!MgdGetCurrentNotificationInfo+0x17b
webengine4.dll!MgdGetCurrentNotificationInfo+0x1ed
webengine4.dll!MgdGetPrincipal+0x199
clr.dll+0x491f
clr.dll+0x494d
clr.dll!LogHelp_TerminateOnAssert+0x15b57
System.Web.ni.dll+0x1ac723
webengine4.dll!MgdGetNextNotification+0x68
System.Web.ni.dll+0x1d18a9
System.Web.ni.dll+0x1ebefc
System.Web.ni.dll+0x23b099
webengine4.dll!MgdCanDisposeManagedContext+0x38
webengine4.dll!MgdCanDisposeManagedContext+0x1d
System.Web.ni.dll+0x23afc5
webengine4.dll!MgdGetResponseChunks+0xc4
webengine4.dll!MgdIndicateCompletion+0x22
System.Web.ni.dll+0x23b10d
System.Web.ni.dll+0x1d6ea6
System.Web.ni.dll+0x1d6a1f
clr.dll+0x3f95
clr.dll!DllGetActivationFactoryImpl+0x10a0b
clr.dll!GetPrivateContextsPerfCounters+0x1f734
clr.dll!DllGetActivationFactoryImpl+0x10979
webengine4.dll!MgdGetCurrentNotificationInfo+0x89
webengine4.dll!PerfDecrementGlobalCounter+0x47
clr.dll!GetPrivateContextsPerfCounters+0x1f285
clr.dll!GetPrivateContextsPerfCounters+0x22d9e
clr.dll!GetPrivateContextsPerfCounters+0x5965
kernel32.dll!BaseThreadInitThunk+0x12
ntdll.dll!RtlInitializeExceptionChain+0x63
ntdll.dll!RtlInitializeExceptionChain+0x36
Evidence 3
This is a sample of a request that froze.
http://1drv.ms/1I8SfAp