High CPU analysis in ASP.NET worker process
Server I was asked to troubleshoot had high CPU utilization in w3wp.exe process. There were in fact 2 w3wp.exe processes for 2 application pools existing on system. Below are steps taken to troubleshoot the issue.
1. You need to identify which application pool is causing high utilization, you can do it in a number of ways. I usually launch process explorer (download and put it under /system32 directory on all production machines).
Launch process explorer and find w3wp.exe process which takes high CPU, double click on that process and on “Image” scroll to end of “Command Line” field, application pool name is passed as -ap parameter.
See ScreenShot below.
Identifying application pool will allow first recycle it if need (this will not get rid of high CPU utilization but will allow your website to continue running after you kill this runaway pool).
2. Create FULL memory dump of process in question. If you are on Windows 2008, you can just go to task manager, right click on process and choose “Create Dump File”. This will generate dump file with size of virtual memory for that process.
3. Copy dump file to your workstation where you will perform dump file analysis and launch windbg. Make sure symbols are configured already.
- load sos module for debugging managed code
0:014> .loadby sos mscorwks
- Find which threads are occupying CPU
0:014> !runaway
User Mode Time
Thread Time
14:f8c 0 days 0:25:57.953
17:738 0 days 0:21:42.312
25:1d8 0 days 0:00:03.156
4:1600 0 days 0:00:00.609
3:1294 0 days 0:00:00.421
11:e24 0 days 0:00:00.109
22:c14 0 days 0:00:00.031
19:150c 0 days 0:00:00.031
0:a78 0 days 0:00:00.031
15:123c 0 days 0:00:00.015
- Switch to thread in question and see managed stack
From stack it appears call which started high CPU originated at Company.SiebelExtract.Services.SiebelExtractLookup.lookupAssetAndEntitlementHeirarchyByGrantNumber(System.String)
- Find which page was being executed
0:014> !do 0x0000000002c25400
Name: System.Web.HttpContext
MethodTable: 000007fef06b0f28
EEClass: 000007fef0312398
Size: 336(0x150) bytes
(C:\Windows\assembly\GAC_64\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
MT Field Offset Type VT Attr Value Name
000007fef06b3a50 4000fe6 8 …IHttpAsyncHandler 0 instance 0000000000000000 _asyncAppHandler
000007fef06b3580 4000fe7 10 …b.HttpApplication 0 instance 0000000002c272a0 _appInstance
000007fef06b3ac0 4000fe8 18 ….Web.IHttpHandler 0 instance 0000000002c733f8 _handler
000007fef06b3f00 4000fe9 20 …m.Web.HttpRequest 0 instance 0000000002c25550 _request
000007fef06b4280 4000fea 28 ….Web.HttpResponse 0 instance 0000000002c256a0 _response
000007fef06b4798 4000feb 30 …HttpServerUtility 0 instance 0000000000000000 _server
000007fef8b2fcf8 4000fec 38 …Collections.Stack 0 instance 0000000000000000 _traceContextStack
000007fef06b9ad0 4000fed 40 ….Web.TraceContext 0 instance 0000000000000000 _topTraceContext
000007fef8b3e0b0 4000fee 48 …ections.Hashtable 0 instance 0000000002c397d8 _items
000007fef8b3d640 4000fef 50 …ections.ArrayList 0 instance 0000000000000000 _errors
000007fef8b36a20 4000ff0 58 System.Exception 0 instance 0000000000000000 _tempError
000007fef8b35928 4000ff1 118 System.Boolean 1 instance 0 _errorCleared
000007fef8b516b0 4000ff2 60 …ncipal.IPrincipal 0 instance 0000000002c2bc60 _user
000007fef8b39160 4000ff3 100 System.IntPtr 1 instance 0 _pManagedPrincipal
000007fef06b31f8 4000ff4 68 …ofile.ProfileBase 0 instance 0000000000000000 _Profile
000007fef8b76c90 4000ff5 128 System.DateTime 1 instance 0000000002c25528 _utcTimestamp
000007fef06b8bf0 4000ff6 70 …HttpWorkerRequest 0 instance 0000000002c25080 _wr
000007fef8b32528 4000ff7 130 …Services.GCHandle 1 instance 0000000002c25530 _root
000007fef8b39160 4000ff8 108 System.IntPtr 1 instance 21c18e0 _ctxPtr
000007fef06ab060 4000ff9 78 …m.Web.VirtualPath 0 instance 0000000000000000 _configurationPath
000007fef8b35928 4000ffa 119 System.Boolean 1 instance 0 _skipAuthorization
000007fef8b3f0d8 4000ffb 80 …ation.CultureInfo 0 instance 0000000002a40270 _dynamicCulture
000007fef8b3f0d8 4000ffc 88 …ation.CultureInfo 0 instance 0000000002a40270 _dynamicUICulture
000007fef8b3d9c8 4000ffd 110 System.Int32 1 instance 0 _serverExecuteDepth
000007fef8b2fcf8 4000ffe 90 …Collections.Stack 0 instance 0000000000000000 _handlerStack
000007fef8b35928 4000fff 11a System.Boolean 1 instance 0 _preventPostback
000007fef8b35928 4001000 11b System.Boolean 1 instance 0 _runtimeErrorReported
000007fef8b35928 4001001 11c System.Boolean 1 instance 1 _firstNotificationInitCalled
000007fef8b76c90 4001002 138 System.DateTime 1 instance 0000000002c25538 _timeoutStartTime
000007fef8b35928 4001003 11d System.Boolean 1 instance 1 _timeoutSet
000007fef8b76b90 4001004 140 System.TimeSpan 1 instance 0000000002c25540 _timeout
000007fef8b3d9c8 4001005 114 System.Int32 1 instance 1 _timeoutState
000007fef06c0fa8 4001006 98 …b.Util.DoubleLink 0 instance 0000000002c2bbb8 _timeoutLink
000007fef8b371f8 4001007 a0 ….Threading.Thread 0 instance 0000000002b0ac58 _thread
000007fef06ad480 4001008 a8 …eb.CachedPathData 0 instance 0000000000000000 _configurationPathData
000007fef06ad480 4001009 b0 …eb.CachedPathData 0 instance 0000000002c2b558 _filePathData
000007fef8b36758 400100a b8 System.String 0 instance 0000000000000000 _sqlDependencyCookie
000007fef06bd218 400100b c0 …essionStateModule 0 instance 0000000000000000 _sessionStateModule
000007fef0623cf0 400100c c8 …I.TemplateControl 0 instance 0000000000000000 _templateControl
000007fef0684ad0 400100d d0 …tificationContext 0 instance 0000000002c734f0 _notificationContext
000007fef8b35928 400100e 11e System.Boolean 1 instance 1 _isAppInitialized
000007fef8b35928 400100f 11f System.Boolean 1 instance 1 _isIntegratedPipeline
000007fef8b35928 4001010 120 System.Boolean 1 instance 0 _finishPipelineRequestCalled
000007fef8b35928 4001011 121 System.Boolean 1 instance 0 _impersonationEnabled
000007fef8b35928 4001012 122 System.Boolean 1 instance 0 HideRequestResponse
000007fef8b35928 4001013 123 System.Boolean 1 instance 1 InIndicateCompletion
000007fef06b4d78 4001014 d8 …ion+ThreadContext 0 instance 0000000002c2bac0 IndicateCompletionContext
000007fef06c0e78 4001015 e0 …ronizationContext 0 instance 0000000002c2ba88 _syncContext
000007fef8b35928 4001016 124 System.Boolean 1 instance 0 RequiresSessionState
000007fef8b35928 4001017 125 System.Boolean 1 instance 0 ReadOnlySessionState
000007fef8b35928 4001018 126 System.Boolean 1 instance 0 InAspCompatMode
000007fef06b3ac0 4001019 e8 ….Web.IHttpHandler 0 instance 0000000000000000 _remapHandler
000007fef06b3ac0 400101a f0 ….Web.IHttpHandler 0 instance 0000000000000000 _currentHandler
000007fef8b35928 400101b 127 System.Boolean 1 instance 1 _ProfileDelayLoad
000007fef06b96d8 400101c f8 …ielessHelperClass 0 instance 0000000002c257e0 _CookielessHelper
000007fef8b39218 4000fe5 1a0 …flection.Assembly 0 shared static SystemWebAssembly
Get request object and dump it
0:014> !do 0000000002c25550
Name: System.Web.HttpRequest
MethodTable: 000007fef06b3f00
EEClass: 000007fef0313c90
Size: 336(0x150) bytes
(C:\Windows\assembly\GAC_64\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
MT Field Offset Type VT Attr Value Name
000007fef06b8bf0 400107a 8 …HttpWorkerRequest 0 instance 0000000002c25080 _wr
000007fef06b0f28 400107b 10 …m.Web.HttpContext 0 instance 0000000002c25400 _context
000007fef8b36758 400107c 18 System.String 0 instance 0000000002c27278 _httpMethod
000007fef066d450 400107d 130 System.Int32 1 instance 5 _httpVerb
000007fef8b36758 400107e 20 System.String 0 instance 0000000000000000 _requestType
000007fef06ab060 400107f 28 …m.Web.VirtualPath 0 instance 0000000002c2ee60 _path
000007fef8b36758 4001080 30 System.String 0 instance 0000000000000000 _rewrittenUrl
000007fef8b35928 4001081 138 System.Boolean 1 instance 0 _computePathInfo
000007fef06ab060 4001082 38 …m.Web.VirtualPath 0 instance 0000000002c25808 _filePath
000007fef06ab060 4001083 40 …m.Web.VirtualPath 0 instance 0000000000000000 _currentExecutionFilePath
000007fef06ab060 4001084 48 …m.Web.VirtualPath 0 instance 0000000000000000 _pathInfo
000007fef8b36758 4001085 50 System.String 0 instance 0000000000000000 _queryStringText
000007fef8b35928 4001086 139 System.Boolean 1 instance 0 _queryStringOverriden
000007fef8b3e798 4001087 58 System.Byte[] 0 instance 0000000000000000 _queryStringBytes
000007fef8b36758 4001088 60 System.String 0 instance 0000000002c25170 _pathTranslated
000007fef8b36758 4001089 68 System.String 0 instance 0000000002c391c0 _contentType
000007fef8b3d9c8 400108a 134 System.Int32 1 instance 1394 _contentLength
000007fef8b36758 400108b 70 System.String 0 instance 0000000000000000 _clientTarget
000007fef8b24548 400108c 78 System.Object[] 0 instance 0000000000000000 _acceptTypes
000007fef8b24548 400108d 80 System.Object[] 0 instance 0000000000000000 _userLanguages
000007fef0620148 400108e 88 …owserCapabilities 0 instance 0000000000000000 _browsercaps
000007fef7fd19a8 400108f 90 System.Uri 0 instance 0000000000000000 _url
000007fef7fd19a8 4001090 98 System.Uri 0 instance 0000000000000000 _referrer
000007fef06c2000 4001091 a0 …b.HttpInputStream 0 instance 0000000002c73370 _inputStream
000007fef0671860 4001092 a8 …ClientCertificate 0 instance 0000000000000000 _clientCertificate
000007fef8b25b30 4001093 b0 …l.WindowsIdentity 0 instance 0000000000000000 _logonUserIdentity
000007fef061fb30 4001094 b8 …tpValueCollection 0 instance 0000000000000000 _params
000007fef061fb30 4001095 c0 …tpValueCollection 0 instance 0000000000000000 _queryString
000007fef061fb30 4001096 c8 …tpValueCollection 0 instance 0000000000000000 _form
000007fef06c1048 4001097 d0 …pHeaderCollection 0 instance 0000000002c38d70 _headers
000007fef061fcc0 4001098 d8 …verVarsCollection 0 instance 0000000000000000 _serverVariables
000007fef06be4f8 4001099 e0 …pCookieCollection 0 instance 0000000000000000 _cookies
000007fef0653b10 400109a e8 …ttpFileCollection 0 instance 0000000000000000 _files
000007fef06c15e0 400109b f0 …awUploadedContent 0 instance 0000000002c70d90 _rawContent
000007fef8b35928 400109c 13a System.Boolean 1 instance 1 _readEntityBody
000007fef8b24548 400109d f8 System.Object[] 0 instance 0000000000000000 _multipartContentElements
000007fef8b3e5a8 400109e 100 System.Text.Encoding 0 instance 0000000000000000 _encoding
000007fef0648ae0 400109f 108 …treamFilterSource 0 instance 0000000000000000 _filterSource
000007fef8b39708 40010a0 110 System.IO.Stream 0 instance 0000000000000000 _installedFilter
000007fef061f530 40010a1 140 …SimpleBitVector32 1 instance 0000000002c25690 _flags
000007fef8b36758 40010a4 118 System.String 0 instance 0000000000000000 _AnonymousId
000007fef06ab060 40010a5 120 …m.Web.VirtualPath 0 instance 0000000002c25808 _clientFilePath
000007fef06ab060 40010a6 128 …m.Web.VirtualPath 0 instance 0000000000000000 _clientBaseDir
000007fef8b36048 40010a2 1b0 System.Object 0 shared static s_browserLock
- Now we need to know what is being passed to method in question which end up in high CPU utilization. From output of !clrstack -p you we can see address for that object