Troubleshooting high CPU utilization in ASP.NET worker process

High CPU analysis in ASP.NET worker process

Server I was asked to troubleshoot had high CPU utilization in w3wp.exe process. There were in fact 2 w3wp.exe processes for 2 application pools existing on system. Below are steps taken to troubleshoot the issue.

1. You need to identify which application pool is causing high utilization, you can do it in a number of ways. I usually launch process explorer (download and put it under /system32 directory on all production machines).
Launch process explorer and find w3wp.exe process which takes high CPU, double click on that process and on “Image” scroll to end of “Command Line” field, application pool name is passed as -ap parameter.
See ScreenShot below.

Identifying application pool will allow first recycle it if need (this will not get rid of high CPU utilization but will allow your website to continue running after you kill this runaway pool).

2. Create FULL memory dump of process in question. If you are on Windows 2008, you can just go to task manager, right click on process and choose “Create Dump File”. This will generate dump file with size of virtual memory for that process.

3. Copy dump file to your workstation where you will perform dump file analysis and launch windbg. Make sure symbols are configured already.

• load sos module for debugging managed code

0:014> .loadby sos mscorwks

• Find which threads are occupying CPU

0:014> !runaway
 User Mode Time
  Thread       Time
  14:f8c       0 days 0:25:57.953
  17:738       0 days 0:21:42.312
  25:1d8       0 days 0:00:03.156
   4:1600      0 days 0:00:00.609
   3:1294      0 days 0:00:00.421
  11:e24       0 days 0:00:00.109
  22:c14       0 days 0:00:00.031
  19:150c      0 days 0:00:00.031
   0:a78       0 days 0:00:00.031
  15:123c      0 days 0:00:00.015

We can see threads 14 and 17 are monopolizing CPU

• Switch to thread in question and see managed stack

0:014> ~14s

ntdll!ZwGetContextThread+0xa:

00000000`77c6783a c3              ret

0:014> !clrstack

OS Thread Id: 0xf8c (14)

Child-SP         RetAddr          Call Site

000000001c51d5b0 000007fef764753d Company.SiebelExtract.Services.BusinessRules.ServiceLookups.b__a(System.Data.DataRow, System.Data.DataRow)

000000001c51d5f0 000007fef7645f73 System.Linq.Enumerable+d__31`3[[System.__Canon, mscorlib],[System.__Canon, mscorlib],[System.__Canon, mscorlib]].MoveNext()

000000001c51d660 000007fef764a023 System.Linq.Enumerable+WhereSelectEnumerableIterator`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].MoveNext()

000000001c51d6b0 000007ff001f40e7 System.Linq.Enumerable+d__7a`1[[System.__Canon, mscorlib]].MoveNext()

000000001c51d720 000007ff001ee868 Company.SiebelExtract.Services.BusinessRules.ServiceLookups.getAccountsList(System.String)

000000001c51d960 000007fefa7913b2 Company.SiebelExtract.Services.SiebelExtractLookup.lookupAssetAndEntitlementHeirarchyByGrantNumber(System.String)

000000001c51e310 000007fef8a69b46 System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo, Boolean)

000000001c51e4b0 000007fef2f44197 System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)

000000001c51e500 000007fef2fc3c5f System.Web.Services.Protocols.LogicalMethodInfo.Invoke(System.Object, System.Object[])

000000001c51e580 000007fef2fc412b System.Web.Services.Protocols.WebServiceHandler.Invoke()

000000001c51e610 000007fef2fc4c90 System.Web.Services.Protocols.WebServiceHandler.CoreProcessRequest()

000000001c51e680 000007fef35cf047 System.Web.Services.Protocols.SyncSessionlessHandler.ProcessRequest(System.Web.HttpContext)

000000001c51e6d0 000007fef0560347 System.Web.Script.Services.ScriptHandlerFactory+HandlerWrapper.ProcessRequest(System.Web.HttpContext)

000000001c51e700 000007fef05245cb System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()

000000001c51e7b0 000007fef0c0caa1 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)

000000001c51e850 000007fef0bfde22 System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)

000000001c51e9e0 000007fef0bdf8f9 System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)

000000001c51ea30 000007fef0d06904 System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)

000000001c51eb50 000007fef0d05abb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)

000000001c51ecc0 000007fef0d05874 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)

000000001c51ed20 000007fefa78ef2a DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)

000000001c51f550 000007fef0d069e4 DomainNeutralILStubClass.IL_STUB(IntPtr, System.Web.RequestNotificationStatus ByRef)

000000001c51f630 000007fef0d05abb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)

000000001c51f7a0 000007fef0d05874 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)

000000001c51f800 000007fefa78f17b DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)


From stack it appears call which started high CPU originated at Company.SiebelExtract.Services.SiebelExtractLookup.lookupAssetAndEntitlementHeirarchyByGrantNumber(System.String)

• Find which page was being executed

0:014> !clrstack -p

OS Thread Id: 0xf8c (14)

Child-SP         RetAddr          Call Site

000000001c51d5b0 000007fef764753d McAfee.SiebelExtract.Services.BusinessRules.ServiceLookups.b__a(System.Data.DataRow, System.Data.DataRow)

    PARAMETERS:

        row =

        rowLink =

000000001c51d5f0 000007fef7645f73 System.Linq.Enumerable+d__31`3[[System.__Canon, mscorlib],[System.__Canon, mscorlib],[System.__Canon, mscorlib]].MoveNext()

    PARAMETERS:

        this = 0x0000000004857198

000000001c51d660 000007fef764a023 System.Linq.Enumerable+WhereSelectEnumerableIterator`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].MoveNext()

    PARAMETERS:

        this =

000000001c51d6b0 000007ff001f40e7 System.Linq.Enumerable+d__7a`1[[System.__Canon, mscorlib]].MoveNext()

    PARAMETERS:

        this = 0x0000000004857288

000000001c51d720 000007ff001ee868 McAfee.SiebelExtract.Services.BusinessRules.ServiceLookups.getAccountsList(System.String)

    PARAMETERS:

        GrantNumber =

000000001c51d960 000007fefa7913b2 McAfee.SiebelExtract.Services.SiebelExtractLookup.lookupAssetAndEntitlementHeirarchyByGrantNumber(System.String)

    PARAMETERS:

        this =

        GrantNumber = 0x0000000002c75a90

000000001c51e310 000007fef8a69b46 System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo, Boolean)

    PARAMETERS:

        this =

        obj =

        invokeAttr =

        binder =

        parameters =

        culture =

        skipVisibilityChecks =

000000001c51e4b0 000007fef2f44197 System.Reflection.RuntimeMethodInfo.Invoke(System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, System.Object[], System.Globalization.CultureInfo)

    PARAMETERS:

        this =

        obj =

        invokeAttr =

        binder =

        parameters =

        culture =

000000001c51e500 000007fef2fc3c5f System.Web.Services.Protocols.LogicalMethodInfo.Invoke(System.Object, System.Object[])

    PARAMETERS:

        this =

        target =

        values =

000000001c51e580 000007fef2fc412b System.Web.Services.Protocols.WebServiceHandler.Invoke()

    PARAMETERS:

        this = 0x0000000002c733b8

000000001c51e610 000007fef2fc4c90 System.Web.Services.Protocols.WebServiceHandler.CoreProcessRequest()

    PARAMETERS:

        this = 0x0000000002c733b8

000000001c51e680 000007fef35cf047 System.Web.Services.Protocols.SyncSessionlessHandler.ProcessRequest(System.Web.HttpContext)

    PARAMETERS:

        this =

        context =

000000001c51e6d0 000007fef0560347 System.Web.Script.Services.ScriptHandlerFactory+HandlerWrapper.ProcessRequest(System.Web.HttpContext)

    PARAMETERS:

        this =

        context =

000000001c51e700 000007fef05245cb System.Web.HttpApplication+CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()

    PARAMETERS:

        this =

000000001c51e7b0 000007fef0c0caa1 System.Web.HttpApplication.ExecuteStep(IExecutionStep, Boolean ByRef)

    PARAMETERS:

        this = 0x0000000002c272a0

        step = 0x0000000002c2a678

        completedSynchronously = 0x000000001c51e908

000000001c51e850 000007fef0bfde22 System.Web.HttpApplication+PipelineStepManager.ResumeSteps(System.Exception)

    PARAMETERS:

        this = 0x0000000002c2a4b0

        error =

000000001c51e9e0 000007fef0bdf8f9 System.Web.HttpApplication.BeginProcessRequestNotification(System.Web.HttpContext, System.AsyncCallback)

    PARAMETERS:

        this =

        context =

        cb =

000000001c51ea30 000007fef0d06904 System.Web.HttpRuntime.ProcessRequestNotificationPrivate(System.Web.Hosting.IIS7WorkerRequest, System.Web.HttpContext)

    PARAMETERS:

        this = 0x0000000002a45fc0

        wr = 0x0000000002c25080

        context = 0x0000000002c25400

000000001c51eb50 000007fef0d05abb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)

    PARAMETERS:

        managedHttpContext =

        nativeRequestContext = 0x0000000002358f98

        moduleData =

        flags =

000000001c51ecc0 000007fef0d05874 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)

    PARAMETERS:

        managedHttpContext =

        nativeRequestContext =

        moduleData =

        flags =

000000001c51ed20 000007fefa78ef2a DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)

    PARAMETERS:

       

       

       

       

000000001c51f550 000007fef0d069e4 DomainNeutralILStubClass.IL_STUB(IntPtr, System.Web.RequestNotificationStatus ByRef)

    PARAMETERS:

       

000000001c51f630 000007fef0d05abb System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr, IntPtr, IntPtr, Int32)

    PARAMETERS:

        managedHttpContext =

        nativeRequestContext =

        moduleData =

        flags =

000000001c51f7a0 000007fef0d05874 System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr, IntPtr, IntPtr, Int32)

    PARAMETERS:

        managedHttpContext =

        nativeRequestContext =

        moduleData =

        flags =

000000001c51f800 000007fefa78f17b DomainNeutralILStubClass.IL_STUB(Int64, Int64, Int64, Int32)

    PARAMETERS:

       

       

       

       

We need to find “context” object address and dump that object

0:014> !do 0x0000000002c25400
Name: System.Web.HttpContext
MethodTable: 000007fef06b0f28
EEClass: 000007fef0312398
Size: 336(0x150) bytes
 (C:\Windows\assembly\GAC_64\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fef06b3a50  4000fe6        8 …IHttpAsyncHandler  0 instance 0000000000000000 _asyncAppHandler
000007fef06b3580  4000fe7       10 …b.HttpApplication  0 instance 0000000002c272a0 _appInstance
000007fef06b3ac0  4000fe8       18 ….Web.IHttpHandler  0 instance 0000000002c733f8 _handler
000007fef06b3f00  4000fe9       20 …m.Web.HttpRequest  0 instance 0000000002c25550 _request
000007fef06b4280  4000fea       28 ….Web.HttpResponse  0 instance 0000000002c256a0 _response
000007fef06b4798  4000feb       30 …HttpServerUtility  0 instance 0000000000000000 _server
000007fef8b2fcf8  4000fec       38 …Collections.Stack  0 instance 0000000000000000 _traceContextStack
000007fef06b9ad0  4000fed       40 ….Web.TraceContext  0 instance 0000000000000000 _topTraceContext
000007fef8b3e0b0  4000fee       48 …ections.Hashtable  0 instance 0000000002c397d8 _items
000007fef8b3d640  4000fef       50 …ections.ArrayList  0 instance 0000000000000000 _errors
000007fef8b36a20  4000ff0       58     System.Exception  0 instance 0000000000000000 _tempError
000007fef8b35928  4000ff1      118       System.Boolean  1 instance                0 _errorCleared
000007fef8b516b0  4000ff2       60 …ncipal.IPrincipal  0 instance 0000000002c2bc60 _user
000007fef8b39160  4000ff3      100        System.IntPtr  1 instance                0 _pManagedPrincipal
000007fef06b31f8  4000ff4       68 …ofile.ProfileBase  0 instance 0000000000000000 _Profile
000007fef8b76c90  4000ff5      128      System.DateTime  1 instance 0000000002c25528 _utcTimestamp
000007fef06b8bf0  4000ff6       70 …HttpWorkerRequest  0 instance 0000000002c25080 _wr
000007fef8b32528  4000ff7      130 …Services.GCHandle  1 instance 0000000002c25530 _root
000007fef8b39160  4000ff8      108        System.IntPtr  1 instance          21c18e0 _ctxPtr
000007fef06ab060  4000ff9       78 …m.Web.VirtualPath  0 instance 0000000000000000 _configurationPath
000007fef8b35928  4000ffa      119       System.Boolean  1 instance                0 _skipAuthorization
000007fef8b3f0d8  4000ffb       80 …ation.CultureInfo  0 instance 0000000002a40270 _dynamicCulture
000007fef8b3f0d8  4000ffc       88 …ation.CultureInfo  0 instance 0000000002a40270 _dynamicUICulture
000007fef8b3d9c8  4000ffd      110         System.Int32  1 instance                0 _serverExecuteDepth
000007fef8b2fcf8  4000ffe       90 …Collections.Stack  0 instance 0000000000000000 _handlerStack
000007fef8b35928  4000fff      11a       System.Boolean  1 instance                0 _preventPostback
000007fef8b35928  4001000      11b       System.Boolean  1 instance                0 _runtimeErrorReported
000007fef8b35928  4001001      11c       System.Boolean  1 instance                1 _firstNotificationInitCalled
000007fef8b76c90  4001002      138      System.DateTime  1 instance 0000000002c25538 _timeoutStartTime
000007fef8b35928  4001003      11d       System.Boolean  1 instance                1 _timeoutSet
000007fef8b76b90  4001004      140      System.TimeSpan  1 instance 0000000002c25540 _timeout
000007fef8b3d9c8  4001005      114         System.Int32  1 instance                1 _timeoutState
000007fef06c0fa8  4001006       98 …b.Util.DoubleLink  0 instance 0000000002c2bbb8 _timeoutLink
000007fef8b371f8  4001007       a0 ….Threading.Thread  0 instance 0000000002b0ac58 _thread
000007fef06ad480  4001008       a8 …eb.CachedPathData  0 instance 0000000000000000 _configurationPathData
000007fef06ad480  4001009       b0 …eb.CachedPathData  0 instance 0000000002c2b558 _filePathData
000007fef8b36758  400100a       b8        System.String  0 instance 0000000000000000 _sqlDependencyCookie
000007fef06bd218  400100b       c0 …essionStateModule  0 instance 0000000000000000 _sessionStateModule
000007fef0623cf0  400100c       c8 …I.TemplateControl  0 instance 0000000000000000 _templateControl
000007fef0684ad0  400100d       d0 …tificationContext  0 instance 0000000002c734f0 _notificationContext
000007fef8b35928  400100e      11e       System.Boolean  1 instance                1 _isAppInitialized
000007fef8b35928  400100f      11f       System.Boolean  1 instance                1 _isIntegratedPipeline
000007fef8b35928  4001010      120       System.Boolean  1 instance                0 _finishPipelineRequestCalled
000007fef8b35928  4001011      121       System.Boolean  1 instance                0 _impersonationEnabled
000007fef8b35928  4001012      122       System.Boolean  1 instance                0 HideRequestResponse
000007fef8b35928  4001013      123       System.Boolean  1 instance                1 InIndicateCompletion
000007fef06b4d78  4001014       d8 …ion+ThreadContext  0 instance 0000000002c2bac0 IndicateCompletionContext
000007fef06c0e78  4001015       e0 …ronizationContext  0 instance 0000000002c2ba88 _syncContext
000007fef8b35928  4001016      124       System.Boolean  1 instance                0 RequiresSessionState
000007fef8b35928  4001017      125       System.Boolean  1 instance                0 ReadOnlySessionState
000007fef8b35928  4001018      126       System.Boolean  1 instance                0 InAspCompatMode
000007fef06b3ac0  4001019       e8 ….Web.IHttpHandler  0 instance 0000000000000000 _remapHandler
000007fef06b3ac0  400101a       f0 ….Web.IHttpHandler  0 instance 0000000000000000 _currentHandler
000007fef8b35928  400101b      127       System.Boolean  1 instance                1 _ProfileDelayLoad
000007fef06b96d8  400101c       f8 …ielessHelperClass  0 instance 0000000002c257e0 _CookielessHelper
000007fef8b39218  4000fe5      1a0 …flection.Assembly  0   shared           static SystemWebAssembly


Get request object and dump it



0:014> !do 0000000002c25550
Name: System.Web.HttpRequest
MethodTable: 000007fef06b3f00
EEClass: 000007fef0313c90
Size: 336(0x150) bytes
 (C:\Windows\assembly\GAC_64\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
000007fef06b8bf0  400107a        8 …HttpWorkerRequest  0 instance 0000000002c25080 _wr
000007fef06b0f28  400107b       10 …m.Web.HttpContext  0 instance 0000000002c25400 _context
000007fef8b36758  400107c       18        System.String  0 instance 0000000002c27278 _httpMethod
000007fef066d450  400107d      130         System.Int32  1 instance                5 _httpVerb
000007fef8b36758  400107e       20        System.String  0 instance 0000000000000000 _requestType
000007fef06ab060  400107f       28 …m.Web.VirtualPath  0 instance 0000000002c2ee60 _path
000007fef8b36758  4001080       30        System.String  0 instance 0000000000000000 _rewrittenUrl
000007fef8b35928  4001081      138       System.Boolean  1 instance                0 _computePathInfo
000007fef06ab060  4001082       38 …m.Web.VirtualPath  0 instance 0000000002c25808 _filePath
000007fef06ab060  4001083       40 …m.Web.VirtualPath  0 instance 0000000000000000 _currentExecutionFilePath
000007fef06ab060  4001084       48 …m.Web.VirtualPath  0 instance 0000000000000000 _pathInfo
000007fef8b36758  4001085       50        System.String  0 instance 0000000000000000 _queryStringText
000007fef8b35928  4001086      139       System.Boolean  1 instance                0 _queryStringOverriden
000007fef8b3e798  4001087       58        System.Byte[]  0 instance 0000000000000000 _queryStringBytes
000007fef8b36758  4001088       60        System.String  0 instance 0000000002c25170 _pathTranslated
000007fef8b36758  4001089       68        System.String  0 instance 0000000002c391c0 _contentType
000007fef8b3d9c8  400108a      134         System.Int32  1 instance             1394 _contentLength
000007fef8b36758  400108b       70        System.String  0 instance 0000000000000000 _clientTarget
000007fef8b24548  400108c       78      System.Object[]  0 instance 0000000000000000 _acceptTypes
000007fef8b24548  400108d       80      System.Object[]  0 instance 0000000000000000 _userLanguages
000007fef0620148  400108e       88 …owserCapabilities  0 instance 0000000000000000 _browsercaps
000007fef7fd19a8  400108f       90           System.Uri  0 instance 0000000000000000 _url
000007fef7fd19a8  4001090       98           System.Uri  0 instance 0000000000000000 _referrer
000007fef06c2000  4001091       a0 …b.HttpInputStream  0 instance 0000000002c73370 _inputStream
000007fef0671860  4001092       a8 …ClientCertificate  0 instance 0000000000000000 _clientCertificate
000007fef8b25b30  4001093       b0 …l.WindowsIdentity  0 instance 0000000000000000 _logonUserIdentity
000007fef061fb30  4001094       b8 …tpValueCollection  0 instance 0000000000000000 _params
000007fef061fb30  4001095       c0 …tpValueCollection  0 instance 0000000000000000 _queryString
000007fef061fb30  4001096       c8 …tpValueCollection  0 instance 0000000000000000 _form
000007fef06c1048  4001097       d0 …pHeaderCollection  0 instance 0000000002c38d70 _headers
000007fef061fcc0  4001098       d8 …verVarsCollection  0 instance 0000000000000000 _serverVariables
000007fef06be4f8  4001099       e0 …pCookieCollection  0 instance 0000000000000000 _cookies
000007fef0653b10  400109a       e8 …ttpFileCollection  0 instance 0000000000000000 _files
000007fef06c15e0  400109b       f0 …awUploadedContent  0 instance 0000000002c70d90 _rawContent
000007fef8b35928  400109c      13a       System.Boolean  1 instance                1 _readEntityBody
000007fef8b24548  400109d       f8      System.Object[]  0 instance 0000000000000000 _multipartContentElements
000007fef8b3e5a8  400109e      100 System.Text.Encoding  0 instance 0000000000000000 _encoding
000007fef0648ae0  400109f      108 …treamFilterSource  0 instance 0000000000000000 _filterSource
000007fef8b39708  40010a0      110     System.IO.Stream  0 instance 0000000000000000 _installedFilter
000007fef061f530  40010a1      140 …SimpleBitVector32  1 instance 0000000002c25690 _flags
000007fef8b36758  40010a4      118        System.String  0 instance 0000000000000000 _AnonymousId
000007fef06ab060  40010a5      120 …m.Web.VirtualPath  0 instance 0000000002c25808 _clientFilePath
000007fef06ab060  40010a6      128 …m.Web.VirtualPath  0 instance 0000000000000000 _clientBaseDir
000007fef8b36048  40010a2      1b0        System.Object  0   shared           static s_browserLock

Get _pathTranslated object and this will provide URL which is being executed

0:014> !do 0000000002c25170

Name: System.String

MethodTable: 000007fef8b36758

EEClass: 000007fef873e520

Size: 204(0xcc) bytes

 (C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)

String: E:\Inetpub\wwwroot\webservices\SiebelExtractInterface\2011\04\01\SiebelExtractLookup.asmx

Fields:

              MT    Field   Offset                 Type VT     Attr            Value Name

000007fef8b3d9c8  4000096        8         System.Int32  1 instance               90 m_arrayLength

000007fef8b3d9c8  4000097        c         System.Int32  1 instance               89 m_stringLength

000007fef8b381a0  4000098       10          System.Char  1 instance               45 m_firstChar

000007fef8b36758  4000099       20        System.String  0   shared           static Empty

                                 >> Domain:Value  00000000023f1fb0:0000000002981308 000000000246c1c0:0000000002981308 00000000024cea50:0000000002981308 00000000024e8b60:0000000002981308 000000001c578a20:0000000002981308 000000001c58f590:0000000002981308 0000000026905040:0000000002981308 0000000026941740:0000000002981308 000000001c56fe10:0000000002981308 0000000026b59100:0000000002981308 000000002684a790:0000000002981308 <<

000007fef8b38050  400009a       28        System.Char[]  0   shared           static WhitespaceChars

• Now we need to know what is being passed to method in question which end up in high CPU utilization. From output of !clrstack -p you we can see address for that object 

0:014> !do 0x0000000002c75a90

Name: System.String

MethodTable: 000007fef8b36758

EEClass: 000007fef873e520

Size: 48(0x30) bytes

 (C:\Windows\assembly\GAC_64\mscorlib\2.0.0.0__b77a5c561934e089\mscorlib.dll)

String: 4236762-COM

Fields:

              MT    Field   Offset                 Type VT     Attr            Value Name

000007fef8b3d9c8  4000096        8         System.Int32  1 instance               12 m_arrayLength

000007fef8b3d9c8  4000097        c         System.Int32  1 instance               11 m_stringLength

000007fef8b381a0  4000098       10          System.Char  1 instance               34 m_firstChar

000007fef8b36758  4000099       20        System.String  0   shared           static Empty

                                 >> Domain:Value  00000000023f1fb0:0000000002981308 000000000246c1c0:0000000002981308 00000000024cea50:0000000002981308 00000000024e8b60:0000000002981308 000000001c578a20:0000000002981308 000000001c58f590:0000000002981308 0000000026905040:0000000002981308 0000000026941740:0000000002981308 000000001c56fe10:0000000002981308 0000000026b59100:0000000002981308 000000002684a790:0000000002981308 <<

000007fef8b38050  400009a       28        System.Char[]  0   shared           static WhitespaceChars

So when this 4236762-COM being passed to Company.SiebelExtract.Services.SiebelExtractLookup.lookupAssetAndEntitlementHeirarchyByGrantNumber it’s causing high CPU utiliation.

This is sufficient information for developers to debug the code on their own at this point.