Диагностика Troubleshooting

• Диагностика Troubleshooting
  • Tools
    • Process Monitor
    • Dump
    • Трассировка
  • Use case
    • Application deadlocks
    • High CPU
    • High Latency Timeout
    • Thread pool starvation
    • High GС .NET LOH
    • Session lock Concurrent Requests

Tools

Process Monitor

• procmon
• process-explorer

Dump

• Windows Debugging Tools installed on the server (takes longer)
• ProcDump (faster)

Трассировка

• Fiddler on iis app
• IIS Failed Request Tracing FREB
  • For slow ASP.NET Webforms applications
  • for troubleshooting slow ASP.NET MVC web sites using PerfView
• CPU, ERROR, RAM leak
• WCF

Use case

Application deadlocks

• Признаки
  • All requests in the application are blocked in the code, due to taking a lock
• Проверки
  • This can only be identified by attaching a debugger at the moment of the hang

High CPU

• Признаки
  • CPU remain at a high level 80% or greater for prolonged periods
  • CPU overload is usually caused by CPU-intensive application code
    • LINQ query compilation
    • JSON serialization
    • regular expressions
    • logging, dependency injection, reflection, etc
  • high concurrency overwhelming the processor’s ability to process tasks concurrently. This is becoming even more and more common with parallel tasks and async code.
• Проверки
  • If you need to identify which application pool is associated with a particular w3wp.exe process
    • run “%windir%\System32\inetsrv\appcmd list wp”
    • This will show the process identifier (PID) of the w3wp.exe process in quotes. You can match that PID with the PID available in Task Manager.
  • To determine what is causing the problem:
    • A Performance Monitor data collector set
    • A user-mode memory dump of the w3wp.exe process
      • collect user-mode process dumps when a high CPU condition occurs is to use Debug Diagnostics.

High Latency Timeout

• Признаки
  • “Http Service Request Queues\MaxQueueItemAge” performance counter increasing
  • “Http Service Request Queues\ArrivalRate” counter exceeds the “W3WP_W3SVC\Requests / sec” counter for the application pool’s worker process over a period of time
  • Slow application code
    • The requests may be blocked by slow data retrieval, e.g. slow SQL queries, or slow responses from Redis/cloud-based databases/remote web services
• Проверки
  • “Http Service Request Queues\CurrentQueueSize” counter
    • If its 0, IIS is having no problems dequeueing requests.
  • “W3WP_W3SVC\Active Threads” counter
    • This will almost always be 0, or 1 because IIS threads almost never block.
      • If its significantly higher, you likely have IIS thread blockage due to a custom module or because you explicitly configured ASP .NET to run on IIS threads
    • Consider increasing your MaxPoolThreads registry key.

Thread pool starvation

• Признаки
  • All threads in the application are blocked, and the thread pool is not able to provide sufficient threads to process new requests, causing them to become queued
  • the threadpool cannot provide enough threads to enable completions of async tasks, resulting in all requests becoming blocked waiting for threads
  • is blocking calls that could be asynchronous
    • Many synchronous blocking calls lead to Thread Pool starvation and degraded response times
• Проверки
  • profiler, such as PerfView, can be used to find threads frequently added to the Thread Pool. The “Microsoft-Windows-DotNETRuntime/ThreadPoolWorkerThread/Start” event indicates a thread added to the thread pool.
• Решение
  • IIS by default will only allow 1 thread per processor core to actively dequeue requests (that MaxConcurrency setting)
    • In a CPU-bound workload, having more threads does not result in a higher RPS, but lower RPS instead because more of the processor bandwidth is spent context switching.
    • More threads only benefit if threads are becoming blocked by waits or blocking IO, which does not typically happen for IIS threads.

High GС .NET LOH

• Minimize large object allocations
  • The .NET Core garbage collector manages allocation and release of memory automatically in ASP.NET Core apps. Automatic garbage collection generally means that developers don’t need to worry about how or when memory is freed.
  • However, cleaning up unreferenced objects takes CPU time, so developers should minimize allocating objects in hot code paths
  • Garbage collection is especially expensive on large objects (>= 85,000 bytes)
    • Large objects are stored on the large object heap (LOH) and require a full (generation 2) garbage collection to clean up
    • Unlike generation 0 and generation 1 collections, a generation 2 collection requires a temporary suspension of app execution. Frequent allocation and de-allocation of large objects can cause inconsistent performance.
    • The allocation cost is high because the memory for a newly allocated large object has to be cleared. The CLR guarantees that memory for all newly allocated objects is cleared.
• Признаки
  • application spends a large percentage of it’s processing time trying to clean up .NET objects
• Проверки
  • If your GC time is over 50% and higher, it’s going to cause poor performance.
  • Memory issues, such as the preceding, can be diagnosed by reviewing garbage collection (GC) stats in PerfView and examining:
    • Garbage collection pause time
    • What percentage of the processor time is spent in garbage collection
    • How many garbage collections are generation 0, 1, and 2

Session lock Concurrent Requests

• Признаки
  • Access to ASP.NET session state is exclusive per session, which means that if two different users make concurrent requests, access to each separate session is granted concurrently.
  • However, if two concurrent requests are made for the same session (by using the same SessionID value), the first request gets exclusive access to the session information.
  • The second request executes only after the first request is finished
• Проверки
• Решение
  • If the EnableSessionState value in the @ Page directive is set to ReadOnly, a request for the read-only session information does not result in an exclusive lock on the session data. However, read-only requests for session data might still have to wait for a lock set by a read-write request for session data to clear.