Showing posts with label Performance counters. Show all posts
Showing posts with label Performance counters. Show all posts

2016-02-29

Linux top command on Windows, further investigations

In my previous post, I spoke of "Normalized" and "Non-Normalized" CPU utilization values:
--

TABLE:

Foreword: Windows is not "process" based OS (like Linux) but rather "thread" based so all of the numbers relating to CPU usage are approximations. I did made a "proper" CPU per Process looping and summing up Threads counter (https://msdn.microsoft.com/en-us/library/aa394279%28v=vs.85%29.aspx) based on PID but that proved too slow given I have ~1 sec to deal with everything. CPU utilization using RAW counters with 1s delay between samples proved to produce a bit more reliable result than just reading Formatted counters but, again, too slow for my 1s ticks (collect sample, wait 1s, collect sample, do the math takes longer than 1s). Thus I use PerfFormatted counters in version 0.9RC.
    Win32_PerfRawData_PerfProc_Process; Win32_PerfFormattedData_PerfProc_Process.
  
    _PID_     Unique identified of the process.
    PPID      Unique identifier of the process that started this one.
    PrioB     Base priority.
    Name      Name of the process.
    CPUpt_(N) % of CPU used by process. On machines with multiple CPUs,
        this number can be over 100% unless you see _CPUpt_N caption which
        means "Normalized" (i.e. CPUutilization / # of CPUs).
        Toggle Normal/Normalized display by pressing the "p" key.
    Thds      # of threads spawned by the process.
    Hndl      # of handles opened by the process.
    WS(MB)    Total RAM used by the process. Working Set is, basically,
        the set of memory pages touched recently by the threads belonging to
        the process. 
    VM(MB)    Size of the virtual address space in use by the process.
    PM(MB)    The current amount of VM that this process has reserved
        for use in the paging files.
However, my approach for displaying "Non-Normalized" CPU utilization didn't work :-/

Proper functionality of this feature is rather important for my job. Looking at "Normalized" CPU utilization values for a process does not tell you much. Say a process has CPU utilization of 100%. This just tells you there is at least 1 CPU that's fully utilized by the process but it does not tell you the overall utilization. "Non-Normalized" value sums CPU utilization over all CPUs that process uses. In my case, the test box has 8 Xeon processors with 6 physical and 6 virtual cores each totaling at 96 CPUs. The system is configured as such that NUMA node corresponds to 1 Xeon processor (socket). Thus, when my process utilizes entire NUMA node (socket) to the fullest, the CPU utilization for that process should be number of CPUs per Numa/Socket (12) x 100% which is 1200%:


If the process scales correctly, I will see more NUMA nodes/Sockets light up while increasing the load:

However, this does not tell me it is my process of interest that is using the CPUs. To confirm it, I need TOP script showing CPU utilization of above 1200%:

This guarantees me mysqld process is running on more than 2 sockets (sysbench is taking up ~7 CPUs and I bet mydesktopservice is the one lighting up 3rd CPU in 2nd row).

How to make it work:

Heavy rework of #region Tasks job which is starting the "Processes" monitoring job was in order. First, I had to remove all of the below code:
<#
        if ($CPUDSw) {
            Get-CimInstance Win32_PerfFormattedData_PerfProc_Process | 
                select @{Name='_PID_'; Expression={$_.IDProcess}},
                @{Name='PPID'; Expression={$_.CreatingProcessID}},
                @{Name='PrioB'; Expression={$_.PriorityBase}},
                @{Name='Name                  '; Expression={(($_.Name).PadRight(22)).substring
                    (0, [System.Math]::Min(22, ($_.Name).Length))}}, 
                @{Name='_CPUpt__'; Expression={($_.PercentProcessorTime).ToString("0.00").PadLeft(8)}},
                @{Name='Thds'; Expression={$_.ThreadCount}},
                @{Name='Hndl'; Expression={$_.HandleCount}},
                @{Name='WS(MB)'; Expression={[math]::Truncate($_.WorkingSet/1MB)}},
                @{Name='VM(MB)'; Expression = {[math]::Truncate($_.VirtualBytes/1MB)}},
                @{Name='PM(MB)'; Expression={[math]::Truncate($_.PageFileBytes/1MB)}} |
                where { $_._PID_ -gt 0} | &$sb | 
                Select-Object -First $procToDisp | FT * -Auto 1> $pth
        } else {
            Get-CimInstance Win32_PerfFormattedData_PerfProc_Process | 
                select @{Name='_PID_'; Expression={$_.IDProcess}},
                @{Name='PPID'; Expression={$_.CreatingProcessID}},
                @{Name='PrioB'; Expression={$_.PriorityBase}},
                @{Name='Name                  '; Expression={(($_.Name).PadRight(22)).substring
                    (0, [System.Math]::Min(22, ($_.Name).Length))}}, 
                @{Name='_CPUpt_N'; Expression={"{0,8:N2}" -f ($_.PercentProcessorTime / $TotProc)}},
                @{Name='Thds'; Expression={$_.ThreadCount}},
                @{Name='Hndl'; Expression={$_.HandleCount}},
                @{Name='WS(MB)'; Expression={[math]::Truncate($_.WorkingSet/1MB)}},
                @{Name='VM(MB)'; Expression = {[math]::Truncate($_.VirtualBytes/1MB)}},
                @{Name='PM(MB)'; Expression={[math]::Truncate($_.PageFileBytes/1MB)}} |
                where { $_._PID_ -gt 0} | &$sb | 
                Select-Object -First $procToDisp | FT * -Auto 1> $pth
        }
#>
and replace it with Get-Counter version:
        $processes = Get-CimInstance Win32_PerfFormattedData_PerfProc_Process | 
            Select @{Name='_PID_'; Expression={$_.IDProcess}},
            @{Name='PPID'; Expression={$_.CreatingProcessID}},
            ElapsedTime, 
            @{Name='PrioB'; Expression={$_.PriorityBase}},
            @{Name='Name'; Expression={($_.Name).ToLower()}},
            @{Name='Thds'; Expression={$_.ThreadCount}},
            @{Name='Hndl'; Expression={$_.HandleCount}}, 
            @{Name='WS(MB)'; Expression={[math]::Truncate($_.WorkingSet/1MB)}},
            @{Name='VM(MB)'; Expression = {[math]::Truncate($_.VirtualBytes/1MB)}},
            @{Name='PM(MB)'; Expression={[math]::Truncate($_.PageFileBytes/1MB)}},
            PoolNonpagedBytes, PoolPagedBytes, PercentProcessorTime |
            Where { $_._PID_ -gt 0}

        $Samples = (Get-Counter “\Process(*)\% Processor Time”).CounterSamples

Just noting Get-Counter example:
PS:511 [HEL01]> (Get-Counter “\Process(*)\% Processor Time”).CounterSamples | FL *
...
Path             : \\hel01\process(system)\% processor time
InstanceName     : system
CookedValue      : 0
RawValue         : 3434062500
SecondValue      : 131012088253272040
MultipleCount    : 1
CounterType      : Timer100Ns
Timestamp        : 29.02.16 09:40:25
Timestamp100NSec : 131012124253270000
Status           : 0
DefaultScale     : 0
TimeBase         : 10000000
Then I had to change the way of putting it all together:
        if ($CPUDSw) { 
            $pcts = $Samples | Select @{Name=”IName"; Expression={($_.InstanceName).ToLower()}}, 
              @{Name=”CPUU”;Expression={[Decimal]::Round(($_.CookedValue), 2)}}
            $processes | select '_PID_', 'PPID', 'PrioB',
                            @{Name='Name                  '; Expression=
                                {
                                    (($_.Name).PadRight(22)).substring(0, [System.Math]::Min(22, ($_.Name).Length))
                                }
                            }, 
                            @{Name='_CPUpt__'; Expression=
                                {
                                    if ($pcts.IName.IndexOf($_.Name) -ge 0) {
                                        ($pcts.CPUU[[array]::IndexOf($pcts.IName, $_.Name)]).ToString("0.00").PadLeft(8)
                                    }
                                }
                            },
            'Thds', 'Hndl', 'WS(MB)', 'VM(MB)', 'PM(MB)' | &$sb | Select-Object -First $procToDisp | FT * -Auto 1> $pth
        } else {
            $pcts = $Samples | Select @{Name=”IName"; Expression={($_.InstanceName).ToLower()}}, 
              @{Name=”CPUU”;Expression={[Decimal]::Round(($_.CookedValue / $TotProc), 2)}}
            $processes | select '_PID_', 'PPID', 'PrioB',
                            @{Name='Name                  '; Expression=
                                {
                                    (($_.Name).PadRight(22)).substring(0, [System.Math]::Min(22, ($_.Name).Length))
                                }
                            },  
                            @{Name='_CPUpt_N'; Expression=
                                {
                                    if ($pcts.IName.IndexOf($_.Name) -ge 0) {
                                        ($pcts.CPUU[[array]::IndexOf($pcts.IName, $_.Name)]).ToString("0.00").PadLeft(8)
                                    } else {
                                        #Not found (yet). Take what you have :-/
                                        ($_.PercentProcessorTime).ToString("0.00").PadLeft(8)
                                    }
                                }
                            },
            'Thds', 'Hndl', 'WS(MB)', 'VM(MB)', 'PM(MB)' | &$sb | Select-Object -First $procToDisp | FT * -Auto 1> $pth
        }
Since Get-Counter, by default, takes samples 1 second apart:
PS:507 [HEL01]> Measure-Command{(Get-Counter “\Process(*)\% Processor Time”).CounterSamples}

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 1
Milliseconds      : 18
Ticks             : 10189709
TotalDays         : 1.17936446759259E-05
TotalHours        : 0.000283047472222222
TotalMinutes      : 0.0169828483333333
TotalSeconds      : 1.0189709
TotalMilliseconds : 1018.9709
I also abandoned all of the code relating to Timer:
    #$sw = New-Object Diagnostics.Stopwatch
    do {
        #$sw.Start()
...
        }
        #$sw.Stop()
        #if ($sw.ElapsedMilliseconds -lt 1000) {
        #    Start-Sleep -Milliseconds (1000-$sw.ElapsedMilliseconds)
        #}
        #$sw.Reset()

    } while ($true)
    #$sw = $null

So now it works! I do not know right now when I will be able to release the new version so stay tuned.

Final thoughts:

I have hit many many problems in Windows during this testing. Just note, for example, the use of ToLower() in ($_.InstanceName).ToLower() but this is something for the new blog post. This one is about TOP script.

2016-01-21

Using Powershell to implement Linux top command on Windows

Welcome to the final blog in Windows PerfCounters and Powershell series and sorry for the delay. The purpose of this blog is to explain the inner workings of top-script.ps1 script and practical usage of Performance counters on Windows through Powershell. It is intended for people who want Linux top - like tool on Windows.

The script is a part of and available in our existing benchmarking package (dbt2-0.37.50.10) developed by Mikael Ronstrom.

On Top:

If you ever did benchmarking on Linux or simply wondered "where did all my resources go", top is your best friend. Since this post is not about Linux, you can google "Linux top explained" for more details.


On Performance counters:

To learn about Windows PerfCounters, please refer to my previous blog entries in this series. I will be addressing System.Diagnostics class as just Diagnostics.


On Powershell:

For ages, Windows users were looking at bash wondering why they do not have anything similar to it. After much trial and error, Microsoft delivered Powershell. In my humble opinion, Powershell is simply great! As per difference between Powershell and bash I would mention just one; Powershell pipe passes objects while bash pipe passes plain text.


On script itself:

Type "perfmon deleting files" in Google and you'll see why I made this script ;-) Joke aside, we have a mature testing/benchmarking framework written in bash and wanted the same look and feel on Windows. Top script is just the latest piece of that effort.
I undertook this work since I firmly believe in native tools when dealing with performance issues. If I was just after measuring performance delatas between versions, some generic tool written for some other platform, such as in Perl or similar, would have been good enough. But, IMO, it would not have been fair to non-native OS.
Also, studying native tools is an integral part of studying the OS itself which is something you can not tackle performance issues without.

The script will evolve naturally to cover for the information we need in our everyday work.

Using Windows performance counters through PowerShell CIM classes it is possible to gather stats on computer performance. The script functions like this:
  o Main code starts 2 background jobs; one for collecting details for header table ("Top_Header_Job") and one for processes table ("Top_Processes_Job").
  o Each of the jobs then collects stats and writes them into a file ("header" job to TempDir\headerres.txt file, "tasks" job to TempDir\thrprocstats.txt) which is, in turn, read by script main code. Main code uses TempDir\topcfg.txt to pass info to tasks (currently, the field to sort the table by). All of the files are overwritten each time so there is not much data in them.
  o After the files are read by main code, the data is displayed. To be able to properly position the output on screen, I used various [console] functions not available in PowerShell_ISE.


Requirements:

The script can not be run in _ISE. Use Powershell console.
The script requires PS 3+.
The script requires at least 50 x 80 console window.
The script might work with .NET FW older than 4 but it was tested only with .NET 4.5.x.


Getting started:

1) Put script somewhere.
2) Start PowerShell (NOT PS_ISE)
3) cd to "somewhere" directory
4) .\top-script.ps1
        a) Get-Help .\top-script.ps1
        b) Get-Help .\top-script.ps1 -Examples
5) While script is running you can use (single key) shortcuts:
  q - Quit
  m - Sort by process WS (occupied RAM) DESC, CPUpt DESC
  p - Sort by process CPU utilization DESC, WS(MB) DESC; Non-Normalized/Normalized.
Note: Script is started with CPU utilization by process as de-normalized (i.e. on multi-CPU boxes, this value can be well over 100%). To display normalized values (i.e. "non-normalized" value / # of CPUs), just press "p" again. IF non-normalized value is the source for data, the column title will be '_CPUpt__'. Normalized CPU utilization value (CPUpt / # of CPUs) will display as '_CPUpt_N'.
  n - Sort by process Name ASC
  r - Sort by process PID ASC, CPUpt DESC
  + - Display individual CPU's. Comma separated list of values (i.e. 0,1,2).
  1 - Display individual Processors (Sockets). Comma separated list of values (i.e. 0,1,2).
Note: Script displays either Socket load or CPU load.
  - - Cancel displaying individual CPUs/Sockets.


The output:

09:21:17, Uptime:00d:00h:36m,  Users:  1,   # thds ready but queued for CPU: 0
-------------------------------------------------------------------------------
| RUNNING           | CPU                           | RAM[MB]                 |
-------------------------------------------------------------------------------
| services:    104  | Sys: 50.78%(P  4.30%/U 46.48%)| Installed:         8192 |
| processes:   126  | Idle:49.22%                   | HW reserv:       320.77 |
| threads:    1483  | HWint:             7087/0.38% | Visible:        7871.23 |
| handles:   32696  | SWint:              278/0.38% | Available:         5136 |
| CoSw/s:     6351  | High-prio thd exec:    50.38% | Modified:        117.07 |
|                   | Total # of cores:           4 | Standby:        3936.44 |
|                   |                               | PagesIn/Read ps:      2 |
-------------------------------------------------------------------------------



_PID_ PPID PrioB Name                      CPUpt Thds Hndl WS(MB) VM(MB) PM(MB)
----- ---- ----- ----                      ----- ---- ---- ------ ------ ------
 3916  852     8 mcshield                  10.14   53  490     46    225    100
 1836 5452     8 powershell#2               1.09   15  377     67    616     46
 7436 5452     8 powershell#1               0.72   17  556     85    624     67
 3984  156     8 WmiPrvSE                   0.72    9  303     14     56      9
 7024  156     8 WmiPrvSE#2                 0.72    7  201     10     52      7
 5452 5148     8 powershell                 0.36   18  454     90    628     79
 2292  852     8 FireSvc                    0.36   28  539     10    156     36
 7864 5148     8 thunderbird                0.00   52  630    293    656    264
 2880 5148     8 powershell_ise             0.00   12  427    181    869    164
 5148 3860     8 explorer                   0.00   25  810     81    267     53
 7028 6648     8 googledrivesync#1          0.00   29  712     76    193     64
 1124  852     8 svchost#5                  0.00   45 1560     46    187     31
 6508 5148     8 sidebar                    0.00   20  433     39    195     20
  816  796    13 csrss#1                    0.00   10  765     35    126      3
 6592 5148     8 iCloudServices             0.00   16  442     32    167     18
 6868 6764     8 pcee4                      0.00    7  202     32    612     32
 1976 1052    13 dwm                        0.00    5  135     31    140     26
 3400  156     8 WmiPrvSE#1                 0.00   12  297     28     91     22
 1088  852     8 svchost#4                  0.00   21  584     27    126     14
 3648  852     8 dataserv                   0.00   10  510     24    224     21
  980  852     8 svchost#2                  0.00   25  575     23    119     26
 2404  852     8 PresentationFontCache      0.00    6  149     21    506     28
...

HEADER data

Current time, uptime, # of active users, # of threads per CPU that are ready for execution but can't get CPU cycles (obviously, you want to keep this as low as possible (<= 2)).
        RUNNING section
            # of services in Started state
            # of user processes
            # of threads spawned
            # of handles open
            # of context switches per second
        CPU section
            % of CPU used (% used by privileged instr. / % used by user instr.)
            % of CPU consumed by Idle process.
            # of HW interrupts per sec./% of CPU used to service HW interrupts
            # of SW interrupts queued for servicing per sec./
                % of CPU used to service SW interrupts
            % of CPU consumed by high-priority threads execution.
            # of phys. and virt. cores. Here, 4 is Dual-Core with HT enabled.
        RAM[MB] section
            Installed RAM.
            RAM reserved by Windows for HW.
            Amount of RAM user actually sees.
            Amount of available RAM for user processes.
            Amount of RAM marked as "Modified".
            Amount of RAM marked as "Standby" (cached).
            Ratio between Memory\Pages Input/sec and Memory\Page Reads/sec.
                Number of pages per disk read. Should keep below 5.

TABLE data

        _PID_    Unique identified of the process.
        PPID     Unique identifier of the process that started this one.
        PrioB    Base priority.
        Name     Name of the process.
        CPUpt    % of CPU used by process.
        Thds     # of threads spawned by the process.
        Hndl     # of handles opened by the process.
        WS(MB)   Total RAM used by the process. Working Set is, basically,
            the set of memory pages touched recently by the threads belonging
            to the process. 
        VM(MB)   Size of the virtual address space in use by the process.
        PM(MB)   The current amount of VM that this process has reserved for
            use in the paging files.

Longer explanation of the values:

HEADER:

Foreword: Since Windows is not "process" based OS (like Linux) it is impossible to calculate the "System load". The next best thing is CPU queue length (see below).
  Uptime: Diagnostics.PerformanceCounter("System", "System Up Time")
  Users:  WMI query using query.exe tool which should be a part of your Windows.
           query user /server:localhost
           Number of users currently logged in. If no query.exe, the value is -1.
  # thds ready but queued for CPU:
          Diagnostics.PerformanceCounter("System", "Processor Queue Length")
           How many threads are in the processor queue ready to be executed but not
           currently able to use cycles. Windows OS has single queue length counter
           thus the value displayed is counter value divided with number of CPU's.
           Link.

    RUNNING section:
      services:
          (Get-Service | Where-Object {$_.Status -ne 'Stopped'} | Measure-Object).Count
           Total # of services actually running.
      processes:
          Diagnostics.PerformanceCounter("System", "Processes")
           Total number of user processes running.
           Link.
      threads:
          Diagnostics.PerformanceCounter("System", "Threads")
           Total # of threads spawned.
      handles:
          Diagnostics.PerformanceCounter("Process", "Handle Count")
           Total # of open handles.
      CoSw/s:
          Diagnostics.PerformanceCounter("System", "Context Switches/sec")
           Context switching happens when a higher priority thread pre-empts a lower
           priority thread that is currently running or when a high priority thread
           blocks. High levels of context switching can occur when many threads share
           the same priority level. This often indicates that there are too many
           threads competing for the processors on the system. If you do not see much
           processor utilization and you see very low levels of context switching, it
           could indicate that threads are blocked.
           Link.

    CPU section:
    Foreword: Windows OS has special thread called "Idle" which consumes free CPU cycles thus these counters return values relating to this one. Also, Windows are not "process" based but rather "thread" based so all of these numbers are approximations. This is even more important in the TABLE which shows CPU utilization per process (see explanation there). Most of these counters are multi-instance so instance name is '_Total' (ie. CPU utilization in total as opposed to per NUMA node, Core, CPU...).
      Sys: nn.nn%(P  mm.mm%/U zz.zz%):
          Diagnostics.PerformanceCounter("Processor Information","% Processor Time"),
           Diagnostics.PerformanceCounter("Processor Information","% Privileged Time"),
           Diagnostics.PerformanceCounter("Processor Information","% User Time").
           First number shows, effectively, % of cycles CPU(s) didn't spend running the
           Idle thread. Second number is the time CPU(s) spent on executing Privileged
           instructions while third is time CPU(s) spent executing user-mode instructions.
           For example, when your application calls operating system functions (say to
           perform file or network I/O or to allocate memory), these operating system
           functions are executed in Privileged mode.
           Link.
      Idle:
          Diagnostics.PerformanceCounter("Processor Information", "% Idle Time")
           Link.
      HWint:
          Diagnostics.PerformanceCounter("Processor Information","Interrupts/sec"),
           Diagnostics.PerformanceCounter("Processor Information","% Interrupt Time").
           Rate of hardware interrupts per second and a percent of CPU time this takes.
           Link.
      SWint:
          Diagnostics.PerformanceCounter("Processor Information","DPCs Queued/sec"),
           Diagnostics.PerformanceCounter("Processor Information","% DPC Time").
           Rate at which software interrupts are queued for execution and a % of CPU time
           this takes.
           Link.
      High-prio thd exec:
          Diagnostics.PerformanceCounter("Processor Information","% Priority Time").
           CPU utilization by high priority threads.
           Link: Can't find any links in MSDN...
      Total # of cores:
          (Get-CimInstance Win32_ComputerSystem).NumberOfLogicalProcessors
           Number of physical and virtual cores present.

    RAM[MB] section:
      Installed:
          (GCIM -class "cim_physicalmemory" | Measure-Object Capacity -Sum).Sum/1024/1024
      HW reserv:
          Installed - Visible ;-)
      Visible:
          (Get-CimInstance win32_operatingsystem).TotalVisibleMemorySize
      Available:
          Diagnostics.PerformanceCounter("Memory","Available MBytes")
      Modified:
          Diagnostics.PerformanceCounter("Memory","Modified Page List Bytes")
      Standby:
          Diagnostics.PerformanceCounter("Memory","Standby Cache Core Bytes") + 
           Diagnostics.PerformanceCounter("Memory",
             "Standby Cache Normal Priority Bytes") + 
           Diagnostics.PerformanceCounter("Memory","Standby Cache Reserve Bytes")
           Basically, cache memory.
      PagesIn/Read ps:
          Diagnostics.PerformanceCounter("Memory","Pages Input/sec")/ 
           Diagnostics.PerformanceCounter("Memory","Page Reads/sec")
           Ratio between Memory\Pages Input/sec and Memory\Page Reads/sec which
           is number of pages per disk read. Should keep below 5.
    

TABLE:

Foreword: Windows is not "process" based OS (like Linux) but rather "thread" based so all of the numbers relating to CPU usage are approximations. I did made a "proper" CPU per Process looping and summing up Threads counter (https://msdn.microsoft.com/en-us/library/aa394279%28v=vs.85%29.aspx) based on PID but that proved too slow given I have ~1 sec to deal with everything. CPU utilization using RAW counters with 1s delay between samples proved to produce a bit more reliable result than just reading Formatted counters but, again, too slow for my 1s ticks (collect sample, wait 1s, collect sample, do the math takes longer than 1s). Thus I use PerfFormatted counters in version 0.9RC.
    Win32_PerfRawData_PerfProc_Process; Win32_PerfFormattedData_PerfProc_Process
    Link.
  
    _PID_     Unique identified of the process.
    PPID      Unique identifier of the process that started this one.
    PrioB     Base priority.
    Name      Name of the process.
    CPUpt_(N) % of CPU used by process. On machines with multiple CPUs,
        this number can be over 100% unless you see _CPUpt_N caption which
        means "Normalized" (i.e. CPUutilization / # of CPUs).
        Toggle Normal/Normalized display by pressing the "p" key.
    Thds      # of threads spawned by the process.
    Hndl      # of handles opened by the process.
    WS(MB)    Total RAM used by the process. Working Set is, basically,
        the set of memory pages touched recently by the threads belonging to
        the process. 
    VM(MB)    Size of the virtual address space in use by the process.
    PM(MB)    The current amount of VM that this process has reserved
        for use in the paging files.
Note that it is possible to display CPU/Socket data for chosen HW by pressing + or 1 keys, entering 0-based index and separating multiple values by ,:

         User  Priv  Idle  HWin  SWIn              User  Priv  Idle  HWin  SWIn
-------------------------------------     -------------------------------------
%CPU  0:   47,    5,   47,    0,    0     %CPU  1:    0,    0,  100,    0,    0
%CPU  2:   35,   11,   52,    0,    0     %CPU  3:    5,    0,   94,    0,    0
The input here was 0,1,2,3 thus displaying data about first 4 cores. The CPU/Socket data is displayed between the Header and the Table areas reducing the number of visible processes. To remove this information from screen, just press "-" key.

INNER WORKINGS:

In general, script output comprises of Header part and Table part showing details on processes. In-between the two, you can show Processor/Core info. There are two background jobs started to accomplish this; "Top_Header_Job" & "Top_Processes_Job". The data about individual processors/cores is calculated in main script body.

Script starts with my usual checks, proceeds to variable declaration part where I initialize some of the performance counters (which takes time) and then starts Header and Processes jobs. The jobs itself follow the same logic. I.e. I first start perfcounter instances (which takes time) and then loop through values passing them back in file.

Main script body collects the data from files refreshing the display. Also, main script is in charge of displaying individual processor/core data as well as monitoring the keyboard input. This means CTRL+C will NOT work but you can still stop the script with CTRL+BREAK:
[console]::TreatControlCAsInput = $true
Regular way to exit is pressing the "q" key.
After you press the "q" key, cleanup code is executed, stopping the background jobs and removing temporary files used for communication. It's worth noting that cleanup code does not throw any errors. This is because nothing bad can happen. Files are less than 1kB in total while background jobs can be stopped either via trick described below or simply by exiting Powershell console.

Lets go deeper into the regions of code now. First region is Check which I described in October 2015 blog so no need to repeat myself. Next is Variable Declarations region where I gather one-time top-level data, mainly related to CPU topology using tricks described in Blog 3 and Blog 4 by manipulating Instances as described in Blog 1 of this series. Executing this part takes couple of seconds.

Next thing is to start the Header job. It takes argument (total number of cores) from the call and proceeds with initializing various counters. As with all initializations, this also takes couple of seconds. Main DO loop starts the timer to ensure samples are collected in 1 second intervals. Also, it checks if you have query.exe tool installed and determines the number of active users, if the tool exists, or displays -1 if it doesn't. There are other ways of determining number of logged users but they are all too slow for 1s tick. After forming the resulting lines, I use [System.IO.StreamWriter] to record them to Env:\TEMP headerres.txt file. The control is then returned to main script which waits for Env:\TEMP headerres.txt file (or 20s, whichever comes first).

Next step is to start the Tasks job which will collect data about running processes. As opposed to Task manager, I show background processes (ie. services) too. Worth noting is that, due to timing issues, I use Process (Win32_PerfFormattedData_PerfProc_Process) and not Thread (win32_PerfFormattedData_PerfProc_Thread) counters.
Since Windows is *thread* based (meaning a Process is just a container for Threads doing the work) this actually means scarifying some of the accuracy (for example CPU utilization data) in favour of faster and smoother execution:
#(Active) Code when using Process counter:
    Get-CimInstance Win32_PerfFormattedData_PerfProc_Process | 
        Select @{Name='_PID_'; Expression={$_.IDProcess}},
        @{Name='PPID'; Expression={$_.CreatingProcessID}},
        @{Name='PrioB'; Expression={$_.PriorityBase}},
        @{Name='Name                  '; Expression={
            (($_.Name).PadRight(22)).substring(0, [System.Math]::Min(22, ($_.Name).Length))
        }}, 
        @{Name='_CPUpt__'; Expression={($_.PercentProcessorTime).ToString("0.00").PadLeft(8)}},
        @{Name='Thds'; Expression={$_.ThreadCount}},
        @{Name='Hndl'; Expression={$_.HandleCount}},
        @{Name='WS(MB)'; Expression={[math]::Truncate($_.WorkingSet/1MB)}},
        @{Name='VM(MB)'; Expression = {[math]::Truncate($_.VirtualBytes/1MB)}},
        @{Name='PM(MB)'; Expression={[math]::Truncate($_.PageFileBytes/1MB)}} | #,
        Where { $_._PID_ -gt 0} | &$sb | 
            Select -First $procToDisp | FT * -Auto 1> $ToFile 
Note: Script-block $sb is used just for sorting the resultset depending on keyboard input.
Note: "Name=" is the same as writing "Label=". Both can be abbreviated so the expression becomes @{L=...";"E={...}}.

#(More precise but slower) Code when scanning recursively the Thread counter:
    #Get the CPU utilization percentages by summing up threads over particular process
    $pcts = Get-CimInstance win32_perfformatteddata_perfproc_thread -Property IDProcess,
      PercentProcessorTime | Group -Property IDProcess | Foreach {
        New-Object PSObject -Property @{
          PID = ($_.Group.IDProcess | Select -First 1)
          CPUpt = "{0,5:N2}" -f (($_.Group | Measure-Object -Property PercentProcessorTime -Sum).Sum)
        }
      }

    #Pair with Process data:
    Get-CimInstance Win32_PerfFormattedData_PerfProc_Process | 
      Select @{Name='_PID_'; Expression={$_.IDProcess}},
      @{Name='PPID'; Expression={$_.CreatingProcessID}},
      @{Name='PrioB'; Expression={$_.PriorityBase}},
      @{Name='Name'; Expression={($_.Name).PadRight(25)}},
      @{Name='CPUpt'; Expression={$pcts.CPUpt[[array]::IndexOf($pcts.PID, $_.IDProcess)]}},
      @{Name='Thds'; Expression={$_.ThreadCount}},
      @{Name='Hndl'; Expression={$_.HandleCount}},
      @{Name='WS(MB)'; Expression={[int]($_.WorkingSet/1MB)}},
      @{Name='VM(MB)'; Expression = {[int]($_.VirtualBytes/1MB)}},
      @{Name='PM(MB)'; Expression={[int]($_.PageFileBytes/1MB)}} | 
      Where { $_.Name -notmatch "_Total" -and $_.Name -notmatch "Idle"} | &$sb |
        Select -First $procToDisp | FT * -Auto 1> $ToFile 
Note: Script-block $sb is used just for sorting the resultset depending on keyboard input.

There is one more way of doing this and that is by expanding Process perf object. I use this approach when checking for congestion on thread level (MSDN):
#Run once:
#Header row, initialize output file:
"PID,Process,ThdID,CPU time (s),PctUser,PctPriv,State,WaitR,PrioLvL,PrioShift,IdealProc,ProcAff" |
  Out-File E:\test\thds.csv
#PIDs of interest to me:
$Processes = Get-Process | 
  Where {($_.ProcessName -match "mysql") -or ($_.ProcessName -match "ndb") -or ($_.ProcessName -match "sysben")} |
  Sort -Property ID
Note: If you check the value of $Processes variable here, you will notice something like
Id                         : 1996
...
Threads                    : {2000, 2012, 2016, 2040...}
...
meaning Threads member is actually an object and can be expanded to show more data:
PS > $Processes.Threads

BasePriority            : 8
CurrentPriority         : 9
Id                      : 1972
IdealProcessor          : 
PriorityBoostEnabled    : 
PriorityLevel           : 
PrivilegedProcessorTime : 
StartAddress            : 2006300688
StartTime               : 
ThreadState             : Wait
TotalProcessorTime      : 
UserProcessorTime       : 
WaitReason              : UserRequest
ProcessorAffinity       : 
Site                    : 
Container               : 
...
#Run following in loop, append result to file
#Threads belonging to PIDs of interest.
Foreach ($Process in $Processes) {
    $ProcessThds = $Process | Select -ExpandProperty Threads | Sort -Property ID
    Foreach ($ProcessThd in $ProcessThds) {
        $ProcName = @{L="Name";E={ $Process.ProcessName}}
        $ProcID = @{L="PID";E={ $Process.Id}}
        $ThdID = @{L="ThreadID";E={ $ProcessThd.Id }}
        $CPUTime = @{L="CPU Time (Sec)";E={ [math]::round($ProcessThd.TotalProcessorTime.TotalSeconds,2) }}
        $UsrCPUTime = @{L="User CPU Time (%)";E={ [math]::round((($ProcessThd.UserProcessorTime.ticks /
          $ProcessThd.TotalProcessorTime.ticks)*100),1) }}
        $State = @{L="State";E={ $ProcessThd.ThreadState }}
        $WR = @{L="WaitR";E={ $ProcessThd.WaitReason}}
        $PrioDelta = @{L="PrioSh";E={ $ProcessThd.CurrentPriority - $ProcessThd.BasePriority}}
        $IdProc = @{L="Ideal proc";E={ $ProcessThd.IdealProcessor}}
        $ProcAf = @{L="Proc affinity";E={ $ProcessThd.ProcessorAffinity}}
        $PrioLvL = @{L="Prio level";E={ $ProcessThd.PriorityLevel}}
        $PrivCPU = @{L="Privil CPU";E={ [math]::round((($ProcessThd.PrivilegedProcessorTime.ticks /
          $ProcessThd.TotalProcessorTime.ticks)*100),1) }}
        $ProcessThd | Select -Property  $ProcName, $ProcID, $ThdID, StartTime, $CPUTime, $UsrCPUTime, $PrivCPU, 
            $State, $WR, $PrioLvL, $PrioDelta, $IdProc, $ProcAf |
        %{'{0},{1},{2},{3},{4},{5},{6},{7},{8},{9},{10},{11}' -f $_.PID,$_.Name, $_.ThreadID,$_."CPU Time (Sec)",
        $_."User CPU Time (%)",$_."Privil CPU",$_.State,$_.WaitR,$_."Prio level",$_.PrioSh,$_."Ideal proc",
        $_."Proc affinity"} | Out-File E:\test\thds.csv -Append
    }
}
This leaves me with neat little CSV file which I then import to Excel and group by Process ID for further analysis.


Back to main script, region Main-start, where I wait for Processes job to start producing data before proceeding. If there is no data generated, the script will stop the jobs and exit.
Next is the neat trick to reduce the flicker while clearing up the screen:
[System.Console]::Clear()
and positioning the cursor at top left corner:
$saveYH = [console]::CursorTop
$saveXH = [console]::CursorLeft

Worth noting here, in terms of reduced flicker, is hiding the cursor itself:
[console]::CursorVisible = $false

After that, you enter region Main-loop which is the main code for the script. If there is fresh header data to be displayed, I move cursor to (0,0) and write it out. Otherwise, I skip this and check if I should display Core/Socket data. The problem here is that user can specify any number of cores/sockets to display data for and I display two of them in each line. Thus I need an array where user input is mapped to absolute index of the requested piece of HW in perf counter. The array is created in key-press handler. For the sake of performance, both core and socket counters were initialized at the start of the script:
#Just the individual CPUs.
$CPUdata = Get-CimInstance Win32_PerfFormattedData_Counters_ProcessorInformation | Where {$_.Name -match "^(\d{1}),(\d{1})"}
#Just the individual Sockets.
$Socketdata = Get-CimInstance Win32_PerfFormattedData_Counters_ProcessorInformation | Where {$_.Name -match "^(\d{1}),_Total"}

Then, if there is fresh data provided by Top_Processes_Job, I display it.

Next comes the keyboard handling routine. First, check that there is something to handle:
  if ($Host.UI.RawUI.KeyAvailable) {
If there is, put it into variable:
    $k = $Host.UI.RawUI.ReadKey("AllowCtrlC,IncludeKeyDown,IncludeKeyUp,NoEcho").Character
Once the keypress is processed, clear the input buffer:
    if ("p" -eq $k) {
      'CPUpt' > $conf
      $HOST.UI.RawUI.Flushinputbuffer()

"+" and "1" keys process input of CPUs/Sockets to display data for, while "-" key stops displaying that data.
Pressing "c" key will clear the screen in case it becomes garbled.
Pressing "q" key moves you to region Cleanup ending the script run.


TIPS & TRICKS

As opposed to Windows TaskManager, I show background processes too (ie. "services").

In an effort to achieve smoother display of data, I am truncating CPU/Socket info to their integer values. Also, I do not use Thread counters but rather Process ones. Due to delay while displaying the data, there will always be some discrepancy between data displayed. I.e. Total CPU utilization in Header will rarely match sum of CPU utilization by processes in table. I can live with that.

Script is started in non-normalized CPU utilization mode which means CPU utilization per process can go well over 100% on modern boxes. Let's say you have Quad core box (8 CPUs) and a process taking 50% of Core0, 60% of Core1, 30% of Core2 and 20% of Core3 then the non-normalized CPU utilization for such process would be 160% while normalized CPU utilization would be 20% (160/8). I did it as such to confirm that process actually uses more than one CPU. To toggle between non-normalized and normalized view, use "p" key.

If, for any reason, display becomes garbled, press the "c" key.

Number of processes to display is controlled by $procToDisp variable which is, atm, hard-coded to 25.

Initial sort order is defined by $procSortBy variable. Default is CPU% ($procSortBy = 'CPUpt').

IF by any chance script does not terminate normally:
- First type Get-Job
- Check that Name has "Top_Header_Job" & "Top_Processes_Job". Remember the Id (or use Name parameter).
Say Id's are 14 and 16.
- Type commands (text after # is just a comment):
[console]::CursorVisible = $true #reclaims the cursor
[console]::TreatControlCAsInput = $false #reverts CTRL+C processing to default value
receive-job -id 14
receive-job -id 16
stop-job -id 16
stop-job -id 14
remove-job -id 14
remove-job -id 16

or just exit the Powershell window.


Hope you'll find this script useful in your work!


This is all from me for this series. Next, I will start new series of blogs describing script used as testing/benchmarking framework on Windows which is also available in the package.

2015-12-14

Windows PerfCounters and Powershell - Network and Contention perf data

In previous blog, I covered DISK/IO counters. This blog will briefly touch on Network, Threading and Contention.

Other counters:


Network I/O

COUNTER: Network Interface\Bytes Total/sec
TYPE: Instantaneous
USAGE:
#Get Instances
PS > (New-Object Diagnostics.PerformanceCounterCategory("Network Interface")).GetInstanceNames("")

Intel[R] Centrino[R] Advanced-N 6205
Microsoft Virtual WiFi Miniport Adapter _2
Microsoft Virtual WiFi Miniport Adapter
Intel[R] 82579LM Gigabit Network Connection

PS > New-Object Diagnostics.PerformanceCounter("Network Interface",
"Bytes Total/sec", "Intel[R] 82579LM Gigabit Network Connection")

CategoryName     : Network Interface
CounterHelp      : Bytes Total/sec is the rate at which bytes are sent and received over each network adapter,
including framing characters. Network Interface\Bytes Total/sec is a sum of Network Interface\Bytes Received/sec
and Network Interface\Bytes Sent/sec.
CounterName      : Bytes Total/sec
CounterType      : RateOfCountsPerSecond64
InstanceLifetime : Global
InstanceName     : Intel[R] 82579LM Gigabit Network Connection
ReadOnly         : True
MachineName      : .
RawValue         : 0
Site             : 
Container        : 

PS > (New-Object Diagnostics.PerformanceCounter("Network Interface",
"Bytes Total/sec", "Intel[R] 82579LM Gigabit Network Connection")).NextValue("")
0

MEANING:This counter indicates the rate at which bytes are sent and received over each network adapter. It helps you know whether the traffic at your network adapter is saturated and if you need to add another network adapter. How quickly you can identify a problem depends on the type of network you have as well as whether you share bandwidth with other applications.
THRESHOLD:Sustained values of more than 80 percent of network bandwidth.

COUNTER: Network Interface\Bytes Received/sec
TYPE: Instantaneous
USAGE: See above.
MEANING:This counter indicates the rate at which bytes are received over each network adapter. You can calculate the rate of incoming data as a part of total bandwidth. This will help you know that you need to optimize on the incoming data from the client or that you need to add another network adapter to handle the incoming traffic.
THRESHOLD:No specific value.

COUNTER: Network Interface\Bytes Sent/sec
TYPE: Instantaneous
USAGE: See above.
MEANING:This counter indicates the rate at which bytes are sent over each network adapter. You can calculate the rate of incoming data as a part of total bandwidth. This will help you know that you need to optimize on the data being sent to the client or you need to add another network adapter to handle the outbound traffic.
THRESHOLD:No specific value.


Threading and Contention

COUNTER: .NET CLR LocksAndThreads\Contention Rate / sec
TYPE: Instantaneous
USAGE:
#Get Instances
PS > (New-Object Diagnostics.PerformanceCounterCategory(".NET CLR LocksAndThreads")).GetInstanceNames("")

_Global_
powershell_ise
PresentationFontCache
dataserv
pcee4

PS > (New-Object Diagnostics.PerformanceCounter(".NET CLR LocksAndThreads",
"Contention Rate / sec", "_Global_")).NextSample("")

RawValue         : 310
BaseValue        : 0
SystemFrequency  : 2533369
CounterFrequency : 0
CounterTimeStamp : 0
TimeStamp        : 47774751876
TimeStamp100nSec : 130927572465737721
CounterType      : RateOfCountsPerSecond32

PS > (New-Object Diagnostics.PerformanceCounter(".NET CLR LocksAndThreads",
"Contention Rate / sec", "_Global_")).NextValue("")
0

PS > (New-Object Diagnostics.PerformanceCounter(".NET CLR LocksAndThreads",
"Contention Rate / sec", "powershell_ise")).NextValue("")
0

MEANING:This counter displays the rate at which the runtime attempts to acquire a managed lock but without a success. Sustained non-zero values may be a cause of concern. You may want to run dedicated tests for a particular piece of code to identify the contention rate for the particular code path.
THRESHOLD:No specific value.

COUNTER: .NET CLR LocksAndThreads\Current Queue Length
TYPE: Instantaneous
USAGE: See above.
MEANING:This counter displays the last recorded number of threads currently waiting to acquire a *managed* lock in an application. You may want to run dedicated tests for a particular piece of code to identify the average queue length for the particular code path. This helps you identify inefficient synchronization mechanisms.
THRESHOLD:No specific value.

COUNTER: Thread\% Processor Time
TYPE: Instantaneous
USAGE:
PS > Get-CimInstance win32_perfformatteddata_perfproc_thread | Select IDProcess, PercentProcessorTime |
Sort PercentProcessorTime -Descending | Group -Property PercentProcessorTime |
Select -ExpandProperty Group | Select -First 5

IDProcess                           PercentProcessorTime
---------                           --------------------
        0                                            100
     3820                                             96
        0                                             84
        0                                             84
        0                                             46
MEANING:This counter gives you the idea as to which thread is actually taking the maximum processor time. If you see idle CPU and low throughput, threads could be waiting or deadlocked. You can take a stack dump of the process and compare the thread IDs from test data with the dump information to identify threads that are waiting or blocked. Or examine Thread State and Thread Wait Reason counters.
THRESHOLD:No specific value.


PS > Get-CimInstance win32_perfformatteddata_perfproc_thread | Select -First 1 | FL *
Caption               : 
Description           : 
Name                  : Idle/0
Frequency_Object      : 
Frequency_PerfTime    : 
Frequency_Sys100NS    : 
Timestamp_Object      : 
Timestamp_PerfTime    : 
Timestamp_Sys100NS    : 
ContextSwitchesPersec : 0
ElapsedTime           : 13092759167
IDProcess             : 0
IDThread              : 0
PercentPrivilegedTime : 0
PercentProcessorTime  : 0
PercentUserTime       : 0
PriorityBase          : 0
PriorityCurrent       : 0
StartAddress          : 59492592
ThreadState           : 2
ThreadWaitReason      : 0
PSComputerName        : 
CimClass              : root/cimv2:Win32_PerfFormattedData_PerfProc_Thread
CimInstanceProperties : {Caption, Description, Name, Frequency_Object...}
CimSystemProperties   : Microsoft.Management.Infrastructure.CimSystemProperties


PS > (New-Object Diagnostics.PerformanceCounterCategory("Thread")).GetCounters("") | 
  Select CounterName | Sort CounterName

CounterName
-----------
% Privileged Time
% Processor Time
% User Time
Context Switches/sec
Elapsed Time
ID Process
ID Thread
Priority Base
Priority Current
Start Address
ThreadState
Thread Wait Reason

PS > (New-Object Diagnostics.PerformanceCounterCategory(".NET CLR LocksAndThreads")).GetCounters("") |
  Select CounterName | Sort CounterName

CounterName
-----------
# of current logical Threads
# of current physical Threads
# of current recognized threads
# of total recognized threads
Contention Rate / sec
Current Queue Length
Queue Length / sec
Queue Length Peak
rate of recognized threads / sec
Total # of Contentions


Next blog will be the last in the Windows PerfCounters series where I will put all of this to work writing Top script for Windows.

In this series:
BLOG 1: PerfCounters infrastructure
BLOG 2: PerfCounters Raw vs. Formatted values
BLOG 3: PerfCounters, fetching the values
BLOG 4: PerfCounters, CPU perf data
BLOG 5: PerfCounters, Memory perf data
BLOG 6: PerfCounters, Disk/IO perf data
BLOG 7: PerfCounters, Network and Contention perf data

2015-12-07

Windows PerfCounters and Powershell - Disk & IO perf data

This post is the hardest for me to write as I generally pay little attention to disks. When they prove too slow, I replace them with faster ones. So now I am writing this on laptop with two SSDs. That said, Disk subsystem could be a major system performance bottleneck and thus there are numerous counters covering this area (Get-CimClass *disk* | Select CimClassName). I would also like to turn your attention to old yet excellent article Top Six FAQs on Windows 2000 Disk Performance if you're interested in subject.

Disk counters:

Note: Microsoft recommends that "when attempting to analyse disk performance bottlenecks, you should always use physical disk counters. However, if you use software RAID, you should use logical disk counters. As for Logical Disk and Physical Disk Counters, the same values are available in each of these counter objects. Logical disk data is tracked by the volume manager(s), and physical disk data is tracked by the partition manager."

The one I look into the most is Disk Queue Length which comes in two flavours; Average and Current.
COUNTER: Win32_PerfFormattedData_PerfDisk_PhysicalDisk\AvgDiskQueueLength (AvgDiskReadQueueLength)
TYPE: Sample, Instance
USAGE:
PS > Get-CimInstance Win32_PerfFormattedData_PerfDisk_PhysicalDisk | Where {$_.Name -eq '_Total'} |
 Select AvgDiskQueueLength, CurrentDiskQueueLength | FL

AvgDiskQueueLength     : 0
CurrentDiskQueueLength : 0
MEANING: Average number of both read and write requests that were queued and waiting for the selected disk during the sample interval as well as requests in service. Since I used "_Total" instance, this means I need to divide the value with number of physical disks on the system. PerfMon shows this value per logical disk.
PS > Get-CimInstance Win32_PerfFormattedData_PerfDisk_LogicalDisk |
 Select Name, AvgDiskQueueLength, CurrentDiskQueueLength | FL

Name                   : HarddiskVolume1 #Boot image on Physical disk 1
AvgDiskQueueLength     : 0
CurrentDiskQueueLength : 0

Name                   : C: #Boot partition on Physical disk 1
AvgDiskQueueLength     : 0
CurrentDiskQueueLength : 0

Name                   : D: #Partition on Physical disk 1
AvgDiskQueueLength     : 0
CurrentDiskQueueLength : 0

Name                   : E: #Partition on Physical disk 2
AvgDiskQueueLength     : 0
CurrentDiskQueueLength : 0

Name                   : G: #Partition on Physical disk 2
AvgDiskQueueLength     : 0
CurrentDiskQueueLength : 0

Name                   : _Total
AvgDiskQueueLength     : 0
CurrentDiskQueueLength : 0 
GOTCHA: Since both "pending" and "in service" requests are counted, this counter might overstate the activity.
THRESHOLD: If more than 2 requests are continuously waiting on a single disk, the disk might be a bottleneck. To analyse queue length data further, use it's components; AvgDiskReadQueueLength and AvgDiskWriteQueueLength.

COUNTER: Win32_PerfFormattedData_PerfDisk_PhysicalDisk\CurrentDiskQueueLength
TYPE: Instantaneous, Instance
USAGE:
PS > Get-CimInstance Win32_PerfFormattedData_PerfDisk_PhysicalDisk | Where {$_.Name -eq '_Total'} |
 Select AvgDiskQueueLength, CurrentDiskQueueLength | FL

AvgDiskQueueLength     : 0
CurrentDiskQueueLength : 0
MEANING: Number of requests outstanding on the disk at the time the performance data is collected. It includes requests being serviced at the time of data collection. The value represents an instantaneous length, not an average over a time interval. Multispindle disk devices can have multiple requests active at one time, but other concurrent requests await service. This property may reflect a transitory high or low queue length. If the disk drive has a sustained load, the value will be consistently high. Requests experience delays proportional to the length of the queue minus the number of spindles on the disks. This difference should average less than two for good performance.
GOTCHA:
THRESHOLD: 2 requests in queue for prolonged period of time for single disk (spindle).

Inner workings of measurement collection:

Values are mostly derived by the diskperf filter driver that provides disk performance statistics. Diskperf is a layer of software sitting in the disk driver stack. As I/O Request packets (IRPs) pass through this layer, diskperf keeps track of the time I/O's start and the time they finish. On the way to the device, diskperf records a timestamp for the IRP. On the way back from the device, the completion time is recorded. The difference is the duration of the I/O request. Averaged over the collection interval, this becomes the Avg. Disk sec/Transfer, a direct measure of disk response time from the point of view of the device driver. Diskperf also maintains byte counts and separate counters for reads and writes, at both the Logical and Physical disk level allowing Avg. Disk sec/Transfer to be broken out into reads and writes. This layer does add to latency but not significantly (up to 5%). Now that we know the mechanics, back to PhysicalDisk\Avg. Disk Queue Length and why we gather both queued and in-service requests in a bunch.
So, AvgDiskQueueLength counter is useful for gathering concurrency data, including data bursts and peak loads. These values represent the number of requests in flight below the driver taking the statistics. This means the requests are not necessarily queued but could actually be in service or completed and on the way back up the path. Possible in-flight locations include the following:
  • SCSIport or Storport queue
  • OEM driver queue
  • Disk controller queue
  • Hard disk queue
  • Actively receiving from a hard disk

Brief account of some other counters:

COUNTER: PhysicalDisk\Disk Writes/sec
MEANING: This counter indicates the rate of write operations on the disk.
THRESHOLD: Depends on manufacturer’s specifications.

COUNTER: PhysicalDisk\Split IO/sec
MEANING: Reports the rate at which the operating system divides I/O requests to the disk into multiple requests. A split I/O request might occur if the program requests data in a size that is too large to fit into a single request or if the disk is fragmented. Factors that influence the size of an I/O request can include application design, the file system, or drivers. A high rate of split I/O might not, in itself, represent a problem. However, on single-disk systems, a high rate for this counter tends to indicate disk fragmentation.
More info in MSDN.

Disk and Memory:

Because memory is cached to disk as physical memory becomes limited, make sure that you have a sufficient amount of memory available. When memory is scarce, more pages are written to disk, resulting in increased disk activity. Also, make sure to set the paging file to an appropriate size. Additional disk memory cache will help offset peaks in disk I/O requests. However, it should be noted that a large disk memory cache seldom solves the problem of not having enough spindles, and having enough spindles can negate the need for a large disk memory cache.

COUNTER: PhysicalDisk\Avg. Disk sec/Transfer
MEANING: This counter indicates the time, in seconds, of the average disk transfer. This may indicate a large amount of disk fragmentation, slow disks, or disk failures.
GOTCHA: Multiply the values of the Physical Disk\Avg. Disk sec/Transfer and Memory\Pages/sec counters. If the product of these counters exceeds 0.1, paging is taking more than 10% of disk access time, so you need more physical memory available.
THRESHOLD: Should not be more than 18 milliseconds.

COUNTER: Memory\Pages/sec
MEANING: This counter indicates the rate at which pages are read from or written to disk to resolve hard page faults. Multiply the values of the Physical Disk\Avg. Disk sec/Transfer and Memory\Pages/sec performance counters. If the product of these values exceeds 0.1, paging is utilizing more than 10 percent of disk access time, which indicates that insufficient physical memory is available.
GOTCHA: A high value for the performance counter could indicate excessive paging which will increase disk I/0. If this occurs, consider adding physical memory to reduce disk I/O and increase performance.
THRESHOLD: A sustained value of more than 5 indicates a bottleneck.


Next I will talk briefly of other counter categories such as Network and Processes.

In this series:
BLOG 1: PerfCounters infrastructure
BLOG 2: PerfCounters Raw vs. Formatted values
BLOG 3: PerfCounters, fetching the values
BLOG 4: PerfCounters, CPU perf data
BLOG 5: PerfCounters, Memory perf data
BLOG 6: PerfCounters, Disk/IO perf data
BLOG 7: PerfCounters, Network and Contention perf data

2015-11-30

Windows PerfCounters and Powershell - Memory perf data

In the last blog I spoke of CPU counters. Now, I'll talk of Memory counters.

MEMORY Counters (CIM_PhysicalMemory class, Win32_PerfFormattedData_PerfOS_Memory class, Memory Performance Information ...):

Note: I introduced the notion of samples and how to fetch them using NextValue() so I will occasionally omit $var.NextValue() going forward.

Let me note here that if you thought previously described performance classes were complicated, you are now entering the realm of black magic ;-) There is a good series of blogs on subject of Memory by Mark Russinovich worth reading although quite old.

Memory is a key resource for any machine so I will look at the most of the values available on Windows. In Resource monitor, Memory tab, you find a bar with Hardware reserved, In use, Modified, Standby and Free values. There are also Available, Cached, Total and Installed values. Let's start with the biggest number, Installed RAM.

In-depth description of Memory Counters important for my use-case:

COUNTER: cim_physicalmemory\Capacity
TYPE: Instantaneous
USAGE: (Get-Ciminstance -class "cim_physicalmemory" | Measure-Object Capacity -Sum).Sum / 1024 / 1024 #MB
MEANING: Total capacity of the physical memory, in bytes. Refers to "Installed".
GOTCHA: You will find tips to use TotalPhysicalMemory but, according to MSDN, it's been deprecated. Also, that page recommends using TotalVisualMemorySize property in the CIM_OperatingSystem class instead but this is wrong as there is no TotalVisualMemorySize property and, even if there was, we need installed memory size.
THRESHOLD:

Intermediate step; how much of the installed memory is available to OS:
COUNTER: win32_operatingsystem\TotalVisibleMemorySize
TYPE: Instantaneous
USAGE: [math]::Round((Get-CimInstance win32_operatingsystem).TotalVisibleMemorySize / 1024,2)
MEANING: Total amount of RAM available to OS. Refers to "Total".
GOTCHA:
THRESHOLD:

Subtracting TotalVisibleMemorySize from Capacity gives us HW reserved RAM, i.e. RAM taken by various HW such as video card. Check this post for details.
COUNTER: HW reserved
TYPE: Calculated
USAGE: cim_physicalmemory\Capacity (Installed) - win32_operatingsystem\TotalVisibleMemorySize (Total)
MEANING: Size of RAM not available to OS although installed on the system. Refers to "Hardware reserved".
GOTCHA: Depends on HW and BIOS settings, not something "fixable" in Windows.
THRESHOLD:

COUNTER: win32_operatingsystem\FreePhysicalMemory (Bytes), Memory\Available MBytes
TYPE: Instantaneous
USAGE:
(Get-WmiObject win32_operatingsystem).FreePhysicalMemory
$Memory_AvailMB = New-Object Diagnostics.PerformanceCounter("Memory", "Available MBytes")
(New-Object Diagnostics.PerformanceCounter("Memory", "Available MBytes")).RawValue

MEANING: Total amount of RAM available to processes. Equal to the sum of memory assigned to the standby (cached), free and zero page lists. Refers to "Available".
GOTCHA:
THRESHOLD: A consistent value of less than 20% of installed RAM. In such situations, consult additional counters, such as Win32_PerfFormattedData_PerfOS_Memory\PagesPerSec to determine if System memory is adequate for the workload.

COUNTER: In use memory
TYPE: Calculated
USAGE: win32_operatingsystem\TotalVisibleMemorySize (Total) - Memory\Available MBytes (Available)
MEANING: Amount of RAM in use by processes running on the box.
GOTCHA:
THRESHOLD:

COUNTER: Memory\Modified Page List Bytes (Win32_PerfFormattedData_PerfOS_Memory)
TYPE: Instantaneous
USAGE: $Memory_ModPLBy = New-Object System.Diagnostics.PerformanceCounter("Memory", "Modified Page List Bytes")
MEANING: The amount of RAM taken by the pages previously belonging to a working set but removed. However, the pages were modified while in use and their current contents haven’t yet been written to storage. The Page Table Entry still refers to the physical page(s) but is marked invalid and in transition. It must be written to the backing store before the physical page can be reused.
GOTCHA: No description in MSDN!?
THRESHOLD: Keep as low as possible.

COUNTER: Win32_PerfFormattedData_PerfOS_Memory\FreeAndZeroPageListBytes
TYPE: Instantaneous
USAGE: (get-wmiobject -computername localhost -Namespace root\CIMV2 -Query "Select * from Win32_PerfFormattedData_PerfOS_Memory").FreeAndZeroPageListBytes / 1024 / 1024 #MB
MEANING: The amount of physical memory, in bytes, that is assigned to the free and zero page lists thus immediately available for allocation to a process or for system use since it does not contain any data. Refers to "Free".
GOTCHA: There is a big difference between Free and Available memory. This is due to most of the pages considered available being in some sort of transition state (i.e. waiting to be written to disk) or have not yet met all of the OS requirements (i.e. page is not considered secure until it's zeroed out).
THRESHOLD: Keep as high as possible.

COUNTER: Standby
TYPE: Calculated
USAGE:
$Memory_SBCCBy = New-Object Diagnostics.PerformanceCounter("Memory", "Standby Cache Core Bytes")
$Memory_SBCNPBy = New-Object Diagnostics.PerformanceCounter("Memory", "Standby Cache Normal Priority Bytes")
$Memory_SBCRBy = New-Object Diagnostics.PerformanceCounter("Memory", "Standby Cache Reserve Bytes")
[math]::Round($Memory_SBCCBy.NextValue()/1024/1024 + $Memory_SBCNPBy.NextValue()/1024/1024+$Memory_SBCRBy.NextValue()/1024/1024,2)

MEANING: The amount of RAM in pages previously belonging to a working set but removed (or marshaled directly into the standby list). The pages weren’t modified since last written to disk. The Page Table Entry still refers to the physical pages but are marked invalid and in transition. Or, simpler explanation, memory that has been removed from a process's working set (its physical memory) en route to disk but is still available to be recalled.
GOTCHA: Please see the explanation of the factors in Win32_PerfFormattedData_PerfOS_Memory or Memory Object MSDN pages.
THRESHOLD:

COUNTER: Cached
TYPE: Calculated
USAGE:
MEANING: This number represents the sum of the system working set, standby list and modified page list. So, Memory\Cache Bytes, Memory\Modified Page List Bytes, Memory\Standby Cache Core Bytes, Memory\Standby Cache Normal Priority Bytes and Memory\Standby Cache Reserve Bytes. In this case, Memory\Cache Bytes + Memory\Modified Page List Bytes + Standby.
GOTCHA: Presented here for the sake of completeness.
THRESHOLD:

More counters of significance:

Win32_PerfFormattedData_PerfOS_Memory\CacheBytes - Number of bytes currently being used by the file system cache. The file system cache is an area of physical memory that stores recently used pages of data for applications. The operating system continually adjusts the size of the cache, making it as large as it can while still preserving the minimum required number of available bytes for processes. This property displays the last observed value only; it is not an average. See also SystemCacheResidentBytes and relatives.
Simpler explanation would be that the memory pages that the System uses are counted in two main counters, Cache Bytes and Pool Nonpaged Bytes. The Cache Bytes counter value is the amount of resident pages allocated in RAM that the Kernel threads can address without causing a Page Fault. This counter includes the Pool Paged Resident Bytes, the System Cache Resident Bytes, the System Code Resident Bytes and the System Driver Resident Bytes.

Note: If Memory\Pool Nonpaged Bytes value is 10% or more higher than its value at system startup, there is probably a leak.

Win32_PerfFormattedData_PerfOS_Memory\CacheFaultsPerSec - Number of faults which occur when a page is not found in the file system cache and must be retrieved from elsewhere in memory (a soft fault) or from disk (a hard fault). The file system cache is an area of physical memory that stores recently used pages of data for applications. Cache activity is a reliable indicator of most application I/O operations. This property counts the number of faults without regard for the number of pages faulted in each operation.

There is a whole set of Paging counters and they do require our attention since we can deduce Memory shortages on Windows by using them. Some of the key counters I will describe below. Dealing with Windows Paging you have to keep in mind that paging occurs for various operations within OS and excessive paging doesn’t automatically indicate a memory shortage. For instance, many network card drivers utilize the Pagefile (sometimes excessively) and this can be misread as a memory shortage.

Win32_PerfFormattedData_PerfOS_Memory\PagesPerSec (and relatives) - A sustained value of over 20 should be closely monitored and a System with a sustained value of over 50 is probably lacking in System Memory. Again, it is normal for this value to spike occasionally, especially if the other Memory counters do not show a lack of System Memory.

COUNTER: Pages Input per second / Page Reads per second
TYPE: Calculated
USAGE:
$Memory_PIps = New-Object Diagnostics.PerformanceCounter("Memory", "Pages Input/sec")
$Memory_PRps = New-Object Diagnostics.PerformanceCounter("Memory", "Page Reads/sec")
[math]::Round ($Memory_PIps.NextValue() / $Memory_PRps.NextValue(),2)

MEANING: The average of Memory\Pages Input/sec divided by average of Memory\Page Reads/sec gives the number of pages per disk read. This value should not generally exceed five pages per second. A value greater than five indicates that the system is spending too much time paging and requires more memory (assuming that the application has been optimized).
GOTCHA:
THRESHOLD: Sustained value of 5 or more.

Some other interesting counters I will not be covering in detail:

Memory\Page Reads/sec
Memory\Page Writes/sec
Paging File(_total)\% Usage
and so on.

In the next blog I will cover Disk counters.

In this series:
BLOG 1: PerfCounters infrastructure
BLOG 2: PerfCounters Raw vs. Formatted values
BLOG 3: PerfCounters, fetching the values
BLOG 4: PerfCounters, CPU perf data
BLOG 5: PerfCounters, Memory perf data
BLOG 6: PerfCounters, Disk/IO perf data
BLOG 7: PerfCounters, Network and Contention perf data

2015-11-23

Windows PerfCounters and Powershell - CPU perf data

So far, I talked of WMI, CIM, WQL, System.Diagnostics.PerformanceCounterCategory, perf-counter data organization and flavour. Now it's time to look at some performance counters I deem important for my use-case more closely.
Note: List of available Counters for Get-Counter command
Get-Counter -ListSet * | Sort-Object CounterSetName | Format-Table CounterSetName

Basic concepts:

I will introduce basic concepts of Processor, Core and CPU now to help you follow the text. Let us use this convention:
  • "Processor" is a piece of hardware you connect to a slot on the motherboard.
  • "Physical Core" is a physical computing unit built into the "Processor".
  • "Virtual Core" is a virtual computing unit built on top of "Physical Core" (i.e. HT is ON).
  • "CPU" is a computing unit inside the "Processor", either physical or virtual.


Putting concepts to work

Now lets calculate number of CPUs for my laptop:
PS > ((Get-CimInstance -Namespace root/CIMV2 -ClassName CIM_Processor).NumberOfLogicalProcessors | Measure-Object -Sum).Sum

4

Note: Many other counters fail for some HW configuration and/or OS! Be sure to check.
Note: HT is ON on my dual-core laptop and no cores are parked so to get number of Physical cores:
PS > ((Get-CimInstance -Namespace root/CIMV2 -ClassName CIM_Processor).NumberOfCores | Measure-Object -Sum).Sum

2

Note: There are many ways to collect this info:
PS > (Get-CimInstance Win32_ComputerSystem).NumberOfLogicalProcessors
PS > ((New-Object Diagnostics.PerformanceCounterCategory("Processor Information")).GetInstanceNames() | ?{$_ -match "^(\d{1}),(\d{1})"} | Measure-Object -Sum).Count
Note: RegEx expression is matching "Number,Number" Instances only (See previous blog about instances).

It is not obvious when working with 1 NUMA node/Slot, but the -Sum might refer to Sum of CPUs per Slot, depending on RegEx.

Before starting on Counters, let me stress that the measurements at the system, process and thread level in Windows are based on a sampling methodology thus the data gathered is subject to typical sampling errors like:
  • accumulating a "sufficient" number of sample observations to be able to make a reliable statistical inference, i.e. the sampling size
and
  • ensuring that there aren’t systemic sources of sampling error that causes results to be under or over-sampled as I will demonstrate shortly.

As of W2K8, the trends are changing towards event driven measurement for CPU utilization which, although more sane and accurate, poses its own set of challenges (say, a clock drift across multiprocessor cores when they are not resynchronized periodically and so on). To compensate for drift, new PerfMon/ResMon work by measuring CPU load in real time using event oriented measurement data gathered by the OS Scheduler each time a context switch occurs.
A context switch occurs in Windows whenever the processor switches its execution context to run a different thread (see more below). Context switches also occur as a result of high priority Interrupt Service Routines (ISRs) as well as the Deferred Procedure Calls (DPCs) that ISRs schedule to complete the interrupt processing. Starting in Windows 6 (Vista/2008), the OS Scheduler began issuing RDTSC instructions to get the internal processor clock each time a context switch occurs. I will talk of context switching and DPC counters in a short while. For more details please see this excellent blog post.

System CPU counters:

First counter I want to talk about is Processor Queue Length. Immediately a Linux users observes that there is no "System load" counter on Windows. This is because Windows OS is Thread based as opposed to Linux which is Process based. This simply means that, in Windows, an execution thread is a basic unit of execution (thus basis for collecting usage statistics too) and a process acts as a container for threads. As simple as it may seem, this actually poses a lot of challenges since one has to start aggregating data about running processes from Threads counters and work his way up. I will talk about this in detail in final blog. So, the WMI counter mimicking Linux "System load" best is, IMO, Processor Queue Length:
PS > Get-Counter '\System\Processor Queue Length'

Timestamp                 CounterSamples                                                      
---------                 --------------                                                      
23.10.15. 10:34:10        \\server_name\system\processor queue length : 1                                          
However, this is slooooow (although subsequent calls return much faster):
PS > Measure-Command { Get-Counter '\System\Processor Queue Length' }

TotalSeconds      : 4.2961321

PS > Measure-Command { Get-Counter '\System\Processor Queue Length' }

TotalSeconds      : 1.007445
So, as described in previous blog, I use System.Diagnostics class to fetch this value:
PS > Measure-Command { New-Object Diagnostics.PerformanceCounter("System", "Processor Queue Length")}

TotalSeconds      : 2.0006457

PS > Measure-Command { New-Object Diagnostics.PerformanceCounter("System", "Processor Queue Length")}

TotalSeconds      : 0.000643
Now, put this into a variable and simply call NextValue():
PS > $System_ProcQL = New-Object Diagnostics.PerformanceCounter("System", "Processor Queue Length")
PS > $System_ProcQL.NextValue()
0
PS > $System_ProcQL.NextValue()
10
The value obtained is for all of the CPU's so you need to calculate the number of CPU's to be your divider and obtain the real value:
$SystemLoad = $System_ProcQL.NextValue() / $totCPU


In-depth description of System Counters important for my use-case:

COUNTER: System\Processor Queue Length
TYPE: Instantaneous
USAGE: New-Object Diagnostics.PerformanceCounter("System", "Processor Queue Length") / ((Get-CimInstance -Namespace root/CIMV2 -ClassName CIM_Processor).NumberOfLogicalProcessors | Measure-Object -Sum).Sum
MEANING: Number of threads per CPU that are ready for execution but can't get CPU cycles for whatever reason thus waiting in OS Scheduler queue. Since Windows have one Scheduler queue, I divide this value with total number of computation units (i.e. CPUs). The actual mechanics is that when Counter value is requested a measurement function traverses the Scheduler Ready Queue and counts the number of threads waiting for an available CPU.
GOTCHA: Even on idle system there can be significant number of threads running on schedule that can bump this number very high. Say you have 4 CPU box and processes fetching values for 100 counters, 10 samples every 1 second. All of these sample requests will lay sleeping for 1 second (thus the Processor Queue Length value will be low) and then all will wake up at the same timer event (clock interrupt) causing Processor Queue Length to spike although there is no real load on the system. It's even worse if your thread(s) is of high priority as it will get executed sooner than the user threads thus pushing Processor Queue Length number very very high. This leads to disproportionate number of Ready Threads waiting for cycles, even (or especially) when the processor itself is not very busy overall. So tip 1 would be to check if CPUs are really busy or not.
THRESHOLD: Pending on above, it is hard to tell what the threshold value is but most people seem to agree it's "sustained value of 2 or more" with CPU utilization of 85%+. This combination tells us we can benefit from adding more CPUs.

COUNTER: System\Context Switches/sec
TYPE: Instantaneous
USAGE:
$System_CSpS = New-Object Diagnostics.PerformanceCounter("System", "Context Switches/sec")
$System_CSpS.NextValue()

MEANING: Context switching happens when a higher priority thread pre-empts a lower priority thread that is currently running or when a high priority thread blocks. High levels of context switching can occur when many threads share the same priority level. This often indicates that there are too many threads competing for the processors on the system. If you do not see much processor utilization and you see very low levels of context switching, it could indicate that threads are blocked (link).
GOTCHA: The number obtained is system-wide! To report the total number of context switches generated per second by all threads use the Thread(_Total)\Context Switches/sec counter (Category((Instance)\Counter):
New-Object Diagnostics.PerformanceCounter("Thread", "Context Switches/sec", "_Total")
THRESHOLD: Context switching rates in excess of 15,000 per second per CPU. The remedy would be to reduce the number of threads and queue more at the application level. This will cause less context switching, and less context switching is good for reducing CPU load.


In-depth description of CPU Counters important for my use-case:

Note: "Processor Information" category, besides overall _Total, has instances for Slot/NUMA node (0,_Total, n,_Total) while "Processor" category gives just _Total for all CPUs as defined above.
Gotcha: On single slot machines, "Processor" category will give info for all the CPUs while on machines with multiple slots, it will give info on just the Physical cores :-/
Thus, if InstanceName is _Total, both yield the same value.

COUNTER: Processor Information(_Total)\% Processor Time, Processor(_Total)\% Processor Time
TYPE: Sample, Instance
USAGE:
$InstanceName = "_Total"
$PI_PT = New-Object Diagnostics.PerformanceCounter("Processor Information", "% Processor Time")
$PI_PT.InstanceName = $InstanceName
$null = $PI_PT.NextValue()
--or--
Get-Counter -Counter "\Processor Information(_Total)\% Processor Time"
Get-Counter -Counter "\Processor(_Total)\% Processor Time"

MEANING: Primary indicator of CPU activity. High values many not necessarily be bad. However, if the other processor-related counters are increasing linearly such as Processor\% Privileged Time or System\Processor Queue Length, high CPU utilization may be worth investigating.
GOTCHA: If this counter is around threshold value, starting new processes will only lead to increased value of Processor Queue Length but the work done will remain the same. Look for some more counters that I'm about to describe in relation to this one.
THRESHOLD: Folks seem to agree on ~85%. Low CPU utilization with sustained Processor Queue Length value of 2 or higher is indicator that requests for CPU time arrive randomly and threads demand irregular amounts of time from the CPU. This means that the processor power is not a bottleneck but that the application threading logic should be improved.

COUNTER: Processor Information(_Total)\% Privileged Time
TYPE: Sample, Instance
USAGE:
$InstanceName = "_Total"
$PI_PPT = New-Object Diagnostics.PerformanceCounter("Processor Information", "% Privileged Time")
$PI_PPT.InstanceName = $InstanceName
$null = $PI_PPT.NextValue()

MEANING: Counter indicates the percentage of non-idle CPU time spent in privileged mode, i.e. calls to OS functions (file or network I/O, memory allocation...). Basically, this is unrestricted mode allowing direct access to hardware and all memory.
GOTCHA:
THRESHOLD: Folks seem to agree on consistently being over 75%.

COUNTER: Processor Information(_Total)\% User Time
TYPE: Sample, Instance
USAGE:
$InstanceName = "_Total"
$PI_PUT = New-Object Diagnostics.PerformanceCounter("Processor Information", "% User Time")
$PI_PUT.InstanceName = $InstanceName
$null = $PI_PUT.NextValue()

MEANING: Percentage of non-idle CPU time spent in user mode. User mode is a restricted processing mode designed for applications, environment subsystems, and integral subsystems.
GOTCHA: Processor Information(_Total)\% Privileged Time +
Processor Information(_Total)\% User Time = Processor Information(_Total)\% Processor Time.
THRESHOLD: Depends on previous two counters.

COUNTER: Processor Information(_Total)\% Idle Time
TYPE: Sample, Instance
USAGE:
$InstanceName = "_Total"
$PI_PIT = New-Object Diagnostics.PerformanceCounter("Processor Information", "% Idle Time")
$PI_PIT.InstanceName = $InstanceName
$null = $PI_PIT.NextValue()

MEANING: Counter indicates the percentage of time OS idle thread was consuming cycles. On Windows, there is a special Kernel thread that consumes cycles when CPU is idling. Counting cycles consumed by this thread gives Idle CPU time.
GOTCHA: Processor Information(_Total)\% Processor Time + Processor Information(_Total)\% Idle Time = 100%
THRESHOLD:

COUNTER: Processor Information(_Total)\% Priority Time
TYPE: Sample, Instance
USAGE:
$InstanceName = "_Total"
$PI_PPRIOT = New-Object Diagnostics.PerformanceCounter("Processor Information", "% Priority Time")
$PI_PPRIOT.InstanceName = $InstanceName
$null = $PI_PPRIOT.NextValue()

MEANING: CPU utilization by high priority threads.
GOTCHA: Kernel scheduler can, on occasion, wake up low priority threads sleeping for "long" time assigning them much more slices on CPU than one would expect given the (low)priority. This, in turn, blocks high-priority threads from execution which is never an expected behaviour. I would look at this value in relation to Context switches/second to determine what's going on.
THRESHOLD:

COUNTER: Processor Information\Interrupts/sec
TYPE: Sample, Instance
USAGE:
$InstanceName = "_Total"
$PI_INTPS = New-Object Diagnostics.PerformanceCounter("Processor Information", "Interrupts/sec")
$PI_INTPS.InstanceName = $InstanceName
$null = $PI_INTPS.NextValue()

MEANING: Number of hardware interrupts per second. This value is the indicator of the activity of devices that generate interrupts, such as network adapters.
GOTCHA: See next counter.
THRESHOLD:

COUNTER: Processor Information\% Interrupt Time
TYPE: Sample, Instance
USAGE:
$InstanceName = "_Total"
$PI_PINTT = New-Object Diagnostics.PerformanceCounter("Processor Information", "% Interrupt Time")
$PI_PINTT.InstanceName = $InstanceName
$null = $PI_PINTT.NextValue()

MEANING: The value indicates the percentage of time CPUs spend receiving and servicing hardware interrupts. This value is an indirect indicator of the activity of devices that generate interrupts, such as network adapters.
GOTCHA: Mass increase in Processor Information\Interrupts/sec and Processor Information\% Interrupt Time indicates potential hardware problems.
THRESHOLD:

COUNTER: Processor Information\DPCs Queued/sec
TYPE: Sample, Instance
USAGE:
$InstanceName = "_Total"
$PI_DPCQPS = New-Object Diagnostics.PerformanceCounter("Processor Information", "DPCs Queued/sec")
$PI_DPCQPS.InstanceName = $InstanceName
$null = $PI_DPCQPS.NextValue()

MEANING: Overall rate at which deferred procedure calls ("SW interrupts") are added to the processor's DPC queue. This property measures the rate at which DPCs are added to the queue, not the number of DPCs in the queue.
GOTCHA: This is NOT the number of SW interrupts in the queue!
THRESHOLD:

COUNTER: Processor Information\DPC Time
TYPE: Sample, Instance
USAGE:
$InstanceName = "_Total"
$PI_PDPCT = New-Object Diagnostics.PerformanceCounter("Processor Information", "% DPC Time")
$PI_PDPCT.InstanceName = $InstanceName
$null = $PI_PDPCT.NextValue()

MEANING: Percentage of time that the processor spent receiving and servicing deferred procedure calls (SW interrupts) during the sample interval. They are counted separately and are not a component of the interrupt counters.
GOTCHA: This property is a component of PercentPrivilegedTime because DPCs are executed in privileged mode.
THRESHOLD:

Other useful counters I would look into in case of trouble are C1/C2/C3TransitionsPerSec. There is a huge penalty waking up CPU from C3 low power state to C2 low power state and considerable penalty transitioning from C2 to C1. So if box is choking and CPUs are idling, look here. And make sure ParkingStatus for each CPU is 0 ;-)
Example: (Physical) CPU 9 in Slot 7 was asleep:
PS > GCim Win32_PerfFormattedData_Counters_ProcessorInformation
...
Name                        : 7,9
AverageIdleTime             : 100
C3TransitionsPersec         : 64
ClockInterruptsPersec       : 64
IdleBreakEventsPersec       : 64
InterruptsPersec            : 64
PercentC3Time               : 99
...
Basically, only processing timer events.

There are also combinations of counters that can point out problems like Processor\% DPC Time, % Interrupt Time and % Privileged Time. If Interrupt Time and DPC time are a large portion of Privileged Time, the kernel is spending significant amount of time processing (most likely) I/O requests. In some cases performance can be improved by configuring interrupts and DPC affinity to a small number of CPUs on a multiprocessor system, which improves cache locality. In other cases, it works best to distribute the interrupts and DPCs among many CPUs, so as to keep the interrupt and DPC activity from becoming a bottleneck.

In the next blog I will cover Memory performance counters.

In this series:
BLOG 1: PerfCounters infrastructure
BLOG 2: PerfCounters Raw vs. Formatted values
BLOG 3: PerfCounters, fetching the values
BLOG 4: PerfCounters, CPU perf data
BLOG 5: PerfCounters, Memory perf data
BLOG 6: PerfCounters, Disk/IO perf data
BLOG 7: PerfCounters, Network and Contention perf data