Showing posts with label CPU counters. Show all posts
Showing posts with label CPU counters. Show all posts

2016-02-29

Linux top command on Windows, further investigations

In my previous post, I spoke of "Normalized" and "Non-Normalized" CPU utilization values:
--

TABLE:

Foreword: Windows is not "process" based OS (like Linux) but rather "thread" based so all of the numbers relating to CPU usage are approximations. I did made a "proper" CPU per Process looping and summing up Threads counter (https://msdn.microsoft.com/en-us/library/aa394279%28v=vs.85%29.aspx) based on PID but that proved too slow given I have ~1 sec to deal with everything. CPU utilization using RAW counters with 1s delay between samples proved to produce a bit more reliable result than just reading Formatted counters but, again, too slow for my 1s ticks (collect sample, wait 1s, collect sample, do the math takes longer than 1s). Thus I use PerfFormatted counters in version 0.9RC.
    Win32_PerfRawData_PerfProc_Process; Win32_PerfFormattedData_PerfProc_Process.
  
    _PID_     Unique identified of the process.
    PPID      Unique identifier of the process that started this one.
    PrioB     Base priority.
    Name      Name of the process.
    CPUpt_(N) % of CPU used by process. On machines with multiple CPUs,
        this number can be over 100% unless you see _CPUpt_N caption which
        means "Normalized" (i.e. CPUutilization / # of CPUs).
        Toggle Normal/Normalized display by pressing the "p" key.
    Thds      # of threads spawned by the process.
    Hndl      # of handles opened by the process.
    WS(MB)    Total RAM used by the process. Working Set is, basically,
        the set of memory pages touched recently by the threads belonging to
        the process. 
    VM(MB)    Size of the virtual address space in use by the process.
    PM(MB)    The current amount of VM that this process has reserved
        for use in the paging files.
However, my approach for displaying "Non-Normalized" CPU utilization didn't work :-/

Proper functionality of this feature is rather important for my job. Looking at "Normalized" CPU utilization values for a process does not tell you much. Say a process has CPU utilization of 100%. This just tells you there is at least 1 CPU that's fully utilized by the process but it does not tell you the overall utilization. "Non-Normalized" value sums CPU utilization over all CPUs that process uses. In my case, the test box has 8 Xeon processors with 6 physical and 6 virtual cores each totaling at 96 CPUs. The system is configured as such that NUMA node corresponds to 1 Xeon processor (socket). Thus, when my process utilizes entire NUMA node (socket) to the fullest, the CPU utilization for that process should be number of CPUs per Numa/Socket (12) x 100% which is 1200%:


If the process scales correctly, I will see more NUMA nodes/Sockets light up while increasing the load:

However, this does not tell me it is my process of interest that is using the CPUs. To confirm it, I need TOP script showing CPU utilization of above 1200%:

This guarantees me mysqld process is running on more than 2 sockets (sysbench is taking up ~7 CPUs and I bet mydesktopservice is the one lighting up 3rd CPU in 2nd row).

How to make it work:

Heavy rework of #region Tasks job which is starting the "Processes" monitoring job was in order. First, I had to remove all of the below code:
<#
        if ($CPUDSw) {
            Get-CimInstance Win32_PerfFormattedData_PerfProc_Process | 
                select @{Name='_PID_'; Expression={$_.IDProcess}},
                @{Name='PPID'; Expression={$_.CreatingProcessID}},
                @{Name='PrioB'; Expression={$_.PriorityBase}},
                @{Name='Name                  '; Expression={(($_.Name).PadRight(22)).substring
                    (0, [System.Math]::Min(22, ($_.Name).Length))}}, 
                @{Name='_CPUpt__'; Expression={($_.PercentProcessorTime).ToString("0.00").PadLeft(8)}},
                @{Name='Thds'; Expression={$_.ThreadCount}},
                @{Name='Hndl'; Expression={$_.HandleCount}},
                @{Name='WS(MB)'; Expression={[math]::Truncate($_.WorkingSet/1MB)}},
                @{Name='VM(MB)'; Expression = {[math]::Truncate($_.VirtualBytes/1MB)}},
                @{Name='PM(MB)'; Expression={[math]::Truncate($_.PageFileBytes/1MB)}} |
                where { $_._PID_ -gt 0} | &$sb | 
                Select-Object -First $procToDisp | FT * -Auto 1> $pth
        } else {
            Get-CimInstance Win32_PerfFormattedData_PerfProc_Process | 
                select @{Name='_PID_'; Expression={$_.IDProcess}},
                @{Name='PPID'; Expression={$_.CreatingProcessID}},
                @{Name='PrioB'; Expression={$_.PriorityBase}},
                @{Name='Name                  '; Expression={(($_.Name).PadRight(22)).substring
                    (0, [System.Math]::Min(22, ($_.Name).Length))}}, 
                @{Name='_CPUpt_N'; Expression={"{0,8:N2}" -f ($_.PercentProcessorTime / $TotProc)}},
                @{Name='Thds'; Expression={$_.ThreadCount}},
                @{Name='Hndl'; Expression={$_.HandleCount}},
                @{Name='WS(MB)'; Expression={[math]::Truncate($_.WorkingSet/1MB)}},
                @{Name='VM(MB)'; Expression = {[math]::Truncate($_.VirtualBytes/1MB)}},
                @{Name='PM(MB)'; Expression={[math]::Truncate($_.PageFileBytes/1MB)}} |
                where { $_._PID_ -gt 0} | &$sb | 
                Select-Object -First $procToDisp | FT * -Auto 1> $pth
        }
#>
and replace it with Get-Counter version:
        $processes = Get-CimInstance Win32_PerfFormattedData_PerfProc_Process | 
            Select @{Name='_PID_'; Expression={$_.IDProcess}},
            @{Name='PPID'; Expression={$_.CreatingProcessID}},
            ElapsedTime, 
            @{Name='PrioB'; Expression={$_.PriorityBase}},
            @{Name='Name'; Expression={($_.Name).ToLower()}},
            @{Name='Thds'; Expression={$_.ThreadCount}},
            @{Name='Hndl'; Expression={$_.HandleCount}}, 
            @{Name='WS(MB)'; Expression={[math]::Truncate($_.WorkingSet/1MB)}},
            @{Name='VM(MB)'; Expression = {[math]::Truncate($_.VirtualBytes/1MB)}},
            @{Name='PM(MB)'; Expression={[math]::Truncate($_.PageFileBytes/1MB)}},
            PoolNonpagedBytes, PoolPagedBytes, PercentProcessorTime |
            Where { $_._PID_ -gt 0}

        $Samples = (Get-Counter “\Process(*)\% Processor Time”).CounterSamples

Just noting Get-Counter example:
PS:511 [HEL01]> (Get-Counter “\Process(*)\% Processor Time”).CounterSamples | FL *
...
Path             : \\hel01\process(system)\% processor time
InstanceName     : system
CookedValue      : 0
RawValue         : 3434062500
SecondValue      : 131012088253272040
MultipleCount    : 1
CounterType      : Timer100Ns
Timestamp        : 29.02.16 09:40:25
Timestamp100NSec : 131012124253270000
Status           : 0
DefaultScale     : 0
TimeBase         : 10000000
Then I had to change the way of putting it all together:
        if ($CPUDSw) { 
            $pcts = $Samples | Select @{Name=”IName"; Expression={($_.InstanceName).ToLower()}}, 
              @{Name=”CPUU”;Expression={[Decimal]::Round(($_.CookedValue), 2)}}
            $processes | select '_PID_', 'PPID', 'PrioB',
                            @{Name='Name                  '; Expression=
                                {
                                    (($_.Name).PadRight(22)).substring(0, [System.Math]::Min(22, ($_.Name).Length))
                                }
                            }, 
                            @{Name='_CPUpt__'; Expression=
                                {
                                    if ($pcts.IName.IndexOf($_.Name) -ge 0) {
                                        ($pcts.CPUU[[array]::IndexOf($pcts.IName, $_.Name)]).ToString("0.00").PadLeft(8)
                                    }
                                }
                            },
            'Thds', 'Hndl', 'WS(MB)', 'VM(MB)', 'PM(MB)' | &$sb | Select-Object -First $procToDisp | FT * -Auto 1> $pth
        } else {
            $pcts = $Samples | Select @{Name=”IName"; Expression={($_.InstanceName).ToLower()}}, 
              @{Name=”CPUU”;Expression={[Decimal]::Round(($_.CookedValue / $TotProc), 2)}}
            $processes | select '_PID_', 'PPID', 'PrioB',
                            @{Name='Name                  '; Expression=
                                {
                                    (($_.Name).PadRight(22)).substring(0, [System.Math]::Min(22, ($_.Name).Length))
                                }
                            },  
                            @{Name='_CPUpt_N'; Expression=
                                {
                                    if ($pcts.IName.IndexOf($_.Name) -ge 0) {
                                        ($pcts.CPUU[[array]::IndexOf($pcts.IName, $_.Name)]).ToString("0.00").PadLeft(8)
                                    } else {
                                        #Not found (yet). Take what you have :-/
                                        ($_.PercentProcessorTime).ToString("0.00").PadLeft(8)
                                    }
                                }
                            },
            'Thds', 'Hndl', 'WS(MB)', 'VM(MB)', 'PM(MB)' | &$sb | Select-Object -First $procToDisp | FT * -Auto 1> $pth
        }
Since Get-Counter, by default, takes samples 1 second apart:
PS:507 [HEL01]> Measure-Command{(Get-Counter “\Process(*)\% Processor Time”).CounterSamples}

Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 1
Milliseconds      : 18
Ticks             : 10189709
TotalDays         : 1.17936446759259E-05
TotalHours        : 0.000283047472222222
TotalMinutes      : 0.0169828483333333
TotalSeconds      : 1.0189709
TotalMilliseconds : 1018.9709
I also abandoned all of the code relating to Timer:
    #$sw = New-Object Diagnostics.Stopwatch
    do {
        #$sw.Start()
...
        }
        #$sw.Stop()
        #if ($sw.ElapsedMilliseconds -lt 1000) {
        #    Start-Sleep -Milliseconds (1000-$sw.ElapsedMilliseconds)
        #}
        #$sw.Reset()

    } while ($true)
    #$sw = $null

So now it works! I do not know right now when I will be able to release the new version so stay tuned.

Final thoughts:

I have hit many many problems in Windows during this testing. Just note, for example, the use of ToLower() in ($_.InstanceName).ToLower() but this is something for the new blog post. This one is about TOP script.

2016-01-21

Using Powershell to implement Linux top command on Windows

Welcome to the final blog in Windows PerfCounters and Powershell series and sorry for the delay. The purpose of this blog is to explain the inner workings of top-script.ps1 script and practical usage of Performance counters on Windows through Powershell. It is intended for people who want Linux top - like tool on Windows.

The script is a part of and available in our existing benchmarking package (dbt2-0.37.50.10) developed by Mikael Ronstrom.

On Top:

If you ever did benchmarking on Linux or simply wondered "where did all my resources go", top is your best friend. Since this post is not about Linux, you can google "Linux top explained" for more details.


On Performance counters:

To learn about Windows PerfCounters, please refer to my previous blog entries in this series. I will be addressing System.Diagnostics class as just Diagnostics.


On Powershell:

For ages, Windows users were looking at bash wondering why they do not have anything similar to it. After much trial and error, Microsoft delivered Powershell. In my humble opinion, Powershell is simply great! As per difference between Powershell and bash I would mention just one; Powershell pipe passes objects while bash pipe passes plain text.


On script itself:

Type "perfmon deleting files" in Google and you'll see why I made this script ;-) Joke aside, we have a mature testing/benchmarking framework written in bash and wanted the same look and feel on Windows. Top script is just the latest piece of that effort.
I undertook this work since I firmly believe in native tools when dealing with performance issues. If I was just after measuring performance delatas between versions, some generic tool written for some other platform, such as in Perl or similar, would have been good enough. But, IMO, it would not have been fair to non-native OS.
Also, studying native tools is an integral part of studying the OS itself which is something you can not tackle performance issues without.

The script will evolve naturally to cover for the information we need in our everyday work.

Using Windows performance counters through PowerShell CIM classes it is possible to gather stats on computer performance. The script functions like this:
  o Main code starts 2 background jobs; one for collecting details for header table ("Top_Header_Job") and one for processes table ("Top_Processes_Job").
  o Each of the jobs then collects stats and writes them into a file ("header" job to TempDir\headerres.txt file, "tasks" job to TempDir\thrprocstats.txt) which is, in turn, read by script main code. Main code uses TempDir\topcfg.txt to pass info to tasks (currently, the field to sort the table by). All of the files are overwritten each time so there is not much data in them.
  o After the files are read by main code, the data is displayed. To be able to properly position the output on screen, I used various [console] functions not available in PowerShell_ISE.


Requirements:

The script can not be run in _ISE. Use Powershell console.
The script requires PS 3+.
The script requires at least 50 x 80 console window.
The script might work with .NET FW older than 4 but it was tested only with .NET 4.5.x.


Getting started:

1) Put script somewhere.
2) Start PowerShell (NOT PS_ISE)
3) cd to "somewhere" directory
4) .\top-script.ps1
        a) Get-Help .\top-script.ps1
        b) Get-Help .\top-script.ps1 -Examples
5) While script is running you can use (single key) shortcuts:
  q - Quit
  m - Sort by process WS (occupied RAM) DESC, CPUpt DESC
  p - Sort by process CPU utilization DESC, WS(MB) DESC; Non-Normalized/Normalized.
Note: Script is started with CPU utilization by process as de-normalized (i.e. on multi-CPU boxes, this value can be well over 100%). To display normalized values (i.e. "non-normalized" value / # of CPUs), just press "p" again. IF non-normalized value is the source for data, the column title will be '_CPUpt__'. Normalized CPU utilization value (CPUpt / # of CPUs) will display as '_CPUpt_N'.
  n - Sort by process Name ASC
  r - Sort by process PID ASC, CPUpt DESC
  + - Display individual CPU's. Comma separated list of values (i.e. 0,1,2).
  1 - Display individual Processors (Sockets). Comma separated list of values (i.e. 0,1,2).
Note: Script displays either Socket load or CPU load.
  - - Cancel displaying individual CPUs/Sockets.


The output:

09:21:17, Uptime:00d:00h:36m,  Users:  1,   # thds ready but queued for CPU: 0
-------------------------------------------------------------------------------
| RUNNING           | CPU                           | RAM[MB]                 |
-------------------------------------------------------------------------------
| services:    104  | Sys: 50.78%(P  4.30%/U 46.48%)| Installed:         8192 |
| processes:   126  | Idle:49.22%                   | HW reserv:       320.77 |
| threads:    1483  | HWint:             7087/0.38% | Visible:        7871.23 |
| handles:   32696  | SWint:              278/0.38% | Available:         5136 |
| CoSw/s:     6351  | High-prio thd exec:    50.38% | Modified:        117.07 |
|                   | Total # of cores:           4 | Standby:        3936.44 |
|                   |                               | PagesIn/Read ps:      2 |
-------------------------------------------------------------------------------



_PID_ PPID PrioB Name                      CPUpt Thds Hndl WS(MB) VM(MB) PM(MB)
----- ---- ----- ----                      ----- ---- ---- ------ ------ ------
 3916  852     8 mcshield                  10.14   53  490     46    225    100
 1836 5452     8 powershell#2               1.09   15  377     67    616     46
 7436 5452     8 powershell#1               0.72   17  556     85    624     67
 3984  156     8 WmiPrvSE                   0.72    9  303     14     56      9
 7024  156     8 WmiPrvSE#2                 0.72    7  201     10     52      7
 5452 5148     8 powershell                 0.36   18  454     90    628     79
 2292  852     8 FireSvc                    0.36   28  539     10    156     36
 7864 5148     8 thunderbird                0.00   52  630    293    656    264
 2880 5148     8 powershell_ise             0.00   12  427    181    869    164
 5148 3860     8 explorer                   0.00   25  810     81    267     53
 7028 6648     8 googledrivesync#1          0.00   29  712     76    193     64
 1124  852     8 svchost#5                  0.00   45 1560     46    187     31
 6508 5148     8 sidebar                    0.00   20  433     39    195     20
  816  796    13 csrss#1                    0.00   10  765     35    126      3
 6592 5148     8 iCloudServices             0.00   16  442     32    167     18
 6868 6764     8 pcee4                      0.00    7  202     32    612     32
 1976 1052    13 dwm                        0.00    5  135     31    140     26
 3400  156     8 WmiPrvSE#1                 0.00   12  297     28     91     22
 1088  852     8 svchost#4                  0.00   21  584     27    126     14
 3648  852     8 dataserv                   0.00   10  510     24    224     21
  980  852     8 svchost#2                  0.00   25  575     23    119     26
 2404  852     8 PresentationFontCache      0.00    6  149     21    506     28
...

HEADER data

Current time, uptime, # of active users, # of threads per CPU that are ready for execution but can't get CPU cycles (obviously, you want to keep this as low as possible (<= 2)).
        RUNNING section
            # of services in Started state
            # of user processes
            # of threads spawned
            # of handles open
            # of context switches per second
        CPU section
            % of CPU used (% used by privileged instr. / % used by user instr.)
            % of CPU consumed by Idle process.
            # of HW interrupts per sec./% of CPU used to service HW interrupts
            # of SW interrupts queued for servicing per sec./
                % of CPU used to service SW interrupts
            % of CPU consumed by high-priority threads execution.
            # of phys. and virt. cores. Here, 4 is Dual-Core with HT enabled.
        RAM[MB] section
            Installed RAM.
            RAM reserved by Windows for HW.
            Amount of RAM user actually sees.
            Amount of available RAM for user processes.
            Amount of RAM marked as "Modified".
            Amount of RAM marked as "Standby" (cached).
            Ratio between Memory\Pages Input/sec and Memory\Page Reads/sec.
                Number of pages per disk read. Should keep below 5.

TABLE data

        _PID_    Unique identified of the process.
        PPID     Unique identifier of the process that started this one.
        PrioB    Base priority.
        Name     Name of the process.
        CPUpt    % of CPU used by process.
        Thds     # of threads spawned by the process.
        Hndl     # of handles opened by the process.
        WS(MB)   Total RAM used by the process. Working Set is, basically,
            the set of memory pages touched recently by the threads belonging
            to the process. 
        VM(MB)   Size of the virtual address space in use by the process.
        PM(MB)   The current amount of VM that this process has reserved for
            use in the paging files.

Longer explanation of the values:

HEADER:

Foreword: Since Windows is not "process" based OS (like Linux) it is impossible to calculate the "System load". The next best thing is CPU queue length (see below).
  Uptime: Diagnostics.PerformanceCounter("System", "System Up Time")
  Users:  WMI query using query.exe tool which should be a part of your Windows.
           query user /server:localhost
           Number of users currently logged in. If no query.exe, the value is -1.
  # thds ready but queued for CPU:
          Diagnostics.PerformanceCounter("System", "Processor Queue Length")
           How many threads are in the processor queue ready to be executed but not
           currently able to use cycles. Windows OS has single queue length counter
           thus the value displayed is counter value divided with number of CPU's.
           Link.

    RUNNING section:
      services:
          (Get-Service | Where-Object {$_.Status -ne 'Stopped'} | Measure-Object).Count
           Total # of services actually running.
      processes:
          Diagnostics.PerformanceCounter("System", "Processes")
           Total number of user processes running.
           Link.
      threads:
          Diagnostics.PerformanceCounter("System", "Threads")
           Total # of threads spawned.
      handles:
          Diagnostics.PerformanceCounter("Process", "Handle Count")
           Total # of open handles.
      CoSw/s:
          Diagnostics.PerformanceCounter("System", "Context Switches/sec")
           Context switching happens when a higher priority thread pre-empts a lower
           priority thread that is currently running or when a high priority thread
           blocks. High levels of context switching can occur when many threads share
           the same priority level. This often indicates that there are too many
           threads competing for the processors on the system. If you do not see much
           processor utilization and you see very low levels of context switching, it
           could indicate that threads are blocked.
           Link.

    CPU section:
    Foreword: Windows OS has special thread called "Idle" which consumes free CPU cycles thus these counters return values relating to this one. Also, Windows are not "process" based but rather "thread" based so all of these numbers are approximations. This is even more important in the TABLE which shows CPU utilization per process (see explanation there). Most of these counters are multi-instance so instance name is '_Total' (ie. CPU utilization in total as opposed to per NUMA node, Core, CPU...).
      Sys: nn.nn%(P  mm.mm%/U zz.zz%):
          Diagnostics.PerformanceCounter("Processor Information","% Processor Time"),
           Diagnostics.PerformanceCounter("Processor Information","% Privileged Time"),
           Diagnostics.PerformanceCounter("Processor Information","% User Time").
           First number shows, effectively, % of cycles CPU(s) didn't spend running the
           Idle thread. Second number is the time CPU(s) spent on executing Privileged
           instructions while third is time CPU(s) spent executing user-mode instructions.
           For example, when your application calls operating system functions (say to
           perform file or network I/O or to allocate memory), these operating system
           functions are executed in Privileged mode.
           Link.
      Idle:
          Diagnostics.PerformanceCounter("Processor Information", "% Idle Time")
           Link.
      HWint:
          Diagnostics.PerformanceCounter("Processor Information","Interrupts/sec"),
           Diagnostics.PerformanceCounter("Processor Information","% Interrupt Time").
           Rate of hardware interrupts per second and a percent of CPU time this takes.
           Link.
      SWint:
          Diagnostics.PerformanceCounter("Processor Information","DPCs Queued/sec"),
           Diagnostics.PerformanceCounter("Processor Information","% DPC Time").
           Rate at which software interrupts are queued for execution and a % of CPU time
           this takes.
           Link.
      High-prio thd exec:
          Diagnostics.PerformanceCounter("Processor Information","% Priority Time").
           CPU utilization by high priority threads.
           Link: Can't find any links in MSDN...
      Total # of cores:
          (Get-CimInstance Win32_ComputerSystem).NumberOfLogicalProcessors
           Number of physical and virtual cores present.

    RAM[MB] section:
      Installed:
          (GCIM -class "cim_physicalmemory" | Measure-Object Capacity -Sum).Sum/1024/1024
      HW reserv:
          Installed - Visible ;-)
      Visible:
          (Get-CimInstance win32_operatingsystem).TotalVisibleMemorySize
      Available:
          Diagnostics.PerformanceCounter("Memory","Available MBytes")
      Modified:
          Diagnostics.PerformanceCounter("Memory","Modified Page List Bytes")
      Standby:
          Diagnostics.PerformanceCounter("Memory","Standby Cache Core Bytes") + 
           Diagnostics.PerformanceCounter("Memory",
             "Standby Cache Normal Priority Bytes") + 
           Diagnostics.PerformanceCounter("Memory","Standby Cache Reserve Bytes")
           Basically, cache memory.
      PagesIn/Read ps:
          Diagnostics.PerformanceCounter("Memory","Pages Input/sec")/ 
           Diagnostics.PerformanceCounter("Memory","Page Reads/sec")
           Ratio between Memory\Pages Input/sec and Memory\Page Reads/sec which
           is number of pages per disk read. Should keep below 5.
    

TABLE:

Foreword: Windows is not "process" based OS (like Linux) but rather "thread" based so all of the numbers relating to CPU usage are approximations. I did made a "proper" CPU per Process looping and summing up Threads counter (https://msdn.microsoft.com/en-us/library/aa394279%28v=vs.85%29.aspx) based on PID but that proved too slow given I have ~1 sec to deal with everything. CPU utilization using RAW counters with 1s delay between samples proved to produce a bit more reliable result than just reading Formatted counters but, again, too slow for my 1s ticks (collect sample, wait 1s, collect sample, do the math takes longer than 1s). Thus I use PerfFormatted counters in version 0.9RC.
    Win32_PerfRawData_PerfProc_Process; Win32_PerfFormattedData_PerfProc_Process
    Link.
  
    _PID_     Unique identified of the process.
    PPID      Unique identifier of the process that started this one.
    PrioB     Base priority.
    Name      Name of the process.
    CPUpt_(N) % of CPU used by process. On machines with multiple CPUs,
        this number can be over 100% unless you see _CPUpt_N caption which
        means "Normalized" (i.e. CPUutilization / # of CPUs).
        Toggle Normal/Normalized display by pressing the "p" key.
    Thds      # of threads spawned by the process.
    Hndl      # of handles opened by the process.
    WS(MB)    Total RAM used by the process. Working Set is, basically,
        the set of memory pages touched recently by the threads belonging to
        the process. 
    VM(MB)    Size of the virtual address space in use by the process.
    PM(MB)    The current amount of VM that this process has reserved
        for use in the paging files.
Note that it is possible to display CPU/Socket data for chosen HW by pressing + or 1 keys, entering 0-based index and separating multiple values by ,:

         User  Priv  Idle  HWin  SWIn              User  Priv  Idle  HWin  SWIn
-------------------------------------     -------------------------------------
%CPU  0:   47,    5,   47,    0,    0     %CPU  1:    0,    0,  100,    0,    0
%CPU  2:   35,   11,   52,    0,    0     %CPU  3:    5,    0,   94,    0,    0
The input here was 0,1,2,3 thus displaying data about first 4 cores. The CPU/Socket data is displayed between the Header and the Table areas reducing the number of visible processes. To remove this information from screen, just press "-" key.

INNER WORKINGS:

In general, script output comprises of Header part and Table part showing details on processes. In-between the two, you can show Processor/Core info. There are two background jobs started to accomplish this; "Top_Header_Job" & "Top_Processes_Job". The data about individual processors/cores is calculated in main script body.

Script starts with my usual checks, proceeds to variable declaration part where I initialize some of the performance counters (which takes time) and then starts Header and Processes jobs. The jobs itself follow the same logic. I.e. I first start perfcounter instances (which takes time) and then loop through values passing them back in file.

Main script body collects the data from files refreshing the display. Also, main script is in charge of displaying individual processor/core data as well as monitoring the keyboard input. This means CTRL+C will NOT work but you can still stop the script with CTRL+BREAK:
[console]::TreatControlCAsInput = $true
Regular way to exit is pressing the "q" key.
After you press the "q" key, cleanup code is executed, stopping the background jobs and removing temporary files used for communication. It's worth noting that cleanup code does not throw any errors. This is because nothing bad can happen. Files are less than 1kB in total while background jobs can be stopped either via trick described below or simply by exiting Powershell console.

Lets go deeper into the regions of code now. First region is Check which I described in October 2015 blog so no need to repeat myself. Next is Variable Declarations region where I gather one-time top-level data, mainly related to CPU topology using tricks described in Blog 3 and Blog 4 by manipulating Instances as described in Blog 1 of this series. Executing this part takes couple of seconds.

Next thing is to start the Header job. It takes argument (total number of cores) from the call and proceeds with initializing various counters. As with all initializations, this also takes couple of seconds. Main DO loop starts the timer to ensure samples are collected in 1 second intervals. Also, it checks if you have query.exe tool installed and determines the number of active users, if the tool exists, or displays -1 if it doesn't. There are other ways of determining number of logged users but they are all too slow for 1s tick. After forming the resulting lines, I use [System.IO.StreamWriter] to record them to Env:\TEMP headerres.txt file. The control is then returned to main script which waits for Env:\TEMP headerres.txt file (or 20s, whichever comes first).

Next step is to start the Tasks job which will collect data about running processes. As opposed to Task manager, I show background processes (ie. services) too. Worth noting is that, due to timing issues, I use Process (Win32_PerfFormattedData_PerfProc_Process) and not Thread (win32_PerfFormattedData_PerfProc_Thread) counters.
Since Windows is *thread* based (meaning a Process is just a container for Threads doing the work) this actually means scarifying some of the accuracy (for example CPU utilization data) in favour of faster and smoother execution:
#(Active) Code when using Process counter:
    Get-CimInstance Win32_PerfFormattedData_PerfProc_Process | 
        Select @{Name='_PID_'; Expression={$_.IDProcess}},
        @{Name='PPID'; Expression={$_.CreatingProcessID}},
        @{Name='PrioB'; Expression={$_.PriorityBase}},
        @{Name='Name                  '; Expression={
            (($_.Name).PadRight(22)).substring(0, [System.Math]::Min(22, ($_.Name).Length))
        }}, 
        @{Name='_CPUpt__'; Expression={($_.PercentProcessorTime).ToString("0.00").PadLeft(8)}},
        @{Name='Thds'; Expression={$_.ThreadCount}},
        @{Name='Hndl'; Expression={$_.HandleCount}},
        @{Name='WS(MB)'; Expression={[math]::Truncate($_.WorkingSet/1MB)}},
        @{Name='VM(MB)'; Expression = {[math]::Truncate($_.VirtualBytes/1MB)}},
        @{Name='PM(MB)'; Expression={[math]::Truncate($_.PageFileBytes/1MB)}} | #,
        Where { $_._PID_ -gt 0} | &$sb | 
            Select -First $procToDisp | FT * -Auto 1> $ToFile 
Note: Script-block $sb is used just for sorting the resultset depending on keyboard input.
Note: "Name=" is the same as writing "Label=". Both can be abbreviated so the expression becomes @{L=...";"E={...}}.

#(More precise but slower) Code when scanning recursively the Thread counter:
    #Get the CPU utilization percentages by summing up threads over particular process
    $pcts = Get-CimInstance win32_perfformatteddata_perfproc_thread -Property IDProcess,
      PercentProcessorTime | Group -Property IDProcess | Foreach {
        New-Object PSObject -Property @{
          PID = ($_.Group.IDProcess | Select -First 1)
          CPUpt = "{0,5:N2}" -f (($_.Group | Measure-Object -Property PercentProcessorTime -Sum).Sum)
        }
      }

    #Pair with Process data:
    Get-CimInstance Win32_PerfFormattedData_PerfProc_Process | 
      Select @{Name='_PID_'; Expression={$_.IDProcess}},
      @{Name='PPID'; Expression={$_.CreatingProcessID}},
      @{Name='PrioB'; Expression={$_.PriorityBase}},
      @{Name='Name'; Expression={($_.Name).PadRight(25)}},
      @{Name='CPUpt'; Expression={$pcts.CPUpt[[array]::IndexOf($pcts.PID, $_.IDProcess)]}},
      @{Name='Thds'; Expression={$_.ThreadCount}},
      @{Name='Hndl'; Expression={$_.HandleCount}},
      @{Name='WS(MB)'; Expression={[int]($_.WorkingSet/1MB)}},
      @{Name='VM(MB)'; Expression = {[int]($_.VirtualBytes/1MB)}},
      @{Name='PM(MB)'; Expression={[int]($_.PageFileBytes/1MB)}} | 
      Where { $_.Name -notmatch "_Total" -and $_.Name -notmatch "Idle"} | &$sb |
        Select -First $procToDisp | FT * -Auto 1> $ToFile 
Note: Script-block $sb is used just for sorting the resultset depending on keyboard input.

There is one more way of doing this and that is by expanding Process perf object. I use this approach when checking for congestion on thread level (MSDN):
#Run once:
#Header row, initialize output file:
"PID,Process,ThdID,CPU time (s),PctUser,PctPriv,State,WaitR,PrioLvL,PrioShift,IdealProc,ProcAff" |
  Out-File E:\test\thds.csv
#PIDs of interest to me:
$Processes = Get-Process | 
  Where {($_.ProcessName -match "mysql") -or ($_.ProcessName -match "ndb") -or ($_.ProcessName -match "sysben")} |
  Sort -Property ID
Note: If you check the value of $Processes variable here, you will notice something like
Id                         : 1996
...
Threads                    : {2000, 2012, 2016, 2040...}
...
meaning Threads member is actually an object and can be expanded to show more data:
PS > $Processes.Threads

BasePriority            : 8
CurrentPriority         : 9
Id                      : 1972
IdealProcessor          : 
PriorityBoostEnabled    : 
PriorityLevel           : 
PrivilegedProcessorTime : 
StartAddress            : 2006300688
StartTime               : 
ThreadState             : Wait
TotalProcessorTime      : 
UserProcessorTime       : 
WaitReason              : UserRequest
ProcessorAffinity       : 
Site                    : 
Container               : 
...
#Run following in loop, append result to file
#Threads belonging to PIDs of interest.
Foreach ($Process in $Processes) {
    $ProcessThds = $Process | Select -ExpandProperty Threads | Sort -Property ID
    Foreach ($ProcessThd in $ProcessThds) {
        $ProcName = @{L="Name";E={ $Process.ProcessName}}
        $ProcID = @{L="PID";E={ $Process.Id}}
        $ThdID = @{L="ThreadID";E={ $ProcessThd.Id }}
        $CPUTime = @{L="CPU Time (Sec)";E={ [math]::round($ProcessThd.TotalProcessorTime.TotalSeconds,2) }}
        $UsrCPUTime = @{L="User CPU Time (%)";E={ [math]::round((($ProcessThd.UserProcessorTime.ticks /
          $ProcessThd.TotalProcessorTime.ticks)*100),1) }}
        $State = @{L="State";E={ $ProcessThd.ThreadState }}
        $WR = @{L="WaitR";E={ $ProcessThd.WaitReason}}
        $PrioDelta = @{L="PrioSh";E={ $ProcessThd.CurrentPriority - $ProcessThd.BasePriority}}
        $IdProc = @{L="Ideal proc";E={ $ProcessThd.IdealProcessor}}
        $ProcAf = @{L="Proc affinity";E={ $ProcessThd.ProcessorAffinity}}
        $PrioLvL = @{L="Prio level";E={ $ProcessThd.PriorityLevel}}
        $PrivCPU = @{L="Privil CPU";E={ [math]::round((($ProcessThd.PrivilegedProcessorTime.ticks /
          $ProcessThd.TotalProcessorTime.ticks)*100),1) }}
        $ProcessThd | Select -Property  $ProcName, $ProcID, $ThdID, StartTime, $CPUTime, $UsrCPUTime, $PrivCPU, 
            $State, $WR, $PrioLvL, $PrioDelta, $IdProc, $ProcAf |
        %{'{0},{1},{2},{3},{4},{5},{6},{7},{8},{9},{10},{11}' -f $_.PID,$_.Name, $_.ThreadID,$_."CPU Time (Sec)",
        $_."User CPU Time (%)",$_."Privil CPU",$_.State,$_.WaitR,$_."Prio level",$_.PrioSh,$_."Ideal proc",
        $_."Proc affinity"} | Out-File E:\test\thds.csv -Append
    }
}
This leaves me with neat little CSV file which I then import to Excel and group by Process ID for further analysis.


Back to main script, region Main-start, where I wait for Processes job to start producing data before proceeding. If there is no data generated, the script will stop the jobs and exit.
Next is the neat trick to reduce the flicker while clearing up the screen:
[System.Console]::Clear()
and positioning the cursor at top left corner:
$saveYH = [console]::CursorTop
$saveXH = [console]::CursorLeft

Worth noting here, in terms of reduced flicker, is hiding the cursor itself:
[console]::CursorVisible = $false

After that, you enter region Main-loop which is the main code for the script. If there is fresh header data to be displayed, I move cursor to (0,0) and write it out. Otherwise, I skip this and check if I should display Core/Socket data. The problem here is that user can specify any number of cores/sockets to display data for and I display two of them in each line. Thus I need an array where user input is mapped to absolute index of the requested piece of HW in perf counter. The array is created in key-press handler. For the sake of performance, both core and socket counters were initialized at the start of the script:
#Just the individual CPUs.
$CPUdata = Get-CimInstance Win32_PerfFormattedData_Counters_ProcessorInformation | Where {$_.Name -match "^(\d{1}),(\d{1})"}
#Just the individual Sockets.
$Socketdata = Get-CimInstance Win32_PerfFormattedData_Counters_ProcessorInformation | Where {$_.Name -match "^(\d{1}),_Total"}

Then, if there is fresh data provided by Top_Processes_Job, I display it.

Next comes the keyboard handling routine. First, check that there is something to handle:
  if ($Host.UI.RawUI.KeyAvailable) {
If there is, put it into variable:
    $k = $Host.UI.RawUI.ReadKey("AllowCtrlC,IncludeKeyDown,IncludeKeyUp,NoEcho").Character
Once the keypress is processed, clear the input buffer:
    if ("p" -eq $k) {
      'CPUpt' > $conf
      $HOST.UI.RawUI.Flushinputbuffer()

"+" and "1" keys process input of CPUs/Sockets to display data for, while "-" key stops displaying that data.
Pressing "c" key will clear the screen in case it becomes garbled.
Pressing "q" key moves you to region Cleanup ending the script run.


TIPS & TRICKS

As opposed to Windows TaskManager, I show background processes too (ie. "services").

In an effort to achieve smoother display of data, I am truncating CPU/Socket info to their integer values. Also, I do not use Thread counters but rather Process ones. Due to delay while displaying the data, there will always be some discrepancy between data displayed. I.e. Total CPU utilization in Header will rarely match sum of CPU utilization by processes in table. I can live with that.

Script is started in non-normalized CPU utilization mode which means CPU utilization per process can go well over 100% on modern boxes. Let's say you have Quad core box (8 CPUs) and a process taking 50% of Core0, 60% of Core1, 30% of Core2 and 20% of Core3 then the non-normalized CPU utilization for such process would be 160% while normalized CPU utilization would be 20% (160/8). I did it as such to confirm that process actually uses more than one CPU. To toggle between non-normalized and normalized view, use "p" key.

If, for any reason, display becomes garbled, press the "c" key.

Number of processes to display is controlled by $procToDisp variable which is, atm, hard-coded to 25.

Initial sort order is defined by $procSortBy variable. Default is CPU% ($procSortBy = 'CPUpt').

IF by any chance script does not terminate normally:
- First type Get-Job
- Check that Name has "Top_Header_Job" & "Top_Processes_Job". Remember the Id (or use Name parameter).
Say Id's are 14 and 16.
- Type commands (text after # is just a comment):
[console]::CursorVisible = $true #reclaims the cursor
[console]::TreatControlCAsInput = $false #reverts CTRL+C processing to default value
receive-job -id 14
receive-job -id 16
stop-job -id 16
stop-job -id 14
remove-job -id 14
remove-job -id 16

or just exit the Powershell window.


Hope you'll find this script useful in your work!


This is all from me for this series. Next, I will start new series of blogs describing script used as testing/benchmarking framework on Windows which is also available in the package.

2015-11-23

Windows PerfCounters and Powershell - CPU perf data

So far, I talked of WMI, CIM, WQL, System.Diagnostics.PerformanceCounterCategory, perf-counter data organization and flavour. Now it's time to look at some performance counters I deem important for my use-case more closely.
Note: List of available Counters for Get-Counter command
Get-Counter -ListSet * | Sort-Object CounterSetName | Format-Table CounterSetName

Basic concepts:

I will introduce basic concepts of Processor, Core and CPU now to help you follow the text. Let us use this convention:
  • "Processor" is a piece of hardware you connect to a slot on the motherboard.
  • "Physical Core" is a physical computing unit built into the "Processor".
  • "Virtual Core" is a virtual computing unit built on top of "Physical Core" (i.e. HT is ON).
  • "CPU" is a computing unit inside the "Processor", either physical or virtual.


Putting concepts to work

Now lets calculate number of CPUs for my laptop:
PS > ((Get-CimInstance -Namespace root/CIMV2 -ClassName CIM_Processor).NumberOfLogicalProcessors | Measure-Object -Sum).Sum

4

Note: Many other counters fail for some HW configuration and/or OS! Be sure to check.
Note: HT is ON on my dual-core laptop and no cores are parked so to get number of Physical cores:
PS > ((Get-CimInstance -Namespace root/CIMV2 -ClassName CIM_Processor).NumberOfCores | Measure-Object -Sum).Sum

2

Note: There are many ways to collect this info:
PS > (Get-CimInstance Win32_ComputerSystem).NumberOfLogicalProcessors
PS > ((New-Object Diagnostics.PerformanceCounterCategory("Processor Information")).GetInstanceNames() | ?{$_ -match "^(\d{1}),(\d{1})"} | Measure-Object -Sum).Count
Note: RegEx expression is matching "Number,Number" Instances only (See previous blog about instances).

It is not obvious when working with 1 NUMA node/Slot, but the -Sum might refer to Sum of CPUs per Slot, depending on RegEx.

Before starting on Counters, let me stress that the measurements at the system, process and thread level in Windows are based on a sampling methodology thus the data gathered is subject to typical sampling errors like:
  • accumulating a "sufficient" number of sample observations to be able to make a reliable statistical inference, i.e. the sampling size
and
  • ensuring that there aren’t systemic sources of sampling error that causes results to be under or over-sampled as I will demonstrate shortly.

As of W2K8, the trends are changing towards event driven measurement for CPU utilization which, although more sane and accurate, poses its own set of challenges (say, a clock drift across multiprocessor cores when they are not resynchronized periodically and so on). To compensate for drift, new PerfMon/ResMon work by measuring CPU load in real time using event oriented measurement data gathered by the OS Scheduler each time a context switch occurs.
A context switch occurs in Windows whenever the processor switches its execution context to run a different thread (see more below). Context switches also occur as a result of high priority Interrupt Service Routines (ISRs) as well as the Deferred Procedure Calls (DPCs) that ISRs schedule to complete the interrupt processing. Starting in Windows 6 (Vista/2008), the OS Scheduler began issuing RDTSC instructions to get the internal processor clock each time a context switch occurs. I will talk of context switching and DPC counters in a short while. For more details please see this excellent blog post.

System CPU counters:

First counter I want to talk about is Processor Queue Length. Immediately a Linux users observes that there is no "System load" counter on Windows. This is because Windows OS is Thread based as opposed to Linux which is Process based. This simply means that, in Windows, an execution thread is a basic unit of execution (thus basis for collecting usage statistics too) and a process acts as a container for threads. As simple as it may seem, this actually poses a lot of challenges since one has to start aggregating data about running processes from Threads counters and work his way up. I will talk about this in detail in final blog. So, the WMI counter mimicking Linux "System load" best is, IMO, Processor Queue Length:
PS > Get-Counter '\System\Processor Queue Length'

Timestamp                 CounterSamples                                                      
---------                 --------------                                                      
23.10.15. 10:34:10        \\server_name\system\processor queue length : 1                                          
However, this is slooooow (although subsequent calls return much faster):
PS > Measure-Command { Get-Counter '\System\Processor Queue Length' }

TotalSeconds      : 4.2961321

PS > Measure-Command { Get-Counter '\System\Processor Queue Length' }

TotalSeconds      : 1.007445
So, as described in previous blog, I use System.Diagnostics class to fetch this value:
PS > Measure-Command { New-Object Diagnostics.PerformanceCounter("System", "Processor Queue Length")}

TotalSeconds      : 2.0006457

PS > Measure-Command { New-Object Diagnostics.PerformanceCounter("System", "Processor Queue Length")}

TotalSeconds      : 0.000643
Now, put this into a variable and simply call NextValue():
PS > $System_ProcQL = New-Object Diagnostics.PerformanceCounter("System", "Processor Queue Length")
PS > $System_ProcQL.NextValue()
0
PS > $System_ProcQL.NextValue()
10
The value obtained is for all of the CPU's so you need to calculate the number of CPU's to be your divider and obtain the real value:
$SystemLoad = $System_ProcQL.NextValue() / $totCPU


In-depth description of System Counters important for my use-case:

COUNTER: System\Processor Queue Length
TYPE: Instantaneous
USAGE: New-Object Diagnostics.PerformanceCounter("System", "Processor Queue Length") / ((Get-CimInstance -Namespace root/CIMV2 -ClassName CIM_Processor).NumberOfLogicalProcessors | Measure-Object -Sum).Sum
MEANING: Number of threads per CPU that are ready for execution but can't get CPU cycles for whatever reason thus waiting in OS Scheduler queue. Since Windows have one Scheduler queue, I divide this value with total number of computation units (i.e. CPUs). The actual mechanics is that when Counter value is requested a measurement function traverses the Scheduler Ready Queue and counts the number of threads waiting for an available CPU.
GOTCHA: Even on idle system there can be significant number of threads running on schedule that can bump this number very high. Say you have 4 CPU box and processes fetching values for 100 counters, 10 samples every 1 second. All of these sample requests will lay sleeping for 1 second (thus the Processor Queue Length value will be low) and then all will wake up at the same timer event (clock interrupt) causing Processor Queue Length to spike although there is no real load on the system. It's even worse if your thread(s) is of high priority as it will get executed sooner than the user threads thus pushing Processor Queue Length number very very high. This leads to disproportionate number of Ready Threads waiting for cycles, even (or especially) when the processor itself is not very busy overall. So tip 1 would be to check if CPUs are really busy or not.
THRESHOLD: Pending on above, it is hard to tell what the threshold value is but most people seem to agree it's "sustained value of 2 or more" with CPU utilization of 85%+. This combination tells us we can benefit from adding more CPUs.

COUNTER: System\Context Switches/sec
TYPE: Instantaneous
USAGE:
$System_CSpS = New-Object Diagnostics.PerformanceCounter("System", "Context Switches/sec")
$System_CSpS.NextValue()

MEANING: Context switching happens when a higher priority thread pre-empts a lower priority thread that is currently running or when a high priority thread blocks. High levels of context switching can occur when many threads share the same priority level. This often indicates that there are too many threads competing for the processors on the system. If you do not see much processor utilization and you see very low levels of context switching, it could indicate that threads are blocked (link).
GOTCHA: The number obtained is system-wide! To report the total number of context switches generated per second by all threads use the Thread(_Total)\Context Switches/sec counter (Category((Instance)\Counter):
New-Object Diagnostics.PerformanceCounter("Thread", "Context Switches/sec", "_Total")
THRESHOLD: Context switching rates in excess of 15,000 per second per CPU. The remedy would be to reduce the number of threads and queue more at the application level. This will cause less context switching, and less context switching is good for reducing CPU load.


In-depth description of CPU Counters important for my use-case:

Note: "Processor Information" category, besides overall _Total, has instances for Slot/NUMA node (0,_Total, n,_Total) while "Processor" category gives just _Total for all CPUs as defined above.
Gotcha: On single slot machines, "Processor" category will give info for all the CPUs while on machines with multiple slots, it will give info on just the Physical cores :-/
Thus, if InstanceName is _Total, both yield the same value.

COUNTER: Processor Information(_Total)\% Processor Time, Processor(_Total)\% Processor Time
TYPE: Sample, Instance
USAGE:
$InstanceName = "_Total"
$PI_PT = New-Object Diagnostics.PerformanceCounter("Processor Information", "% Processor Time")
$PI_PT.InstanceName = $InstanceName
$null = $PI_PT.NextValue()
--or--
Get-Counter -Counter "\Processor Information(_Total)\% Processor Time"
Get-Counter -Counter "\Processor(_Total)\% Processor Time"

MEANING: Primary indicator of CPU activity. High values many not necessarily be bad. However, if the other processor-related counters are increasing linearly such as Processor\% Privileged Time or System\Processor Queue Length, high CPU utilization may be worth investigating.
GOTCHA: If this counter is around threshold value, starting new processes will only lead to increased value of Processor Queue Length but the work done will remain the same. Look for some more counters that I'm about to describe in relation to this one.
THRESHOLD: Folks seem to agree on ~85%. Low CPU utilization with sustained Processor Queue Length value of 2 or higher is indicator that requests for CPU time arrive randomly and threads demand irregular amounts of time from the CPU. This means that the processor power is not a bottleneck but that the application threading logic should be improved.

COUNTER: Processor Information(_Total)\% Privileged Time
TYPE: Sample, Instance
USAGE:
$InstanceName = "_Total"
$PI_PPT = New-Object Diagnostics.PerformanceCounter("Processor Information", "% Privileged Time")
$PI_PPT.InstanceName = $InstanceName
$null = $PI_PPT.NextValue()

MEANING: Counter indicates the percentage of non-idle CPU time spent in privileged mode, i.e. calls to OS functions (file or network I/O, memory allocation...). Basically, this is unrestricted mode allowing direct access to hardware and all memory.
GOTCHA:
THRESHOLD: Folks seem to agree on consistently being over 75%.

COUNTER: Processor Information(_Total)\% User Time
TYPE: Sample, Instance
USAGE:
$InstanceName = "_Total"
$PI_PUT = New-Object Diagnostics.PerformanceCounter("Processor Information", "% User Time")
$PI_PUT.InstanceName = $InstanceName
$null = $PI_PUT.NextValue()

MEANING: Percentage of non-idle CPU time spent in user mode. User mode is a restricted processing mode designed for applications, environment subsystems, and integral subsystems.
GOTCHA: Processor Information(_Total)\% Privileged Time +
Processor Information(_Total)\% User Time = Processor Information(_Total)\% Processor Time.
THRESHOLD: Depends on previous two counters.

COUNTER: Processor Information(_Total)\% Idle Time
TYPE: Sample, Instance
USAGE:
$InstanceName = "_Total"
$PI_PIT = New-Object Diagnostics.PerformanceCounter("Processor Information", "% Idle Time")
$PI_PIT.InstanceName = $InstanceName
$null = $PI_PIT.NextValue()

MEANING: Counter indicates the percentage of time OS idle thread was consuming cycles. On Windows, there is a special Kernel thread that consumes cycles when CPU is idling. Counting cycles consumed by this thread gives Idle CPU time.
GOTCHA: Processor Information(_Total)\% Processor Time + Processor Information(_Total)\% Idle Time = 100%
THRESHOLD:

COUNTER: Processor Information(_Total)\% Priority Time
TYPE: Sample, Instance
USAGE:
$InstanceName = "_Total"
$PI_PPRIOT = New-Object Diagnostics.PerformanceCounter("Processor Information", "% Priority Time")
$PI_PPRIOT.InstanceName = $InstanceName
$null = $PI_PPRIOT.NextValue()

MEANING: CPU utilization by high priority threads.
GOTCHA: Kernel scheduler can, on occasion, wake up low priority threads sleeping for "long" time assigning them much more slices on CPU than one would expect given the (low)priority. This, in turn, blocks high-priority threads from execution which is never an expected behaviour. I would look at this value in relation to Context switches/second to determine what's going on.
THRESHOLD:

COUNTER: Processor Information\Interrupts/sec
TYPE: Sample, Instance
USAGE:
$InstanceName = "_Total"
$PI_INTPS = New-Object Diagnostics.PerformanceCounter("Processor Information", "Interrupts/sec")
$PI_INTPS.InstanceName = $InstanceName
$null = $PI_INTPS.NextValue()

MEANING: Number of hardware interrupts per second. This value is the indicator of the activity of devices that generate interrupts, such as network adapters.
GOTCHA: See next counter.
THRESHOLD:

COUNTER: Processor Information\% Interrupt Time
TYPE: Sample, Instance
USAGE:
$InstanceName = "_Total"
$PI_PINTT = New-Object Diagnostics.PerformanceCounter("Processor Information", "% Interrupt Time")
$PI_PINTT.InstanceName = $InstanceName
$null = $PI_PINTT.NextValue()

MEANING: The value indicates the percentage of time CPUs spend receiving and servicing hardware interrupts. This value is an indirect indicator of the activity of devices that generate interrupts, such as network adapters.
GOTCHA: Mass increase in Processor Information\Interrupts/sec and Processor Information\% Interrupt Time indicates potential hardware problems.
THRESHOLD:

COUNTER: Processor Information\DPCs Queued/sec
TYPE: Sample, Instance
USAGE:
$InstanceName = "_Total"
$PI_DPCQPS = New-Object Diagnostics.PerformanceCounter("Processor Information", "DPCs Queued/sec")
$PI_DPCQPS.InstanceName = $InstanceName
$null = $PI_DPCQPS.NextValue()

MEANING: Overall rate at which deferred procedure calls ("SW interrupts") are added to the processor's DPC queue. This property measures the rate at which DPCs are added to the queue, not the number of DPCs in the queue.
GOTCHA: This is NOT the number of SW interrupts in the queue!
THRESHOLD:

COUNTER: Processor Information\DPC Time
TYPE: Sample, Instance
USAGE:
$InstanceName = "_Total"
$PI_PDPCT = New-Object Diagnostics.PerformanceCounter("Processor Information", "% DPC Time")
$PI_PDPCT.InstanceName = $InstanceName
$null = $PI_PDPCT.NextValue()

MEANING: Percentage of time that the processor spent receiving and servicing deferred procedure calls (SW interrupts) during the sample interval. They are counted separately and are not a component of the interrupt counters.
GOTCHA: This property is a component of PercentPrivilegedTime because DPCs are executed in privileged mode.
THRESHOLD:

Other useful counters I would look into in case of trouble are C1/C2/C3TransitionsPerSec. There is a huge penalty waking up CPU from C3 low power state to C2 low power state and considerable penalty transitioning from C2 to C1. So if box is choking and CPUs are idling, look here. And make sure ParkingStatus for each CPU is 0 ;-)
Example: (Physical) CPU 9 in Slot 7 was asleep:
PS > GCim Win32_PerfFormattedData_Counters_ProcessorInformation
...
Name                        : 7,9
AverageIdleTime             : 100
C3TransitionsPersec         : 64
ClockInterruptsPersec       : 64
IdleBreakEventsPersec       : 64
InterruptsPersec            : 64
PercentC3Time               : 99
...
Basically, only processing timer events.

There are also combinations of counters that can point out problems like Processor\% DPC Time, % Interrupt Time and % Privileged Time. If Interrupt Time and DPC time are a large portion of Privileged Time, the kernel is spending significant amount of time processing (most likely) I/O requests. In some cases performance can be improved by configuring interrupts and DPC affinity to a small number of CPUs on a multiprocessor system, which improves cache locality. In other cases, it works best to distribute the interrupts and DPCs among many CPUs, so as to keep the interrupt and DPC activity from becoming a bottleneck.

In the next blog I will cover Memory performance counters.

In this series:
BLOG 1: PerfCounters infrastructure
BLOG 2: PerfCounters Raw vs. Formatted values
BLOG 3: PerfCounters, fetching the values
BLOG 4: PerfCounters, CPU perf data
BLOG 5: PerfCounters, Memory perf data
BLOG 6: PerfCounters, Disk/IO perf data
BLOG 7: PerfCounters, Network and Contention perf data