Windows PerfCounters and Powershell - Memory perf data

In the last blog I spoke of CPU counters. Now, I'll talk of Memory counters.

MEMORY Counters (CIM_PhysicalMemory class, Win32_PerfFormattedData_PerfOS_Memory class, Memory Performance Information ...):

Note: I introduced the notion of samples and how to fetch them using NextValue() so I will occasionally omit $var.NextValue() going forward.

Let me note here that if you thought previously described performance classes were complicated, you are now entering the realm of black magic ;-) There is a good series of blogs on subject of Memory by Mark Russinovich worth reading although quite old.

Memory is a key resource for any machine so I will look at the most of the values available on Windows. In Resource monitor, Memory tab, you find a bar with Hardware reserved, In use, Modified, Standby and Free values. There are also Available, Cached, Total and Installed values. Let's start with the biggest number, Installed RAM.

In-depth description of Memory Counters important for my use-case:

COUNTER: cim_physicalmemory\Capacity
TYPE: Instantaneous
USAGE: (Get-Ciminstance -class "cim_physicalmemory" | Measure-Object Capacity -Sum).Sum / 1024 / 1024 #MB
MEANING: Total capacity of the physical memory, in bytes. Refers to "Installed".
GOTCHA: You will find tips to use TotalPhysicalMemory but, according to MSDN, it's been deprecated. Also, that page recommends using TotalVisualMemorySize property in the CIM_OperatingSystem class instead but this is wrong as there is no TotalVisualMemorySize property and, even if there was, we need installed memory size.
THRESHOLD:

Intermediate step; how much of the installed memory is available to OS:
COUNTER: win32_operatingsystem\TotalVisibleMemorySize
TYPE: Instantaneous
USAGE: [math]::Round((Get-CimInstance win32_operatingsystem).TotalVisibleMemorySize / 1024,2)
MEANING: Total amount of RAM available to OS. Refers to "Total".
GOTCHA:
THRESHOLD:

Subtracting TotalVisibleMemorySize from Capacity gives us HW reserved RAM, i.e. RAM taken by various HW such as video card. Check this post for details.
COUNTER: HW reserved
TYPE: Calculated
USAGE: cim_physicalmemory\Capacity (Installed) - win32_operatingsystem\TotalVisibleMemorySize (Total)
MEANING: Size of RAM not available to OS although installed on the system. Refers to "Hardware reserved".
GOTCHA: Depends on HW and BIOS settings, not something "fixable" in Windows.
THRESHOLD:

COUNTER: win32_operatingsystem\FreePhysicalMemory (Bytes), Memory\Available MBytes
TYPE: Instantaneous
USAGE:
(Get-WmiObject win32_operatingsystem).FreePhysicalMemory
$Memory_AvailMB = New-Object Diagnostics.PerformanceCounter("Memory", "Available MBytes")
(New-Object Diagnostics.PerformanceCounter("Memory", "Available MBytes")).RawValue
MEANING: Total amount of RAM available to processes. Equal to the sum of memory assigned to the standby (cached), free and zero page lists. Refers to "Available".
GOTCHA:
THRESHOLD: A consistent value of less than 20% of installed RAM. In such situations, consult additional counters, such as Win32_PerfFormattedData_PerfOS_Memory\PagesPerSec to determine if System memory is adequate for the workload.

COUNTER: In use memory
TYPE: Calculated
USAGE: win32_operatingsystem\TotalVisibleMemorySize (Total) - Memory\Available MBytes (Available)
MEANING: Amount of RAM in use by processes running on the box.
GOTCHA:
THRESHOLD:

COUNTER: Memory\Modified Page List Bytes (Win32_PerfFormattedData_PerfOS_Memory)
TYPE: Instantaneous
USAGE: $Memory_ModPLBy = New-Object System.Diagnostics.PerformanceCounter("Memory", "Modified Page List Bytes")
MEANING: The amount of RAM taken by the pages previously belonging to a working set but removed. However, the pages were modified while in use and their current contents haven’t yet been written to storage. The Page Table Entry still refers to the physical page(s) but is marked invalid and in transition. It must be written to the backing store before the physical page can be reused.
GOTCHA: No description in MSDN!?
THRESHOLD: Keep as low as possible.

COUNTER: Win32_PerfFormattedData_PerfOS_Memory\FreeAndZeroPageListBytes
TYPE: Instantaneous
USAGE: (get-wmiobject -computername localhost -Namespace root\CIMV2 -Query "Select * from Win32_PerfFormattedData_PerfOS_Memory").FreeAndZeroPageListBytes / 1024 / 1024 #MB
MEANING: The amount of physical memory, in bytes, that is assigned to the free and zero page lists thus immediately available for allocation to a process or for system use since it does not contain any data. Refers to "Free".
GOTCHA: There is a big difference between Free and Available memory. This is due to most of the pages considered available being in some sort of transition state (i.e. waiting to be written to disk) or have not yet met all of the OS requirements (i.e. page is not considered secure until it's zeroed out).
THRESHOLD: Keep as high as possible.

COUNTER: Standby
TYPE: Calculated
USAGE:
$Memory_SBCCBy = New-Object Diagnostics.PerformanceCounter("Memory", "Standby Cache Core Bytes")
$Memory_SBCNPBy = New-Object Diagnostics.PerformanceCounter("Memory", "Standby Cache Normal Priority Bytes")
$Memory_SBCRBy = New-Object Diagnostics.PerformanceCounter("Memory", "Standby Cache Reserve Bytes")
[math]::Round($Memory_SBCCBy.NextValue()/1024/1024 + $Memory_SBCNPBy.NextValue()/1024/1024+$Memory_SBCRBy.NextValue()/1024/1024,2)
MEANING: The amount of RAM in pages previously belonging to a working set but removed (or marshaled directly into the standby list). The pages weren’t modified since last written to disk. The Page Table Entry still refers to the physical pages but are marked invalid and in transition. Or, simpler explanation, memory that has been removed from a process's working set (its physical memory) en route to disk but is still available to be recalled.
GOTCHA: Please see the explanation of the factors in Win32_PerfFormattedData_PerfOS_Memory or Memory Object MSDN pages.
THRESHOLD:

COUNTER: Cached
TYPE: Calculated
USAGE:
MEANING: This number represents the sum of the system working set, standby list and modified page list. So, Memory\Cache Bytes, Memory\Modified Page List Bytes, Memory\Standby Cache Core Bytes, Memory\Standby Cache Normal Priority Bytes and Memory\Standby Cache Reserve Bytes. In this case, Memory\Cache Bytes + Memory\Modified Page List Bytes + Standby.
GOTCHA: Presented here for the sake of completeness.
THRESHOLD:

More counters of significance:

Win32_PerfFormattedData_PerfOS_Memory\CacheBytes - Number of bytes currently being used by the file system cache. The file system cache is an area of physical memory that stores recently used pages of data for applications. The operating system continually adjusts the size of the cache, making it as large as it can while still preserving the minimum required number of available bytes for processes. This property displays the last observed value only; it is not an average. See also SystemCacheResidentBytes and relatives.
Simpler explanation would be that the memory pages that the System uses are counted in two main counters, Cache Bytes and Pool Nonpaged Bytes. The Cache Bytes counter value is the amount of resident pages allocated in RAM that the Kernel threads can address without causing a Page Fault. This counter includes the Pool Paged Resident Bytes, the System Cache Resident Bytes, the System Code Resident Bytes and the System Driver Resident Bytes.

Note: If Memory\Pool Nonpaged Bytes value is 10% or more higher than its value at system startup, there is probably a leak.

Win32_PerfFormattedData_PerfOS_Memory\CacheFaultsPerSec - Number of faults which occur when a page is not found in the file system cache and must be retrieved from elsewhere in memory (a soft fault) or from disk (a hard fault). The file system cache is an area of physical memory that stores recently used pages of data for applications. Cache activity is a reliable indicator of most application I/O operations. This property counts the number of faults without regard for the number of pages faulted in each operation.

There is a whole set of Paging counters and they do require our attention since we can deduce Memory shortages on Windows by using them. Some of the key counters I will describe below. Dealing with Windows Paging you have to keep in mind that paging occurs for various operations within OS and excessive paging doesn’t automatically indicate a memory shortage. For instance, many network card drivers utilize the Pagefile (sometimes excessively) and this can be misread as a memory shortage.

Win32_PerfFormattedData_PerfOS_Memory\PagesPerSec (and relatives) - A sustained value of over 20 should be closely monitored and a System with a sustained value of over 50 is probably lacking in System Memory. Again, it is normal for this value to spike occasionally, especially if the other Memory counters do not show a lack of System Memory.

COUNTER: Pages Input per second / Page Reads per second
TYPE: Calculated
USAGE:
$Memory_PIps = New-Object Diagnostics.PerformanceCounter("Memory", "Pages Input/sec")
$Memory_PRps = New-Object Diagnostics.PerformanceCounter("Memory", "Page Reads/sec")
[math]::Round ($Memory_PIps.NextValue() / $Memory_PRps.NextValue(),2)
MEANING: The average of Memory\Pages Input/sec divided by average of Memory\Page Reads/sec gives the number of pages per disk read. This value should not generally exceed five pages per second. A value greater than five indicates that the system is spending too much time paging and requires more memory (assuming that the application has been optimized).
GOTCHA:
THRESHOLD: Sustained value of 5 or more.

Some other interesting counters I will not be covering in detail:

Memory\Page Reads/sec
Memory\Page Writes/sec
Paging File(_total)\% Usage
and so on.

In the next blog I will cover Disk counters.

In this series:
BLOG 1: PerfCounters infrastructure
BLOG 2: PerfCounters Raw vs. Formatted values
BLOG 3: PerfCounters, fetching the values
BLOG 4: PerfCounters, CPU perf data
BLOG 5: PerfCounters, Memory perf data
BLOG 6: PerfCounters, Disk/IO perf data
BLOG 7: PerfCounters, Network and Contention perf data

Windows PerfCounters and Powershell - CPU perf data

So far, I talked of WMI, CIM, WQL, System.Diagnostics.PerformanceCounterCategory, perf-counter data organization and flavour. Now it's time to look at some performance counters I deem important for my use-case more closely.
Note: List of available Counters for Get-Counter command
Get-Counter -ListSet * | Sort-Object CounterSetName | Format-Table CounterSetName

Basic concepts:

I will introduce basic concepts of Processor, Core and CPU now to help you follow the text. Let us use this convention:

• "Processor" is a piece of hardware you connect to a slot on the motherboard.
• "Physical Core" is a physical computing unit built into the "Processor".
• "Virtual Core" is a virtual computing unit built on top of "Physical Core" (i.e. HT is ON).
• "CPU" is a computing unit inside the "Processor", either physical or virtual.

Putting concepts to work

Now lets calculate number of CPUs for my laptop:
PS > ((Get-CimInstance -Namespace root/CIMV2 -ClassName CIM_Processor).NumberOfLogicalProcessors | Measure-Object -Sum).Sum

4
Note: Many other counters fail for some HW configuration and/or OS! Be sure to check.
Note: HT is ON on my dual-core laptop and no cores are parked so to get number of Physical cores:
PS > ((Get-CimInstance -Namespace root/CIMV2 -ClassName CIM_Processor).NumberOfCores | Measure-Object -Sum).Sum

2
Note: There are many ways to collect this info:
PS > (Get-CimInstance Win32_ComputerSystem).NumberOfLogicalProcessors
PS > ((New-Object Diagnostics.PerformanceCounterCategory("Processor Information")).GetInstanceNames() | ?{$_ -match "^(\d{1}),(\d{1})"} | Measure-Object -Sum).Count
Note: RegEx expression is matching "Number,Number" Instances only (See previous blog about instances).

It is not obvious when working with 1 NUMA node/Slot, but the -Sum might refer to Sum of CPUs per Slot, depending on RegEx.

Before starting on Counters, let me stress that the measurements at the system, process and thread level in Windows are based on a sampling methodology thus the data gathered is subject to typical sampling errors like:

• accumulating a "sufficient" number of sample observations to be able to make a reliable statistical inference, i.e. the sampling size

and

• ensuring that there aren’t systemic sources of sampling error that causes results to be under or over-sampled as I will demonstrate shortly.

As of W2K8, the trends are changing towards event driven measurement for CPU utilization which, although more sane and accurate, poses its own set of challenges (say, a clock drift across multiprocessor cores when they are not resynchronized periodically and so on). To compensate for drift, new PerfMon/ResMon work by measuring CPU load in real time using event oriented measurement data gathered by the OS Scheduler each time a context switch occurs.
A context switch occurs in Windows whenever the processor switches its execution context to run a different thread (see more below). Context switches also occur as a result of high priority Interrupt Service Routines (ISRs) as well as the Deferred Procedure Calls (DPCs) that ISRs schedule to complete the interrupt processing. Starting in Windows 6 (Vista/2008), the OS Scheduler began issuing RDTSC instructions to get the internal processor clock each time a context switch occurs. I will talk of context switching and DPC counters in a short while. For more details please see this excellent blog post.

System CPU counters:

First counter I want to talk about is Processor Queue Length. Immediately a Linux users observes that there is no "System load" counter on Windows. This is because Windows OS is Thread based as opposed to Linux which is Process based. This simply means that, in Windows, an execution thread is a basic unit of execution (thus basis for collecting usage statistics too) and a process acts as a container for threads. As simple as it may seem, this actually poses a lot of challenges since one has to start aggregating data about running processes from Threads counters and work his way up. I will talk about this in detail in final blog. So, the WMI counter mimicking Linux "System load" best is, IMO, Processor Queue Length:

PS > Get-Counter '\System\Processor Queue Length'

Timestamp                 CounterSamples                                                      
---------                 --------------                                                      
23.10.15. 10:34:10        \\server_name\system\processor queue length : 1                                          

However, this is slooooow (although subsequent calls return much faster):

PS > Measure-Command { Get-Counter '\System\Processor Queue Length' }

TotalSeconds      : 4.2961321

PS > Measure-Command { Get-Counter '\System\Processor Queue Length' }

TotalSeconds      : 1.007445

So, as described in previous blog, I use System.Diagnostics class to fetch this value:

PS > Measure-Command { New-Object Diagnostics.PerformanceCounter("System", "Processor Queue Length")}

TotalSeconds      : 2.0006457

PS > Measure-Command { New-Object Diagnostics.PerformanceCounter("System", "Processor Queue Length")}

TotalSeconds      : 0.000643

Now, put this into a variable and simply call NextValue():

PS > $System_ProcQL = New-Object Diagnostics.PerformanceCounter("System", "Processor Queue Length")
PS > $System_ProcQL.NextValue()
0
PS > $System_ProcQL.NextValue()
10

The value obtained is for all of the CPU's so you need to calculate the number of CPU's to be your divider and obtain the real value:
$SystemLoad = $System_ProcQL.NextValue() / $totCPU

In-depth description of System Counters important for my use-case:

COUNTER: System\Processor Queue Length
TYPE: Instantaneous
USAGE: New-Object Diagnostics.PerformanceCounter("System", "Processor Queue Length") / ((Get-CimInstance -Namespace root/CIMV2 -ClassName CIM_Processor).NumberOfLogicalProcessors | Measure-Object -Sum).Sum
MEANING: Number of threads per CPU that are ready for execution but can't get CPU cycles for whatever reason thus waiting in OS Scheduler queue. Since Windows have one Scheduler queue, I divide this value with total number of computation units (i.e. CPUs). The actual mechanics is that when Counter value is requested a measurement function traverses the Scheduler Ready Queue and counts the number of threads waiting for an available CPU.
GOTCHA: Even on idle system there can be significant number of threads running on schedule that can bump this number very high. Say you have 4 CPU box and processes fetching values for 100 counters, 10 samples every 1 second. All of these sample requests will lay sleeping for 1 second (thus the Processor Queue Length value will be low) and then all will wake up at the same timer event (clock interrupt) causing Processor Queue Length to spike although there is no real load on the system. It's even worse if your thread(s) is of high priority as it will get executed sooner than the user threads thus pushing Processor Queue Length number very very high. This leads to disproportionate number of Ready Threads waiting for cycles, even (or especially) when the processor itself is not very busy overall. So tip 1 would be to check if CPUs are really busy or not.
THRESHOLD: Pending on above, it is hard to tell what the threshold value is but most people seem to agree it's "sustained value of 2 or more" with CPU utilization of 85%+. This combination tells us we can benefit from adding more CPUs.

COUNTER: System\Context Switches/sec
TYPE: Instantaneous
USAGE:
$System_CSpS = New-Object Diagnostics.PerformanceCounter("System", "Context Switches/sec")
$System_CSpS.NextValue()
MEANING: Context switching happens when a higher priority thread pre-empts a lower priority thread that is currently running or when a high priority thread blocks. High levels of context switching can occur when many threads share the same priority level. This often indicates that there are too many threads competing for the processors on the system. If you do not see much processor utilization and you see very low levels of context switching, it could indicate that threads are blocked (link).
GOTCHA: The number obtained is system-wide! To report the total number of context switches generated per second by all threads use the Thread(_Total)\Context Switches/sec counter (Category((Instance)\Counter):
New-Object Diagnostics.PerformanceCounter("Thread", "Context Switches/sec", "_Total")
THRESHOLD: Context switching rates in excess of 15,000 per second per CPU. The remedy would be to reduce the number of threads and queue more at the application level. This will cause less context switching, and less context switching is good for reducing CPU load.

In-depth description of CPU Counters important for my use-case:

Note: "Processor Information" category, besides overall _Total, has instances for Slot/NUMA node (0,_Total, n,_Total) while "Processor" category gives just _Total for all CPUs as defined above.
Gotcha: On single slot machines, "Processor" category will give info for all the CPUs while on machines with multiple slots, it will give info on just the Physical cores :-/
Thus, if InstanceName is _Total, both yield the same value.

COUNTER: Processor Information(_Total)\% Processor Time, Processor(_Total)\% Processor Time
TYPE: Sample, Instance
USAGE:
$InstanceName = "_Total"
$PI_PT = New-Object Diagnostics.PerformanceCounter("Processor Information", "% Processor Time")
$PI_PT.InstanceName = $InstanceName
$null = $PI_PT.NextValue()
--or--
Get-Counter -Counter "\Processor Information(_Total)\% Processor Time"
Get-Counter -Counter "\Processor(_Total)\% Processor Time"
MEANING: Primary indicator of CPU activity. High values many not necessarily be bad. However, if the other processor-related counters are increasing linearly such as Processor\% Privileged Time or System\Processor Queue Length, high CPU utilization may be worth investigating.
GOTCHA: If this counter is around threshold value, starting new processes will only lead to increased value of Processor Queue Length but the work done will remain the same. Look for some more counters that I'm about to describe in relation to this one.
THRESHOLD: Folks seem to agree on ~85%. Low CPU utilization with sustained Processor Queue Length value of 2 or higher is indicator that requests for CPU time arrive randomly and threads demand irregular amounts of time from the CPU. This means that the processor power is not a bottleneck but that the application threading logic should be improved.

COUNTER: Processor Information(_Total)\% Privileged Time
TYPE: Sample, Instance
USAGE:
$InstanceName = "_Total"
$PI_PPT = New-Object Diagnostics.PerformanceCounter("Processor Information", "% Privileged Time")
$PI_PPT.InstanceName = $InstanceName
$null = $PI_PPT.NextValue()
MEANING: Counter indicates the percentage of non-idle CPU time spent in privileged mode, i.e. calls to OS functions (file or network I/O, memory allocation...). Basically, this is unrestricted mode allowing direct access to hardware and all memory.
GOTCHA:
THRESHOLD: Folks seem to agree on consistently being over 75%.

COUNTER: Processor Information(_Total)\% User Time
TYPE: Sample, Instance
USAGE:
$InstanceName = "_Total"
$PI_PUT = New-Object Diagnostics.PerformanceCounter("Processor Information", "% User Time")
$PI_PUT.InstanceName = $InstanceName
$null = $PI_PUT.NextValue()
MEANING: Percentage of non-idle CPU time spent in user mode. User mode is a restricted processing mode designed for applications, environment subsystems, and integral subsystems.
GOTCHA: Processor Information(_Total)\% Privileged Time +
Processor Information(_Total)\% User Time = Processor Information(_Total)\% Processor Time.
THRESHOLD: Depends on previous two counters.

COUNTER: Processor Information(_Total)\% Idle Time
TYPE: Sample, Instance
USAGE:
$InstanceName = "_Total"
$PI_PIT = New-Object Diagnostics.PerformanceCounter("Processor Information", "% Idle Time")
$PI_PIT.InstanceName = $InstanceName
$null = $PI_PIT.NextValue()
MEANING: Counter indicates the percentage of time OS idle thread was consuming cycles. On Windows, there is a special Kernel thread that consumes cycles when CPU is idling. Counting cycles consumed by this thread gives Idle CPU time.
GOTCHA: Processor Information(_Total)\% Processor Time + Processor Information(_Total)\% Idle Time = 100%
THRESHOLD:

COUNTER: Processor Information(_Total)\% Priority Time
TYPE: Sample, Instance
USAGE:
$InstanceName = "_Total"
$PI_PPRIOT = New-Object Diagnostics.PerformanceCounter("Processor Information", "% Priority Time")
$PI_PPRIOT.InstanceName = $InstanceName
$null = $PI_PPRIOT.NextValue()
MEANING: CPU utilization by high priority threads.
GOTCHA: Kernel scheduler can, on occasion, wake up low priority threads sleeping for "long" time assigning them much more slices on CPU than one would expect given the (low)priority. This, in turn, blocks high-priority threads from execution which is never an expected behaviour. I would look at this value in relation to Context switches/second to determine what's going on.
THRESHOLD:

COUNTER: Processor Information\Interrupts/sec
TYPE: Sample, Instance
USAGE:
$InstanceName = "_Total"
$PI_INTPS = New-Object Diagnostics.PerformanceCounter("Processor Information", "Interrupts/sec")
$PI_INTPS.InstanceName = $InstanceName
$null = $PI_INTPS.NextValue()
MEANING: Number of hardware interrupts per second. This value is the indicator of the activity of devices that generate interrupts, such as network adapters.
GOTCHA: See next counter.
THRESHOLD:

COUNTER: Processor Information\% Interrupt Time
TYPE: Sample, Instance
USAGE:
$InstanceName = "_Total"
$PI_PINTT = New-Object Diagnostics.PerformanceCounter("Processor Information", "% Interrupt Time")
$PI_PINTT.InstanceName = $InstanceName
$null = $PI_PINTT.NextValue()
MEANING: The value indicates the percentage of time CPUs spend receiving and servicing hardware interrupts. This value is an indirect indicator of the activity of devices that generate interrupts, such as network adapters.
GOTCHA: Mass increase in Processor Information\Interrupts/sec and Processor Information\% Interrupt Time indicates potential hardware problems.
THRESHOLD:

COUNTER: Processor Information\DPCs Queued/sec
TYPE: Sample, Instance
USAGE:
$InstanceName = "_Total"
$PI_DPCQPS = New-Object Diagnostics.PerformanceCounter("Processor Information", "DPCs Queued/sec")
$PI_DPCQPS.InstanceName = $InstanceName
$null = $PI_DPCQPS.NextValue()
MEANING: Overall rate at which deferred procedure calls ("SW interrupts") are added to the processor's DPC queue. This property measures the rate at which DPCs are added to the queue, not the number of DPCs in the queue.
GOTCHA: This is NOT the number of SW interrupts in the queue!
THRESHOLD:

COUNTER: Processor Information\DPC Time
TYPE: Sample, Instance
USAGE:
$InstanceName = "_Total"
$PI_PDPCT = New-Object Diagnostics.PerformanceCounter("Processor Information", "% DPC Time")
$PI_PDPCT.InstanceName = $InstanceName
$null = $PI_PDPCT.NextValue()
MEANING: Percentage of time that the processor spent receiving and servicing deferred procedure calls (SW interrupts) during the sample interval. They are counted separately and are not a component of the interrupt counters.
GOTCHA: This property is a component of PercentPrivilegedTime because DPCs are executed in privileged mode.
THRESHOLD:

Other useful counters I would look into in case of trouble are C1/C2/C3TransitionsPerSec. There is a huge penalty waking up CPU from C3 low power state to C2 low power state and considerable penalty transitioning from C2 to C1. So if box is choking and CPUs are idling, look here. And make sure ParkingStatus for each CPU is 0 ;-)
Example: (Physical) CPU 9 in Slot 7 was asleep:

PS > GCim Win32_PerfFormattedData_Counters_ProcessorInformation
...
Name                        : 7,9
AverageIdleTime             : 100
C3TransitionsPersec         : 64
ClockInterruptsPersec       : 64
IdleBreakEventsPersec       : 64
InterruptsPersec            : 64
PercentC3Time               : 99
...

Basically, only processing timer events.

There are also combinations of counters that can point out problems like Processor\% DPC Time, % Interrupt Time and % Privileged Time. If Interrupt Time and DPC time are a large portion of Privileged Time, the kernel is spending significant amount of time processing (most likely) I/O requests. In some cases performance can be improved by configuring interrupts and DPC affinity to a small number of CPUs on a multiprocessor system, which improves cache locality. In other cases, it works best to distribute the interrupts and DPCs among many CPUs, so as to keep the interrupt and DPC activity from becoming a bottleneck.

In the next blog I will cover Memory performance counters.

In this series:
BLOG 1: PerfCounters infrastructure
BLOG 2: PerfCounters Raw vs. Formatted values
BLOG 3: PerfCounters, fetching the values
BLOG 4: PerfCounters, CPU perf data
BLOG 5: PerfCounters, Memory perf data
BLOG 6: PerfCounters, Disk/IO perf data
BLOG 7: PerfCounters, Network and Contention perf data

Windows PerfCounters and Powershell - Raw vs. Formatted values

How to interpret Raw data from Windows performance counters.

Tip: An alias for Get-CimInstance is GCim and alias for Get-WmiObject is GWmi.

In the first blog post, I covered what WMI/CIM is and how to get info from there. Last I talked about was RawData counters:
Get-CimInstance -Class Win32_PerfRawData_PerfOS_Processor

Name : _Total
...
PercentIdleTime : 78061457390

Understanding RawData:

By itself, a RawData value is a sample but important thing is to determine what concrete sample value actually is and how to convert it to a form we understand. In this example, MSDN tells us PercentIdleTime is a counter of type 542180608:
  PercentIdleTime
        Data type: uint64
        Access type: Read-only
        Qualifiers: DisplayName ("% Idle Time") , CounterType (542180608) , DefaultScale (0) , PerfDetail (400)

Bear in mind, most of RawData counters need 2 samples to produce humanly readable result thus now we need a formula to convert Raw counter values into something meaningful.
Numeric-to-Name conversion (542180608 -> PERF_100NSEC_TIMER) of counter type values is listed in this MSDN page. The actual formula is then located under entries listed here as described in this page:

(N1 - N0) / (D1 - D0) x 100, where the denominator (D) represents the total elapsed time of the sample interval, and the numerator (N) represents the portions of the sample interval during which the monitored components were active.

This translates to:

      $Val =
      (
      (PercentIdleTime_Sample[n] - PercentIdleTime_Sample[n-1])
      /
      (Timestamp_Sys100NSSample[n] - Timestamp_Sys100NSSample[n-1])
      ) *100

Note: Although formula is correct according to documentation, it usually summarizes result over all CPU's (say, when fetching CPU utilization per process/thread) thus the result will most likely be well over 100% on modern boxes. In such case, we need to divide samples with total number of CPU's:
      $Val =
      (
      (PercentIdleTime_Sample[n] - PercentIdleTime_Sample[n-1])
      /(
      (Timestamp_Sys100NSSample[n] - Timestamp_Sys100NSSample[n-1])
      *$TotProc)
      ) *100

With the Class Win32_PerfRawData_PerfOS_Processor it's easy. It has Instance named _Total and when you apply formula to its values you will get proper result. I will talk of this more in next blogs.

Conclusion:

So, why Formatted and Raw counters? After all, Formatted data is coming from Raw counters. First, we have to remember that Raw counters are used for collecting N samples as the naked number obtained is meaningless. So, let's say that in above example I asked for CPU usage by certain process that I only just started. Formatted counter will either have 0 or NaN value in it while Raw counter will produce some number given that sampling time is usually 100ns. Well, you might say, it's the same as "get formatted sample | check if it is a number | no? -> take another formatted sample" and you'd be right. But you should take into account the rounding happening in calculating Formatted values internally as well as in your script (check that data type of Formatted counters is usually UINT!) and also the latency involved in WMI provider populating formatted data counters.

If you are set on using Raw counters, bear in mind you need formulas for transforming Samples into Values.

All said, it is actually a question of choice whether to use one type or the other as you will see in the script I'll be describing in the final blog.

Examples of calculating Value from Counter type and samples @Sysinternals and @MSDN.

Next blog will deal with various ways to obtain performance data.

In this series:
BLOG 1: PerfCounters infrastructure
BLOG 2: PerfCounters Raw vs. Formatted values
BLOG 3: PerfCounters, fetching the values
BLOG 4: PerfCounters, CPU perf data
BLOG 5: PerfCounters, Memory perf data
BLOG 6: PerfCounters, Disk/IO perf data
BLOG 7: PerfCounters, Network and Contention perf data