tag:blogger.com,1999:blog-78508469401985405922024-02-20T02:59:58.321+01:00Tonči GrginTončihttp://www.blogger.com/profile/10650361408804728027noreply@blogger.comBlogger16125tag:blogger.com,1999:blog-7850846940198540592.post-31177057258854824362021-09-13T10:14:00.004+02:002021-12-09T15:34:45.629+01:00Bokashi composting, hands on guide[WIKIPEDIA] Bokashi is a process that converts food waste and similar organic matter into a <b>soil amendment</b> which adds nutrients and improves soil texture. It differs from traditional composting methods in several respects. The most important are:
<ul>
<li>The input matter is fermented by specialist bacteria, not decomposed.</li>
<li>The fermented matter is fed directly to field or garden soil, without requiring further time to mature.</li>
<li>As a result, virtually all input carbon, energy and nutrients enter the soil food web, having been neither emitted in greenhouse gases and heat nor leached out.</li>
</ul>
Put it simply, you are preparing food for worms, not the soil itself so this method is suitable for people with gardens.
<table>
<tr>
<th><b>PROS</b> </th>
<th><b>CONS</b> </th>
</tr>
<tr>
<td>No hustle.</td>
<td>No soil produced.</td>
</tr>
<tr>
<td>No excessive CO2 production.</td>
<td>Pungent smell.</td>
</tr>
<tr>
<td>Easy to get right.</td>
</tr>
<tr>
<td>Does not attract mice/rats/harmful insects.</td>
</tr>
</table>
<br><br>
<h2>Chemistry</h2>
<br>
Please see this <a href="https://en.wikipedia.org/wiki/Bokashi_%28horticulture%29" target="_blank">Wikipedia article</a> for details.
<br><br><br>
<h2>The HOW</h2>
<br>
You need to harvest and grow lactobacillus. For this you can basically use any organic food produce like cabbage or rice. I will give receipt for rice (any rice but paraboiled should do):<br>
<ul>
<li>0.5l of water (non-chlorinated)</li>
<li>70-80g of rice (any except paraboiled, the more the better, rice can be reused for cooking)</li>
<li>15g of sugar (any)</li>
<li>5g of salt (any)</li>
</ul>
<br>
<b>GOTCHA:</b>
If you do not have access to non-chlorinated water, please boil tap water and leave it for a day or two before using it.<br>
There is no point in using any special rice, sugar or salt. This is urban myth. I'm guessing paraboiled rice is not fit for this being that most of the useful organisms will die during pre-boiling.<br>
Quantities scale linearly so to make larger batch, just multiply with same number.<br>
When weighing ingridients, it is not important to be exactly precise so, say, rice can be more. But, it is better to have 1g of sugar more than 1g of salt. Meaning, rice 85g, sugar 16g, salt 4g is perfectly good receipt.<br><br>
<b>Equipment:</b>
<ul>
<li>Plastic bottle with <b>loose</b> cap or <a href="https://www.amazon.com/Twin-Bubble-Airlock-Wine-Making/dp/B008ACWSZU/ref=pd_bxgy_img_2/133-4759852-4278956?pd_rd_w=FF7Zc&pf_rd_p=c64372fa-c41c-422e-990d-9e034f73989b&pf_rd_r=HKTQ7KVPH1XVECP9KMFQ&pd_rd_r=ec246e79-707d-4f68-a194-aba4d155e583&pd_rd_wg=Kmi49&pd_rd_i=B008ACWSZU&psc=1" target="_blank">fermentor cap</a> such as this one. It is important that a cap is loose because of CO2 buildup while lactobacteria processes sugar to grow.</li>
<li>Funnel with sieve to prevent rice grains getting into solution. Impurities will clog nozzle.</li>
<li>ph strips covering acidic spectrum (say 1-5).</li>
<li>A bowl to mix rice in.</li>
<li>Spray bottle.</li>
<li><a href="https://www.bokashibin.com" target="_blank">Bokashi bin</a>.</li>
</ul>
<b>The process:</b>
<ul>
<li>Pour dechlorinated water and rice into bowl. Grind rice grains against each other with your hands until the water is cloudy white. The whiter the better. This should not take more than couple of minutes.</li>
<li>Pour half of the mixture into the bottle using funnel.</li>
<li>Add salt and sugar into bottle.</li>
<li>Pour the rest of the mixture into bottle and close it.</li>
<li>Shake the bottle vigorously and place into dark, warm corner.</li>
<li>Rinse the bowl and funnel under running tap water.</li>
</ul>
When it occurs to you, shake the bottle and, if needed, release CO2 from it. This can be once a day or so. Lactobacillus is photo/aerophobic but this just means it will grow slower when aerated/insolated not die altogether.<br>
After 2-3 days, depending on how warm your corner is, you can start checking if there is enough lactobacillus in your mixture. Open the bottle and place pH strip into it for couple of seconds. You are looking for pH value of 4 and less (I usually wait for 3). This is becaus of two things; botulism bacteria naturally occuring in rice die off at about 4.5 and values of 3 and lower indicate "runaway" reaction mening we have almost pure lactobacteria culture. The mixture smells sour.<br>
When the mixture is prepared, pour into spray-bottle, it is ready to use.<br>
<br>
<b>Composting:</b><br>
Open the bin, spray some of the liquid on the sides, put the food leftovers in it, press hard, close. Repeat until bin is full. You can add a spoon of sugar occasionally.<br>
Once the bin is full, leave for some 2 weeks to ferment. Regardless of urban myths, this is not mandatory as the fermentation process will continue even after you bury the contents into ground.<br>
After the maturation, please bury the contents into a hole in earth or dispose into composter. Personally, I have 2 boxes made of thick wire so they have contact with ground attracting worms. Make sure the mixture in composter is always kept wet. This is good for worms and keeps mice away.<br>
Once the bin is empty, you can wash it. I use rain water to preserve what remained of lactobacteria but tap water is also fine. Drain fully for a day or so especially if you washed it with tap water.</br>
The soil is ready to be used when there are no discernible pieces of food in it any more. Take this with grain of salt, of course you can see some pieces of food, depending on type of food and how big the pieces were. This does not make soil unusable in any way.
<br>
<br>
<h2>Finer points of the process and what to watch for:</h2><br>
It is important to press the mixture in the bin firmly to get rid of excess air. This will also crush, for example, eggshells and similar. The pieces should be small enough not to trap air in them. In normal language; I chop orange/lemon peels into several pieces instead of putting citrus dome into bin. On the other hand, cabbage, salad, fish remains etc. I do not cut at all.<br>
It is OK to have pieces of raw flesh in the bin. Once fermented, it will not attract rats and such.<br>
Do not make excessive amount of bokashi mixture. After processing the sugar in it, bacteria will start to die. Generally, if I have mixture for more than 10 days, I add a few grams of sugar directly to spray bottle and shake.<br>
You can put anything but the bones and nut shells into the bin. Lactobacteria will eat through even fish bones and egg shells.<br>
If you ever wondered why the pipe on the bin, it is because liquid is produced during fermentation. You need to relieve this liquid from the bin so it does not soak the contents. Mind you, the smell is <b>pungent</b>! The liquid obtained can be diluted and used as fertilizer or simply poured into pipes, earth, compost bin, septic tank and so on. The warmer the place for the bin, the more liquid is produced! Bacteria count drops by half with each degree Celsius drop in temperature.<br>
<b>GOTCHA:</b><br>
Harmful mold can develop in the bin if you fail to drain it! If you open the lid and see black mold on top, take the bin somewhere far and pour the contents over the ground. Do not bury it as we want the elements to do the desinfection for us. This goes for the bin too. On the other hand, if you see webby-white mold in the bin, that's perfectly fine, even desirable.<br>
Earth worms are not very active below 20C so it will take more time for them to process the food in winter.<br>
<br>
<h2>The RESULT</h2>
Dense, black soil-like matter teaming with worms and other living things. It can significantly improve soil quality in your garden, flower pots etc.
Tončihttp://www.blogger.com/profile/10650361408804728027noreply@blogger.com0tag:blogger.com,1999:blog-7850846940198540592.post-10850757081872552322018-06-01T16:57:00.002+02:002018-06-01T16:57:45.278+02:00MySQL Cluster 7.6GA in the Cloud: the RPM/YUM platformIn this series of blogs I will do my best to demonstrate how to set up and run Cluster in Cloud environment by hand and by utilizing <strong>MCC</strong>. For detailed configuration, I will use <strong>MCC</strong> (Auto-installer). Some of the information regarding this setup is provided in <a href="http://mikaelronstrom.blogspot.com/2017/04/setting-up-mysql-cluster-on-amazon-cloud.html">post by Mikael</a>. Information regarding Auto-installer is available in our documentation, HTML help files in share/MCC distribution directory and in <a href="http://toncigrgin.blogspot.com/2018/02/oracle-mysql-cluster-76-includes-new.html">my blog post</a>.<br/>
<br/>
<h3>
Cloud setup</h3>
It might come as surprise but for initial testing any instance should do. I started with 1CPU/1GB RAM ones. The target topology was: <br/>
<ul>
<li> Host #1: Management node and Cluster client tools.</li>
<li> Host #2: Multi-threaded DATA node.</li>
<li> Host #3: Multi-threaded DATA node.</li>
<li> Host #4: SQL node (MySQL server)/API node.</li>
<li> Host #5: SQL node (MySQL server)/API node.</li>
</ul>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjqsn6_jhTQ13KI-fvkFiKMMUK67dqVwFtPCKJdiugZ2ksaIcTbYi51SNPIXddNwE3a4gt4uESdDLYYu1fGnM3HC7q41d94lrKHUNWBbMaHj8ahjxZTxPOYiNbObVnUA_HoJ-aHNQGbmyPm/s1600/RH-5-node-cluster-processes.PNG" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjqsn6_jhTQ13KI-fvkFiKMMUK67dqVwFtPCKJdiugZ2ksaIcTbYi51SNPIXddNwE3a4gt4uESdDLYYu1fGnM3HC7q41d94lrKHUNWBbMaHj8ahjxZTxPOYiNbObVnUA_HoJ-aHNQGbmyPm/s1600/RH-5-node-cluster-processes.PNG" data-original-width="296" data-original-height="425" /></a>
<br/><br/>
<strong>Cloud instances:</strong><br/>
At your provider of choice, set up topology as described. Most of the providers prefer using keys so set the security and save the key to your local box. After that, set the firewall rules. Here is the list of ports with explanation for this example:<br/>
<ul>
<li> <strong>Port#1186</strong>: Port used by management node for communication. It is also used by <strong>MCC</strong> to connect to Management node host and gather status from/shut down the Cluster. Open it on host #1.</li>
<li> <strong>Port#11860</strong>: "ServerPort" used by DATA nodes to communicate with each other. Open it on hosts 2 and 3.</li>
<li> <strong>Port#3306</strong>: Standard MySQL server port. You need to open it on hosts 4 and 5.</li>
</ul>
To open ports via script, please check <a href="http://mikaelronstrom.blogspot.com/2017/04/setting-up-mysql-cluster-in-oracle-bare.html">Mikael's post</a>.<br/>
<strong>GOTCHA:</strong> Port 3306 on hosts 4 and 5 could already be taken by existing MySQL server installation which you might remove first or change this port to, say, 3316.<pre style="font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; color: #000000; background-color: #eee;font-size: 12px;border: 1px dashed #999999;line-height: 14px;padding: 5px; overflow: auto; width: 100%"><code>$ sudo netstat -tulpn | grep LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 929/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 894/master
tcp6 0 0 :::3306 :::* LISTEN 983/mysqld
tcp6 0 0 :::22 :::* LISTEN 929/sshd
tcp6 0 0 ::1:25 :::* LISTEN 894/master
$ sudo /sbin/service mysqld stop
Redirecting to /bin/systemctl stop mysqld.service
$ sudo netstat -tulpn | grep LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 929/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 894/master
tcp6 0 0 :::22 :::* LISTEN 929/sshd
tcp6 0 0 ::1:25 :::* LISTEN 894/master
</code></pre><br/>
<strong>GOTCHA:</strong> Usually, MySQL Cluster DATA node will look for available port and auto-assign <strong>ServerPort</strong>. However, it is most likely that your Cloud firewall will block everything but specific ports so this value is needed.<br/>
If you have localhost deployment or you have two or more DATA nodes on same host, do not assign this value.<br/><br/>
After dealing with security and choosing your preferred OS, make note of both public and private IP addresses. Public one we will use to control the processes from <strong>MCC</strong> while private ones will be used by processes for inter-communication bypassing DNS service:<br/>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEitoZbuPK4DAX8I2imiDVn6UpbVTnWTyfHcCgSaFDhP8gRc-DpnbQEmT6Fk4r-m_eJVY1UI8sWP5rPUwvkgQh0jkLfCyU6EvFZC4-_H5lfjxlsKiGHn0yEqDRE4bV7JfhAru8dL8cS_FGg2/s1600/RH-5-node-cluster-hosts.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEitoZbuPK4DAX8I2imiDVn6UpbVTnWTyfHcCgSaFDhP8gRc-DpnbQEmT6Fk4r-m_eJVY1UI8sWP5rPUwvkgQh0jkLfCyU6EvFZC4-_H5lfjxlsKiGHn0yEqDRE4bV7JfhAru8dL8cS_FGg2/s1600/RH-5-node-cluster-hosts.png" data-original-width="987" data-original-height="375" /></a><br/><strong>NOTE:</strong> The "Open..." refers to "OpenFirewall" field of Host configuration.<br/>
<br/>
<h3>
Preparing the host(s)</h3>
If you're using <strong>MCC</strong>, you can configure all the processes and options and create directories/deploy configurations by clicking DEPLOY button on last page.<br/>
If you want to do things by hand, please, in your HOME, prepare the directories:<pre style="font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; color: #000000; background-color: #eee;font-size: 12px;border: 1px dashed #999999;line-height: 14px;padding: 5px; overflow: auto; width: 100%"><code>mkdir ndb (or MySQL_Cluster)
cd ndb
mkdir data
cd ..</code></pre>Place the config.ini file in MGMT node(s) DATADIR you just created (~/ndb/data), place my.cnf in DATADIR on hosts 4 and 5. I will talk of config.ini more later.<br/><br/>
Prepare the utilities:<pre style="font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; color: #000000; background-color: #eee;font-size: 12px;border: 1px dashed #999999;line-height: 14px;padding: 5px; overflow: auto; width: 100%"><code>sudo yum update
sudo yum install curl wget unzip zip</code></pre>
Get the RPM and extract: <pre style="font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; color: #000000; background-color: #eee;font-size: 12px;border: 1px dashed #999999;line-height: 14px;padding: 5px; overflow: auto; width: 100%"><code>wget https://dev.mysql.com/get/mysql57-community-release-el7-11.noarch.rpm
sudo rpm -ivh mysql57-community-release-el7-11.noarch.rpm</code></pre>
Check things: <pre style="font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; color: #000000; background-color: #eee;font-size: 12px;border: 1px dashed #999999;line-height: 14px;padding: 5px; overflow: auto; width: 100%"><code>rpm -qa | grep mysql mysql57-community-release-el7-11.noarch</code></pre>
The default values in MySQL repository file enable MySQL server but we need Cluster so change "enabled" value using your favorite editor:<pre style="font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; color: #000000; background-color: #eee;font-size: 12px;border: 1px dashed #999999;line-height: 14px;padding: 5px; overflow: auto; width: 100%"><code>cat /etc/yum.repos.d/mysql-community.repo
</code></pre>
<pre><samp>[mysql57-community]
name=MySQL 5.7 Community Server
baseurl=http://repo.mysql.com/yum/mysql-5.7-community/el/7/$basearch/
enabled=1 <<<<< 0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mysql
[mysql-cluster-7.6-community]
name=MySQL Cluster 7.6 Community
baseurl=http://repo.mysql.com/yum/mysql-cluster-7.6-community/el/7/$basearch/
enabled=0 <<<<< 1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mysql
</pre></samp> or by issuing:<pre style="font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; color: #000000; background-color: #eee;font-size: 12px;border: 1px dashed #999999;line-height: 14px;padding: 5px; overflow: auto; width: 100%"><code>sudo yum-config-manager --disable mysql57-community
sudo yum-config-manager --enable mysql-cluster-7.6-community</code></pre>
Make sure everything checks out:<pre style="font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; color: #000000; background-color: #eee;font-size: 12px;border: 1px dashed #999999;line-height: 14px;padding: 5px; overflow: auto; width: 100%"><code>cat /etc/yum.repos.d/mysql-community.repo</code></pre>
<pre><samp>[mysql-connectors-community]
...
[mysql57-community]
name=MySQL 5.7 Community Server
baseurl=http://repo.mysql.com/yum/mysql-5.7-community/el/7/$basearch/
enabled=0
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mysql
...
[mysql-cluster-7.6-community]
name=MySQL Cluster 7.6 Community
baseurl=http://repo.mysql.com/yum/mysql-cluster-7.6-community/el/7/$basearch/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-mysql</pre></samp>
Installing Host#1 for the NDB management server is now easy:<pre style="font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; color: #000000; background-color: #eee;font-size: 12px;border: 1px dashed #999999;line-height: 14px;padding: 5px; overflow: auto; width: 100%"><code>sudo yum install mysql-cluster-community-management-server
</code></pre>
To properly stop MySQL Cluster or to play with it from <strong>MCC</strong>, you will need <strong>ndb_mgm</strong> client so you can poll for status and execute various commands:<pre style="font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; color: #000000; background-color: #eee;font-size: 12px;border: 1px dashed #999999;line-height: 14px;padding: 5px; overflow: auto; width: 100%"><code>sudo rpm -ivh http://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/e/epel-release-7-11.noarch.rpm
sudo yum install mysql-cluster-community-client</code></pre><pre><samp>ndb_mgm --ndb-connectstring=172.31.24.27:1186
ndb_mgm> show
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=1 (not connected, accepting connect from 172.31.22.116)
id=2 (not connected, accepting connect from 172.31.19.15)
[ndb_mgmd(MGM)] 1 node(s)
id=49 @172.31.24.27 (mysql-5.7.18 ndb-7.6.3)
...</samp></pre>
<strong>Note:</strong> Perl is needed to install MySQL Cluster community package. To install Perl we need access to the EPEL packages.<br/><br/>
Installing the NDB data nodes on Host#2 and #3 is also very easy:<pre style="font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; color: #000000; background-color: #eee;font-size: 12px;border: 1px dashed #999999;line-height: 14px;padding: 5px; overflow: auto; width: 100%"><code>sudo yum install mysql-cluster-community-data-node</code></pre><br/>
MySQL Cluster server package depends on Perl which is not part of a standard Red Hat installation. So on the Host#4 and #5 used for MySQL Server we first need to install support for Perl by installing EPEL packages (Extra Packages for Enterprise Linux) much like we did for MGMT node host (#1). This is performed through the command:<pre style="font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; color: #000000; background-color: #eee;font-size: 12px;border: 1px dashed #999999;line-height: 14px;padding: 5px; overflow: auto; width: 100%"><code>sudo rpm -ivh http://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/e/epel-release-7-11.noarch.rpm</code></pre> You might need to check which package actually exists.<br/><br/>
After installing this, it is straightforward to install the MySQL Server package. This package will also download the MySQL client package that contains the NDB management client package: <pre style="font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; color: #000000; background-color: #eee;font-size: 12px;border: 1px dashed #999999;line-height: 14px;padding: 5px; overflow: auto; width: 100%"><code>sudo yum install mysql-cluster-community-server
</code></pre><br/>
<h3>
Running the Cluster</h3>
You can run Cluster by hand or let the <strong>MCC</strong> deal with it. In any case, the start sequence is MGMT node(s), DATA node(s), SQL node(s) and API node(s). When running by hand, sequence of commands by host are like this:<br/>
<pre style="font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; color: #000000; background-color: #eee;font-size: 12px;border: 1px dashed #999999;line-height: 14px;padding: 5px; overflow: auto; width: 100%"><code> ndb_mgmd -f /home/user/ndb/config.ini --initial --configdir=/home/user/ndb/data --ndb-nodeid=49
</code></pre>
There are many ways in which you can mix parameters, for example:
<pre style="font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; color: #000000; background-color: #eee;font-size: 12px;border: 1px dashed #999999;line-height: 14px;padding: 5px; overflow: auto; width: 100%"><code>ndb_mgmd --configdir=/home/user/ndb/data/ --config-file=/home/user/ndb/data/49/config.ini
</code></pre> and so on, depending on where DATADIR and config.ini are placed.<br/><br/>
Use <strong>ndb_mgm</strong> client to confirm node is up as described above.<br/><br/>
Then start DATA nodes on hosts 2 and 3:
<pre style="font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; color: #000000; background-color: #eee;font-size: 12px;border: 1px dashed #999999;line-height: 14px;padding: 5px; overflow: auto; width: 100%"><code>ndbmtd --connect-string="nodeid=1;172.31.24.27" (Host#2)
ndbmtd --connect-string="nodeid=2;172.31.24.27" (Host#3)
</code></pre> Notice how DATA nodes refer to MGMT node internal IP address and represent themselves with node ID found in config.ini file. It is possible to use external IP too but internal one does not depend on DNS service.<br/><br/>
Use <strong>ndb_mgm</strong> client to confirm nodes are up as described above.<br/><br/>
Next we bootstrap MySQL server and start it:<br/>
<pre style="font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; color: #000000; background-color: #eee;font-size: 12px;border: 1px dashed #999999;line-height: 14px;padding: 5px; overflow: auto; width: 100%"><code>mysqld --defaults-file=/home/user/ndb/data/my.cnf --initialize-insecure
mysqld --defaults-file=/home/user/ndb/data/my.cnf --ndb-connectstring=172.31.24.27
</code></pre>
Since we are preparing test setup, doing this as root is acceptable. However, you will need a lot of SUDO's later.<br/>
<strong>GOTCHA:</strong> Normally, you would never never leave your MySQL setup unprotected! Please observe procedures noted in our <a href="https://dev.mysql.com/doc/refman/5.7/en/data-directory-initialization-mysqld.html">manual</a> and secure your installation:<br/>
<ul>
<li>Initialize the data directory by invoking <strong>mysqld</strong> with the <code>--initialize</code> to let server generate a random initial password for the <code>root@localhost</code> account.</li>
<li>On *nix systems, it is important to make sure that the database directories and files are owned by the <code>mysql</code> account so that the server has read and write access to them. To ensure this, start <strong>mysqld</strong> from the system <code>root</code> account and include the <code>--user</code> option like this:<br />
<code>$ bin/mysqld --initialize --user=mysql</code></li>
<li>To add <code>mysql</code> user/group please use following commands (or <code>addgroup, adduser</code> depending on the Linux flavor):<br/>
<code>$ groupadd mysql</code><br/>
<code>$ useradd -r -g mysql -s /bin/false mysql</code><br/>
Because the user is required only for ownership purposes, not login purposes, the <code>useradd</code> command uses the <code>-r</code> and <code>-s /bin/false</code> options to create a user that does not have login permissions to your server host. Omit these options if your <code>useradd</code> does not support them.</li>
</ul>
<br/>
Use <strong>ndb_mgm</strong> or <strong>MCC</strong> to confirm Cluster is up and running:<br/>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhfzKrdKowFjLf_elpvvlgRGcXZ0NOTh_IUJzd-NkYIq1mUFsmKLn_2-IbTdQ-IX2ayAhB029QuOHevg3EerNmp6FlbODogqSlpkTXvkJgc06SRETxva_gqIevvOVeQkYP8Ea-XH5i7BMpL/s1600/RH-5-node-cluster-started.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhfzKrdKowFjLf_elpvvlgRGcXZ0NOTh_IUJzd-NkYIq1mUFsmKLn_2-IbTdQ-IX2ayAhB029QuOHevg3EerNmp6FlbODogqSlpkTXvkJgc06SRETxva_gqIevvOVeQkYP8Ea-XH5i7BMpL/s1600/RH-5-node-cluster-started.png" data-original-width="746" data-original-height="433" /></a><br/>
<strong>GOTCHA:</strong> By default, <strong>MCC</strong> will set <strong>SharedGlobalMemory</strong> to 384MB which is too much for microinstance so DATA nodes might not start. Lower this value to 128MB or 256MB, whichever works for you. You can also change <strong>LockPagesInMainMemory=0</strong> to <strong>LockPagesInMainMemory=1</strong> to make this problem more visible.<br/><br/>
<strong>config.ini</strong>: Assuming you let <strong>MCC</strong> create configuration and directories for you, ~/ndb/data becomes ~/MySQL_Cluster/NodeID
<pre style="font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; color: #000000; background-color: #eee;font-size: 12px;border: 1px dashed #999999;line-height: 14px;padding: 5px; overflow: auto; width: 100%"><code>#
# Configuration file for ...
#
[NDB_MGMD DEFAULT]
ArbitrationRank=1
HeartbeatIntervalMgmdMgmd=1500
Portnumber=1186
[NDB_MGMD]
NodeId=49
HostName=172.31.24.27
DataDir=/home/user/MySQL_Cluster/49/
Portnumber=1186
[TCP DEFAULT]
SendBufferMemory=2M
ReceiveBufferMemory=2M
[NDBD DEFAULT]
NoOfReplicas=1
ServerPort=11860
CompressedLCP=false
LogLevelStartup=1
EventLogBufferSize=8K
DataMemory=4M
IndexMemory=2M
MinFreePct=5
MaxNoOfAttributes=1000
MaxNoOfTables=128
MaxNoOfOrderedIndexes=128
MaxNoOfTriggers=768
MaxNoOfConcurrentSubOperations=256
StringMemory=25
MaxAllocate=32M
Diskless=false
LockPagesInMainMemory=1
ODirect=false
Arbitration=Default
DiskPageBufferEntries=10
DiskPageBufferMemory=128M
MaxNoOfConcurrentTransactions=4096
MaxNoOfConcurrentOperations=131072
MaxNoOfConcurrentScans=256
FragmentLogFileSize=64M
NoOfFragmentLogFiles=4
InitFragmentLogFiles=SPARSE
RedoBuffer=32M
RedoOverCommitCounter=3
RedoOverCommitLimit=20
MaxNoOfExecutionThreads=2
StopOnError=0
CrashOnCorruptedTuple=true
StartFailRetryDelay=0
MaxStartFailRetries=3
[NDBD]
NodeId=1
HostName=172.31.22.116
DataDir=/home/user/MySQL_Cluster/1/
[NDBD]
NodeId=2
HostName=172.31.19.15
DataDir=/home/user/MySQL_Cluster/2/
[MYSQLD DEFAULT]
AutoReconnect=false
BatchByteSize=16K
BatchSize=256
MaxScanBatchSize=256K
[MYSQLD]
NodeId=53
HostName=172.31.21.152
[MYSQLD]
NodeId=54
HostName=172.31.22.129
[API]
NodeId=231
HostName=172.31.21.152
[API]
NodeId=232
HostName=172.31.22.129
</code></pre>
I have removed a lot of options <strong>MCC</strong> sets in [NDBD DEFAULT] section but feel free to play with them.<br/><br/>
my.cnf file for SQL node on host 4 looks like:
<pre style="font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; color: #000000; background-color: #eee;font-size: 12px;border: 1px dashed #999999;line-height: 14px;padding: 5px; overflow: auto; width: 100%"><code>my.cnf
#
# Configuration file for ...
# Generated by mcc
#
[mysqld]
log-error=mysqld.53.err
datadir="/home/user/MySQL_Cluster/53/data"
tmpdir="/home/user/MySQL_Cluster/53/tmp"
basedir="/usr/sbin/"
port=3306
ndbcluster=on
ndb-nodeid=53
ndb-connectstring=172.31.24.27:1186,
socket="/home/user/MySQL_Cluster/53/mysql.socket"
</code></pre> and same for host 5 only replacing 53 with 54.<br/>
<strong>GOTCHA:</strong> Client programs (<strong>ndb_mgm</strong> and <strong>mysqladmin</strong>) do not reside in /usr/sbin/ on RHEL but rather in /usr/bin/ thus <strong>MCC</strong> now tries sbin, bin and checks if necessary client libs reside in PATH. What this means for you is that now you will be able to check status of Cluster and shut it down from within <strong>MCC</strong>. To check status you can also SSH to MGMT node host (#1) and use <strong>ndb_mgm</strong> as described above.<br/><br/>
<h3>
Stopping the Cluster</h3>
Either press "Stop Cluster" button in <strong>MCC</strong> or issue commands by hand in reverse order than the one you used for starting Cluster:<br/>
<ul>
<li> SSH to Host#4, issue <code>(sudo) mysqladmin -uroot shutdown</code></li>
<li> SSH to Host#5, issue <code>(sudo) mysqladmin -uroot shutdown</code></li>
<li> SSH to Host#1, issue <code>(sudo) ndb_mgm --ndb-connectstring="172.31.24.27:1186", --execute=shutdown</code></li>
</ul><br/>
Result should be full cluster stop:<br/>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirlFMP7bLiFMlie-boRMRKpINIkOV3vpK3bseOnCXjxhf5ss-Jaj8zMAZOaw7naYa1XJt-QZStyf35zJDxGgkSrEem5q1eraETmas-BLcqk16GKPGA9-JX92XDz6CAJu87NOA8wDVI78HY/s1600/RH-5-node-cluster-shutdown.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirlFMP7bLiFMlie-boRMRKpINIkOV3vpK3bseOnCXjxhf5ss-Jaj8zMAZOaw7naYa1XJt-QZStyf35zJDxGgkSrEem5q1eraETmas-BLcqk16GKPGA9-JX92XDz6CAJu87NOA8wDVI78HY/s1600/RH-5-node-cluster-shutdown.png" data-original-width="749" data-original-height="439" /></a><br/><br/>
<h3>
Summary</h3>
<li>Stop any existing mysqld's and/or change the port for SQL nodes.</li>
<li>Open correct ports in firewall, set the authentication for your Cloud deployment.</li>
<li><strong>ServerPort</strong> needs setting only in multi-host Cloud deployment. Beware that each DATA node residing on same host needs different value!</li>
<li>If necessary, update YUM, download tools such as wget.</li>
<li>Get MySQL repository, disable Server, enable Cluster.</li>
<li>Install MySQL software on hosts according to your topology.</li>
<li>Install MySQL Cluster software locally so you get /share/MCC.</li>
<li>Recreate topology in <strong>MCC</strong> using "Add Host" button, putting internal IP address and private key in proper place.</li>
<li>If using instances with very limited RAM, set <strong>SharedGlobalMemory</strong> to 128MB, set <strong>LockPagesInMainMemory</strong> to 1.</li>
<li>Deploy the configuration from <strong>MCC</strong> so you get all the directories created and configuration files distributed.</li>
<br/><br/>
<a href="http://toncigrgin.blogspot.com/2018/05/mysql-cluster-76-in-cloud-rpmyumdocker.html">Next</a>, I will talk of problems encountered, troubleshooting and Docker deployment.Tončihttp://www.blogger.com/profile/10650361408804728027noreply@blogger.com0tag:blogger.com,1999:blog-7850846940198540592.post-28790141073502761122018-02-05T16:22:00.000+01:002018-02-05T16:26:00.391+01:00MySQL Cluster 7.6 includes new Cluster ConfiguratorAfter dealing with Windows performance, I switched to MySQL Cluster Configurator (MCC for short) project. This was quite a change for me having to deal with Python back-end and JavaScript front-end, languages I was not so familiar with.<br />
<br />
For the history of the project, please refer to Andrew's <a href="http://www.clusterdb.com/mysql-cluster/mysql-cluster-7-3-auto-installer">blog post</a> for I will concentrate on changes made in new version.<br />
<br />
There are many exciting new features in MySQL Cluster 7.6.4DMR including new MCC. To download MySQL Cluster 7.6.4DMR, please go to <a href="https://dev.mysql.com/downloads/cluster/#downloads">Development Releases</a> tab. To see what's new in 7.6.4DMR, please follow this <a href="https://dev.mysql.com/doc/relnotes/mysql-cluster/7.6/en/mysql-cluster-news-7-6-4.html">link</a>.
<br />
<br />
<h3>
MySQL Cluster Configurator in short</h3>
With a single command launch the web-based wizard which then steps you through configuring, deploying configuration and starting the cluster; to keep things even simpler, it will automatically detect the resources on your target machines and use these results together with the type of workload you specify in order to determine values for the key configuration parameters.<br />
The software is part of the MySQL Cluster package (NDB). To run on Windows, just double click setup.bat – note that if you installed from the MSI and didn’t change the install directory then this will be located somewhere like C:\Program Files...\MySQL\MySQL Cluster 7.6. On Linux, just run ndb_setup from the bin folder.<br />
On Windows, we provide all necessary libraries and the Python while on Linux, you should have Python 2.7 (latest) installed as well as Paramiko 2 or newer. Paramiko will install Cryptography and other required libraries.
<br />
If you launch the installer from a desktop environment shell then the first page of the wizard will automatically open in your web browser, if not then just browse to the URL that is displayed on the command line.
<br />
<b>Recap:</b> Start the Python web server from shell using setup.bat (Windows) or bin/ndb_setup (*nix), JavaScript front-end will be opened for you in default browser. If it's not, copy the link from console to your favorite browser. Make sure you have administrator rights on all the hosts you will use in cluster. Also, make sure to have MySQL Cluster installed on all the boxes making up cluster.<br />
There is an extensive help file in /hlp/html directory, please read if you're new to product.
<br />
<br />
<h3>
Who is this software for</h3>
Typically, MCC would be used by developers working on Cluster software and people looking for quick evaluation of MySQL Cluster on various hardware configurations including cloud instances.
<br />
<br />
<h3>
New version highlights</h3>
<ul>
<li>Multiple reusable configurations. Take your configuration with you to work on different boxes.</li>
<li>Much safer configuration files allowing for saving passwords/passphrases resulting in full restoration of NDB Cluster from configuration file.</li>
<li>Many new options to configure both for NDB Cluster processes as well as for mysqld. Multiple choice options have proper drop-down lists associated.</li>
<li>Many new members in Cluster and Host objects. GUI changes to present them.</li>
<li>Nodes can have external IP address (to gain access from MCC for example) as well as internal IP address (for faster and safer communication inside cloud deployment or other VPN environment).</li>
<li>Extended ways of authentication such as using different username, private key on per host basis, using keys with passphrases and so on.</li>
<li>Better detection of running cluster processes and option to stop them even if startup fails.</li>
</ul>
<br />
<h3>
Language choice</h3>
JavaScript was obvious choice since it's common to have GUI run in browser. Due to it's limitations (none likes browser sniffing around file system), we have back-end web server in Python handling various requests related to files, remote hosts etc. Front and back end communicate via POST messages so it is really important that you protect this communication. For a start, we provide self signed certificates for securing it. If you are testing things in sand-box, you may switch https for http to speed up things.<br />
<b>Recap:</b> Python back end manipulates encryption/decryption, files, connections to local/remote hosts and so on while JavaScript front-end provides GUI in your browser for convenient presentation of configuration options.
<br />
<br />
<h3>
Changes</h3>
<h4>
Changes in configuration files and authentication methods:</h4>
First change tackled is the <b>configuration</b>. Versions shipped with previous Cluster releases had two major drawbacks; they kept configuration in cookies and could handle just one Cluster configuration which was locked to box where you created it. Keeping Cluster configuration in cookies comes with two major drawbacks; cookies are size-limited and very insecure.<br />
To remedy this situation, we opted for external configuration file saved in current user's HOME directory (this is done through Python web server that's running MCC); one configuration per file. The file itself is AES encrypted using passphrase provided in front-end and passed down to Python web server via POST message. Passphrase is then kept in memory for the duration of session. With provided passphrase and file name, Python server sends decrypted configuration back to front-end. Configuration is kept in global.windowname variable for the duration of the session.<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhcNPpbSQgCJjEd19eeE8wqp4mS6CMAkpG_adll_cQchPGnuNSisIuOFk6RigLolmdkhuIv2SFpsTD8ujPX8mHiz66iajgj4qQVIr6BZELZtcDBh-3F9hEP47AsWuSKrKY5UGrZSLaI_LUX/s1600/MCC-ViewCfg02.PNG" imageanchor="1"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhcNPpbSQgCJjEd19eeE8wqp4mS6CMAkpG_adll_cQchPGnuNSisIuOFk6RigLolmdkhuIv2SFpsTD8ujPX8mHiz66iajgj4qQVIr6BZELZtcDBh-3F9hEP47AsWuSKrKY5UGrZSLaI_LUX/s320/MCC-ViewCfg02.PNG" width="320" height="200" data-original-width="493" data-original-height="308" /></a>
<br />
This has solved several issues:<br />
<ul>
<li>Configuration now stores all your passwords, passphrases for keys and such allowing for quick and full recreation of Cluster.</li>
<li>Extending configuration size allows for per-host credentials. I.e. each host in Cluster can have it's own way of authentication and set of credentials.</li>
</ul>
Essentially, when looking for quick and dirty deployment over hosts that use same credentials, it is enough to provide them on Page 2, Cluster Configuration. When working with hosts that have their own separate credentials, you can define them on Host level (Page 3, Add Hosts). We prefer adding hosts on Page 3 via Add Host button to defining them on Page 2 "Host list" box.<br />
Credentials can be:<br />
<ul>
<li>Username/password.</li>
<li>Username/Key name.</li>
<li>Username/Key name/Passphrase.</li>
<li>Key name/Passphrase.</li>
<li>Key name.</li>
<li>Nothing.</li>
</ul>
If PATH is not provided along with Key name, ~/.ssh is assumed. If using keys and no key name is provided, ~/.ssh/id_rsa is assumed. So, for authentication using standard private key without passphrase stored in default place, you just need to check "Key based SSH" (Page 2, Cluster level) or "Key-based auth" for particular host (Page 3, Add/Edit Host) checkbox and nothing more. If there are absolutely no credentials at host level, program behaves as in old versions meaning whatever was provided on Cluster level is used.<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2eTvuy58rBZdfxP2u9YqxDYW7W_D2C-yDEedKCwxHHLb-qpy9XaKDvYmp8tysGfm1TIbjy4wexhnDi7VVY_gq39zu4j9nvmpWdgNk5pGMcrzqaSkpov_q3m2zepXHJzzUVouAnZhpjgrT/s1600/Cluster-Host-key-based-cb.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2eTvuy58rBZdfxP2u9YqxDYW7W_D2C-yDEedKCwxHHLb-qpy9XaKDvYmp8tysGfm1TIbjy4wexhnDi7VVY_gq39zu4j9nvmpWdgNk5pGMcrzqaSkpov_q3m2zepXHJzzUVouAnZhpjgrT/s1600/Cluster-Host-key-based-cb.png" data-original-width="686" data-original-height="173" /></a><br />
<b>WARNING:</b> Although asynchronous, call to save configuration changes does take some time to complete. The saving of configuration takes place after pressing "Save&Next" button:<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgoZHVa09HKRNmjOqLnmQNfQlrhxYc1LYPAM90AlxmME3ymAw71ya2H4Go6G9eUyvrWkqVo7tEmcrqHzsJfvLVsoExJ_L3nYwLCSpS43YieeWuTvMtMY9iGk9T_C3NO_QFNXO_6dOEGwIf2/s1600/page_buttons.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgoZHVa09HKRNmjOqLnmQNfQlrhxYc1LYPAM90AlxmME3ymAw71ya2H4Go6G9eUyvrWkqVo7tEmcrqHzsJfvLVsoExJ_L3nYwLCSpS43YieeWuTvMtMY9iGk9T_C3NO_QFNXO_6dOEGwIf2/s1600/page_buttons.png" data-original-width="271" data-original-height="62" /></a><br />
No changes are saved if you close the tab or browser abruptly! After making extensive changes and pressing Save&Next I like to allow for some time for configuration to be saved. If you open debug console, you will see the notification.<br />
If you Cluster is all set up and ready to go and you just want to take one more look at various configuration options, you can use breadcrumb buttons as they do not trigger save method:<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjqs9D-KRKBNmTvDq5rOJwDQ_jAcTA4gLb_PsDG5CZM-NKTk5_M93Up6txVYLwrV_r_TyVqZLis7_NV-L9cRP_OzcnlkF_TKgFAahjnvdVHbXxoez7CdzsCnKd7NoBOXUDVmhMyjr_cWHPl/s1600/breadcrumb_trail.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjqs9D-KRKBNmTvDq5rOJwDQ_jAcTA4gLb_PsDG5CZM-NKTk5_M93Up6txVYLwrV_r_TyVqZLis7_NV-L9cRP_OzcnlkF_TKgFAahjnvdVHbXxoez7CdzsCnKd7NoBOXUDVmhMyjr_cWHPl/s1600/breadcrumb_trail.png" data-original-width="738" data-original-height="65" /></a><br />
Since all credentials are saved, you can skip looking into loaded configuration altogether and go directly to "Deploy configuration" page.<br />
<b>Gotcha:</b> In order to make configurations portable, we had to limit the usage of "localhost". If you include localhost in your Cluster, you will not be able to add any more remote hosts.<br />
<b>Gotcha:</b> List with available configurations is provided to front-end upon welcome page load so if you add more configurations to your HOME using external tools they will not be shown (unless you reload page or restart entire program). However, if you choose "New Configuration" and provide the name of <b>existing</b> one, the existing configuration will be loaded.<br />
<b>Gotcha:</b> "New Configuration" requires you to provide a passphrase and a confirmation. Reading existing configuration just requires passphrase. Please keep your passphrase(s) safe as there is no way to reverse engineer contents of configuration file without it! Each file/configuration can have it's own passphrase.<br />
<b>Recap:</b> There can be any number of portable, encrypted configurations now which you can find in your HOME on box running MCC. Each host can have it's own way of authentication and a set of credentials. We did our best to guess which auth method and credentials to use based on input provided.
<br />
<br />
<h4>
Changes in Cluster object (Page 2):</h4>
Cluster object has two new attributes:
<ul>
<li><b>Install MySQL Cluster:</b> Option for installing Cluster on hosts<br />
NONE: No installation of Cluster will take place (default).<br />
BOTH: Both DOCKER and installation from REPO will be considered depending on OS and available images. Should both installation mechanisms be available on certain host, we will prefer REPO over DOCKER.<br />
REPO: For every host in Cluster, the attempt will be made to install Cluster SW from the repository URL.<br />
DOCKER: For every host in Cluster, the attempt will be made to install Cluster SW DOCKER image.</li>
<li><b>Open FW ports:</b> Check if you need the tool to try to open necessary ports for Cluster processes on every hosts.</li>
</ul>
<b>Gotcha:</b> In this version "Install MySQL Cluster" is not functional so you need Cluster installed on all hosts beforehand.<br />
<br />
<h4>
Changes in Host object and GUI (Page 3):</h4>
Host object has undergone major rework. Host name is now used for <b>external</b> host IP address i.e. IP address at which MCC web server instance can reach that particular host. Host Internal IP refers to that particular host IP address inside VPN. If there is no VPN in play, both IP addresses are the same. We strongly encourage using IP addresses here to skip potential problems with resolving host names.<br />
Authentication, as per above, could be a) using keys or b) ordinary username/password. If you check Key-based auth when Adding/Editing host and provide nothing, ~/.ssh/id_rsa key will be used without passphrase. You can also define alternate user (to one starting MCC) which comes handy when logging in to domains. Each key can have passphrase and you can provide the path to and name of specific key to use for that host.<br />
If you check Configure installation, you will be presented with additional fields relating to repository and Docker image. We do our best to provide you with default values based on OS running on particular host you're Adding/Editing.<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj-mln1Hsc9D92LvAOOAHEpcNTCFveFVqzKYl0FO50Zy2s7PFL6OHyEX2mxTAx9Ni3k6tt7j9mulj8o29LijJrp8nNtmdqAhf8Zy58MidJorah35LpXJ-82tpkkqgDnY2Imm98bRfBh4zzM/s1600/edit_selected_hosts.png" imageanchor="1" ><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj-mln1Hsc9D92LvAOOAHEpcNTCFveFVqzKYl0FO50Zy2s7PFL6OHyEX2mxTAx9Ni3k6tt7j9mulj8o29LijJrp8nNtmdqAhf8Zy58MidJorah35LpXJ-82tpkkqgDnY2Imm98bRfBh4zzM/s1600/edit_selected_hosts.png" data-original-width="723" data-original-height="526" /></a>
<br />
We have also added Show/Hide extended host info toggle button switching between single and double line host info representation.
<br />
<br />
<h4>
Changes to Define parameters (Page 5):</h4>
We are constantly adding more configurable parameters to keep up with MySQL server and Cluster evolution but this is a moving target so we ask for patience if certain parameter you'd like to configure is missing.<br />
In addition to many new configurable parameters, we have extended a GUI so that, for appropriate options, you get drop-down list of allowed values.<br />
<br />
<h4>
Changes to Deploy configuration (Page 6, last):</h4>
The Start and Stop Cluster buttons now behave more intelligently in terms of determining if the Cluster or any of its processes is running and enabling/disabling appropriate buttons.<br />
Previously, if Cluster was stuck in any of the startup phases, you had to terminate MCC and kill all the processes on all of the hosts manually. Now, if you think something is wrong, you can close the progress window and it will give control back to MCC enabling Stop Cluster button which you can then press to stop the stray processes properly. It will also provide you with list of log files which you can then check for problems.<br />
Stopping mysqld process(es) might not work if you changed the credentials from command line.<br />
<br />
<br />
These changes are available in version 7.6.4DMR and up. We encourage you to try MCC and MySQL Cluster in your environment!Tončihttp://www.blogger.com/profile/10650361408804728027noreply@blogger.com0tag:blogger.com,1999:blog-7850846940198540592.post-27682847323788836632016-06-29T11:46:00.001+02:002016-06-29T11:46:44.441+02:00Working with more than 64 CPUs in PowershellWrote this several months ago but was too busy to publish :-/<br />
<br />
As noted in one of the previous <a href="http://toncigrgin.blogspot.com/2015/11/windows-perf-counters-blog4.html">blog post</a>, I will use following terminology:<br />
<ul>
<li> "Processor" is a piece of hardware you connect to a socket on the motherboard.</li>
<li> "Physical Core" is a physical computing unit built into the "Processor".</li>
<li> "Virtual Core" is a virtual computing unit built on top of "Physical Core" (i.e. HT is ON).</li>
<li> "CPU" is a computing unit inside the "Processor", either physical or virtual.</li>
</ul>
<br />
After a series of blogs on Windows performance counters and after <a href="https://dev.mysql.com/downloads/benchmarks.html">releasing sysb.ps1</a> testing/benchmarking framework version 0.9RC (dbt2-0.37.50.10) I set out to eliminate some unknowns from the testing. First to tackle was Kernel scheduler in an effort to run processes, from inside the Powershell script, on controlled subset of CPUs much like TASKSET does on Linux. Also worth noting is that proximity rocks, on occasion, meaning you can get <b>up to</b> 20% better results when the workload is distributed perfectly. However, this is hard to achieve thus I'm more going after consistency in test environment.<br />
This posed quite a bit of challenges; knowing the details of hardware, NUMA node assignments, finding out and evaluating various ways of controlling the CPU pinning to calculating CPU affinity mask for more than 64 CPUs.<br />
One interesting challenge was to calculate the CPU indexes for MySQL Cluster thread config.<br />
As a first step, I had to find out as much as possible about my hardware.
<br />
<br />
<h3>
Know your hardware:</h3>
<pre><small>PS> Get-CimInstance Win32_BIOS
SMBIOSBIOSVersion : 11018100
Manufacturer : American Megatrends Inc.
Name : Default System BIOS
SerialNumber : 1207FMA00C
Version : SUN - 20151001
PS> Get-CimInstance Win32_ComputerSystem | FL *
Status : OK
Name : HEL01
Roles : {LM_Workstation, LM_Server, NT, Server_NT}
AutomaticManagedPagefile : False
DomainRole : 3
HypervisorPresent : False
Manufacturer : Oracle Corporation
Model : Sun Fire X4800
NetworkServerModeEnabled : True
NumberOfLogicalProcessors : 96
NumberOfProcessors : 8
PartOfDomain : True
SystemType : x64-based PC
TotalPhysicalMemory : 549746266112
PS> Get-CimInstance Win32_ComputerSystemProcessor | FL *
GroupComponent : Win32_ComputerSystem (Name = "HEL01")
PartComponent : Win32_Processor (DeviceID = "CPU0")
CimClass : root/cimv2:Win32_ComputerSystemProcessor
CimInstanceProperties : {GroupComponent, PartComponent}
...
PartComponent : Win32_Processor (DeviceID = "CPU1")
PartComponent : Win32_Processor (DeviceID = "CPU2")
PartComponent : Win32_Processor (DeviceID = "CPU3")
PartComponent : Win32_Processor (DeviceID = "CPU4")
PartComponent : Win32_Processor (DeviceID = "CPU5")
PartComponent : Win32_Processor (DeviceID = "CPU6")
PartComponent : Win32_Processor (DeviceID = "CPU7")
PS> Get-CimInstance Win32_PerfFormattedData_PerfOS_NUMANodeMemory
Name : 0
AvailableMBytes : 64530
FreeAndZeroPageListMBytes : 63989
StandbyListMBytes : 541
TotalMBytes : 65526
...
Name : 7
AvailableMBytes : 64600
FreeAndZeroPageListMBytes : 64387
StandbyListMBytes : 213
TotalMBytes : 65536
PS> Get-CimInstance Win32_SystemSlot
SlotDesignation : EM00 PCIExp
Tag : System Slot 0
SupportsHotPlug : True
Status : OK
Shared : True
PMESignal : True
MaxDataWidth : 8
...
SlotDesignation : EM01 PCIExp
Tag : System Slot 1
SlotDesignation : EM30 PCIExp
Tag : System Slot 2
SlotDesignation : EM31 PCIExp
Tag : System Slot 3
SlotDesignation : EM10 PCIExp
Tag : System Slot 4
SlotDesignation : EM11 PCIExp
Tag : System Slot 5
SlotDesignation : EM20 PCIExp
Tag : System Slot 6
SlotDesignation : EM21 PCIExp
Tag : System Slot 7
PS> Get-CimInstance Win32_PerfFormattedData_Counters_ProcessorInformation
Name : 0,0
PercentofMaximumFrequency : 100
PercentPerformanceLimit : 100
PercentProcessorPerformance : 69
ProcessorFrequency : 2001
...
Name : 0,11
---
Name : 7,0
PercentofMaximumFrequency : 100
PercentPerformanceLimit : 100
PercentProcessorPerformance : 72
ProcessorFrequency : 2001
...
Name : 7,11
</small></pre>
Or, in short, my test box has 2 Processor groups with 48 CPUs each. This makes for Max. CPU affinity mask of 281474976710655d (or 111111111111111111111111111111111111111111111111b). The total number of CPUs is 96, total number of sockets and NUMA nodes is 8.<br />
<br />
<b>Note:</b> Notice there are exactly 48 "1" in Max CPU Affinity mask which is the number of CPUs in each Processor group. This implies you can only set process affinity mask on per Processor group basis, not machine-wide! This limitation is caused by CPUs affinity mask being 64 bits long.<br />
Groups, NUMA nodes etc. assignments are not chiseled in stone. Please see <a href="https://msdn.microsoft.com/en-us/library/windows/hardware/ff542298%28v=vs.85%29.aspx">MSDN</a> for details on how to manipulate these settings.<br />
<br />
Once done playing with WMI, you can turn to <a href="https://technet.microsoft.com/en-us/sysinternals/cc835722.aspx">coreinfo</a> from Sysinternals suite as it's extremely informative:
<pre><small>Intel(R) Xeon(R) CPU E7540 @ 2.00GHz
Intel64 Family 6 Model 46 Stepping 6, GenuineIntel
Microcode signature: 00000009
HTT * Hyperthreading enabled
HYPERVISOR - Hypervisor is present
VMX * Supports Intel hardware-assisted virtualization
SVM - Supports AMD hardware-assisted virtualization
X64 * Supports 64-bit mode
SMX - Supports Intel trusted execution
SKINIT - Supports AMD SKINIT
...
</small></pre>
Important to notice is that, in my configuration, Sockets map to NUMA nodes 1-1:
<pre><small>Logical Processor to Socket Map: Logical Processor to NUMA Node Map:
Socket 0: NUMA Node 0:
************------------------------------------ ************------------------------------------
------------------------------------------------ ------------------------------------------------
Socket 1: NUMA Node 1:
------------------------------------------------ ------------------------------------------------
************------------------------------------ ************------------------------------------
Socket 2: NUMA Node 2:
------------************------------------------ ------------************------------------------
------------------------------------------------ ------------------------------------------------
Socket 3: NUMA Node 3:
------------------------------------------------ ------------------------------------------------
------------************------------------------ ------------************------------------------
Socket 4: NUMA Node 4:
------------------------************------------ ------------------------************------------
------------------------------------------------ ------------------------------------------------
Socket 5: NUMA Node 5:
------------------------------------------------ ------------------------------------------------
------------------------************------------ ------------------------************------------
Socket 6: NUMA Node 6:
------------------------------------************ ------------------------------------************
------------------------------------------------ ------------------------------------------------
Socket 7: NUMA Node 7:
------------------------------------------------ ------------------------------------------------
------------------------------------************ ------------------------------------************
</small></pre>
so I can use Processor/Socket/NUMA node as though they are synonyms. Also, notice that NUMA node/Socket 0 and even ones are in <b>Processor group 0</b> while odd sockets are in <b>Processor group 1</b>. Here is how CPU utilization looks like in Task manager/Performance tab when just ProcessorGroup 0 is used:<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhfoviTi_axP8Qw-xpI5XKiZyNUgUSis-kNy0AI2J_B4fkb10resrZ6gFR_mE0VYf3TuiIyJVfbDlH831IAICe4Fu-e-7r6vffb9p4Q5cqnckoilbCZrPZRFKlGcEP5RZ4TnoxdveDAloQa/s1600/Just-1-Proc-Group.png" imageanchor="1"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhfoviTi_axP8Qw-xpI5XKiZyNUgUSis-kNy0AI2J_B4fkb10resrZ6gFR_mE0VYf3TuiIyJVfbDlH831IAICe4Fu-e-7r6vffb9p4Q5cqnckoilbCZrPZRFKlGcEP5RZ4TnoxdveDAloQa/s400/Just-1-Proc-Group.png" /></a>
<br />
<pre><small>Logical Processor to Group Map:
Group 0: Group 1:
************************************************ ------------------------------------------------
------------------------------------------------ ************************************************</small></pre>
<b>Note:</b> Coreinfo provides NUMA nodes latency too:
<pre><small>Approximate Cross-NUMA Node Access Cost (relative to fastest):
00 01 02 03 04 05 06 07
00: 1.4 1.7 2.1 1.7 1.7 2.1 2.2 2.1
01: 1.7 1.4 1.7 2.1 2.1 1.7 2.0 1.3
02: 2.1 1.7 1.4 1.7 2.1 2.1 1.6 1.2
03: 1.8 2.1 1.7 1.4 2.1 2.1 2.0 1.1
04: 1.7 2.1 2.1 2.1 1.4 1.7 1.7 1.4
05: 2.1 1.7 2.1 2.1 1.7 1.4 2.0 1.0
06: 2.1 2.1 1.7 2.1 1.7 2.1 1.4 1.3
07: 2.1 2.1 2.1 1.7 2.1 1.7 1.6 1.0
</small></pre>
<br />
<h3>
The software:</h3>
Primary tool used is sysb.ps1 Powershell script version 1.0 (not available for download atm). Version 0.9x RC is <a href="http://downloads.mysql.com/source/dbt2-0.37.50.10.tar.gz">available for download</a> and placed in dbt2-0.37.50.10.tar.gz\dbt2-0.37.50.10.tar\dbt2-0.37.50.10\windows_scripts\sysb-script\ directory.<br />
<br />
OS details:
<pre><small>PS:518 [HEL01]> Get-CimInstance Win32_OperatingSystem | FL *
Status : OK
Name : Microsoft Windows Server 2012 R2 Standard
FreePhysicalMemory : 528660256
FreeSpaceInPagingFiles : 8388608
FreeVirtualMemory : 537242324
Distributed : False
MaxNumberOfProcesses : 4294967295
MaxProcessMemorySize : 137438953344
OSType : 18
SizeStoredInPagingFiles : 8388608
TotalSwapSpaceSize :
TotalVirtualMemorySize : 545250196
TotalVisibleMemorySize : 536861588
Version : 6.3.9600
BootDevice : \Device\HarddiskVolume1
BuildNumber : 9600
BuildType : Multiprocessor Free
CodeSet : 1252
DataExecutionPrevention_32BitApplications : True
DataExecutionPrevention_Available : True
DataExecutionPrevention_Drivers : True
DataExecutionPrevention_SupportPolicy : 3
Debug : False
ForegroundApplicationBoost : 2
LargeSystemCache :
Manufacturer : Microsoft Corporation
OperatingSystemSKU : 7
OSArchitecture : 64-bit
PAEEnabled :
ServicePackMajorVersion : 0
ServicePackMinorVersion : 0
</small></pre>
<br />
<h3>
So how do the Windows work?</h3>
<b>Process</b> is just a container for threads doing the work providing you with fancy name, PID etc. This effectively means you can not calculate "System load" like on Linux. This also explains why there is no ProcessorGroup member attached to Process class while there is one for <a href="https://msdn.microsoft.com/en-us/library/jj665638%28v=vs.110%29.aspx">Threads</a>. This also makes all sorts of problems regarding CPU utilization as described in previous blogs <a href="http://toncigrgin.blogspot.hr/2016/01/windows-perf-counters-blog8.html">here</a> and <a href="http://toncigrgin.blogspot.hr/2016/02/linux-top-command-on-windows-further.html">here</a>.<br />
<b>Processor group</b> is a collection of up to 64 CPUs as explained <a href="https://msdn.microsoft.com/en-us/library/dd405503%28VS.120%29.aspx">here</a> and <a href="https://blogs.msdn.microsoft.com/saponsqlserver/2010/09/27/windows-2008-r2-groups-processors-sockets-cores-threads-numa-nodes-what-is-all-this/">here</a>.<br />
<b>Thread</b> is a basic unit of execution. Setting the Thread affinity will influence the Process class and dictate what you can do with it. There is a great paper on this you can <a href="https://msdn.microsoft.com/en-us/library/windows/hardware/dn653313%28v=vs.85%29.aspx">download</a> from MSDN to figure it out. The focus of this blog is on scripting.<br />
<br />
<br />
<h3>
Know the OS pitfalls:</h3>
<b>The setup:</b> I have a script acting as testing/benchmarking framework. Script controls the way processes are launched, collects data from running processes and generally helps me do part of my job of identifying performance issues and testing solutions.<br />
<b>The problem:</b> Windows is thread based OS and I can not control the threads in binary from within the script.<br />
Next, .NET <a href="https://msdn.microsoft.com/en-us/library/system.diagnostics.process_properties.aspx">System.Diagnostics.Process</a> class does not expose Processor group bit. This means there is no way to control Processor group and thus no way to guarantee the kernel scheduler will start all of your processes inside the Processor group you want :-/ I consider this a bug and not deficiency in Windows because of the following scenario:<br />
<pre> "ProcessA" is pinned, by scheduler, to Processor group 0 with ability to run on all CPUs within that group.
"ProcessB" is pinned, by scheduler, to Processor group 1 with ability to run on all CPUs within that group.
ProcessorAffinity member of System.Diagnostics.Process class is the same in both cases!
$procA = Get-Process -Name ProcessA
$procA.ProcessorAffinity
281474976710655 #For my 48 CPUs in each Processor group.
$procB = Get-Process -Name ProcessB
$procB.ProcessorAffinity
281474976710655 #For my 48 CPUs in each Processor group.
</pre>
This leads you to believe that both processes run in the same Processor group, which might not be true as the information is ambiguous. I have set up mysqld to run on 1st NUMA node and part of second (12 + 8 CPUs). At the same time, Sysbench is pinned to NUMA node 0, last 4 CPUs. When scheduler decides to run mysqld on Processor group 1, the CPU load distribution is like this:<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjs0nOqDNisIbiaFZ4-AZ1sJPO91wLRveo1wIalrgmLVyyvJ3c9ldlb7Ysg6D4VzmYFbFCv8TFk6H6Hh7FxDCSx1MQ7V6EJPeC_rQeudqNr6XgppZudC7BBNSNKmIHuQ_0CzOWQ7YEcHy_q/s1600/How-to-set-up-affinity-masks.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img alt="" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjs0nOqDNisIbiaFZ4-AZ1sJPO91wLRveo1wIalrgmLVyyvJ3c9ldlb7Ysg6D4VzmYFbFCv8TFk6H6Hh7FxDCSx1MQ7V6EJPeC_rQeudqNr6XgppZudC7BBNSNKmIHuQ_0CzOWQ7YEcHy_q/s400/How-to-set-up-affinity-masks.png" title="NUMA #0, last 4 CPUs lit up by Sysbench. NUMA #1 and part of 3, lit up by mysqld." /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">NUMA #0, last 4 CPUs lit up by Sysbench. NUMA #1 and part of 3, lit up by mysqld.</td></tr>
</tbody></table>
<br />
Using the same(!) Process.ProcessorAffinity for mysqld for subsequent run but this time the scheduler decides it will run mysqld on Processor group 0:<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi_ksc45rpwunLFYoCW2PsVGw8PuaoXJNRhsfxUvSphvdc4dBxLnXTmkYofol8o_qyWieV4MZ3f1IRqULpE2CTSaiICn1Z_kfRb7BxrHDQOsHPsQXkknclb6cA228FCcpv5nD3vvTYEJH51/s1600/Compl-wrong-choice-mysqld-win-sb-all-on-nnode-0-2100.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img alt="" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi_ksc45rpwunLFYoCW2PsVGw8PuaoXJNRhsfxUvSphvdc4dBxLnXTmkYofol8o_qyWieV4MZ3f1IRqULpE2CTSaiICn1Z_kfRb7BxrHDQOsHPsQXkknclb6cA228FCcpv5nD3vvTYEJH51/s400/Compl-wrong-choice-mysqld-win-sb-all-on-nnode-0-2100.png" title="NUMA #0, last 4 CPUs lit up by Sysbench and mysqld. NUMA #2 in part lit up by mysqld." /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><small>NUMA #0, last 4 CPUs lit up by Sysbench and mysqld.<br />
NUMA #2 in part lit up by mysqld.</small></td></tr>
</tbody></table>
<br />
It is obvious how later case will most likely produce much lower results since mysqld is competing with Sysbench (on last 4 CPUs of the NUMA node 0) and Windows (first 2 CPUs of NUMA node 0). This is indicative of 2 things:<br />
a) Microsoft rushed solution for big boxes (> 64 CPUs) and it is not mature nor will it scale.<br />
b) You can not trust Kernel scheduler to do the right thing on its own as it has no clue as to what will be your next move.<br />
I might add here that even the display in Task manager lacks the ability to display CPU load per ProcessorGroup...<br />
<br />
Before you send me to RTFM and do this the "proper" way, please notice that the CPU usage pattern for NUMA nodes 5 and 7 is the same in both runs. This is because our Cluster knows how to pin <b>threads</b> to CPUs "properly". Alas, I do not think this is possible from the Powershell.<br />
Also notice the lack of ProcessorGroup member in <a href="https://msdn.microsoft.com/en-us/library/system.diagnostics.process_properties.aspx">System.Diagnostic.Process</a> class. I expected at least ProcessorGroup with getter function (if not complete getter/setter) so I can break the run if scheduler makes the choice I'm not happy with.<br />
The last problem to mention is late binding of Affinity mask :-/. The code might look like this:<br />
<pre><code><small>
$sb_psi = New-object System.Diagnostics.ProcessStartInfo
$sb_psi.CreateNoWindow = $true
$sb_psi.UseShellExecute = $false
$sb_psi.RedirectStandardOutput = $true
$sb_psi.RedirectStandardError = $true
$sb_psi.FileName = "$PathToSB" + '\sysbench.exe '
$sb_psi.Arguments = @("$sbArgList")
$sb_process = $null
$sb_process = New-Object System.Diagnostics.Process
$sb_process.StartInfo = $sb_psi
[void]$sb_process.Start() <<<<
#Now you can set the Affinity mask:
$sb_process.ProcessorAffinity = $SCRIPT:SP_BENCHMARK_CPU
$sb_process.WaitForExit()
</small></code></pre>
IMO, process.ProcessorAffinity should go to System.Diagnostics.ProcessStartInfo.<br />
I can't help but to wonder what will happen if Intel decides to release single processor with 64+ CPUs?<br />
<br />
<br />
<h3>
What are our options in Powershell then?</h3>
Essentially, you can use 3 techniques to start the process in Powershell and bind it to CPUs but you have to bear in mind that this is not what Microsoft expects you to do so each approach has its pro's and con's:<br />
1) Using <a href="http://ss64.com/nt/start.html">START</a> in cmd.exe (start /HIGH /NODE 2 /AFFINITY 0x4096 /B /WAIT E:\test\...\sysbench.exe --test=oltp...)<br />
<pre>Settings:
sysbench.conf:
BENCHMARK_NODE=5
BENCHMARK_CPU="111100000000" # Xeon E7540 has 12 CPUs per socket so I'm running on LAST 4 (9,10,11 and 12).
These options allow user to run Sysbench on certain NUMA node as well as certain CPUs within that NUMA node.
autobench.conf:
SERVER_NUMA_NODE=3
SERVER_CPU="111111111" #(Or, 000111111111) Running on first 9 CPUs.
It is not necessary to set CPUs to run on if you're running on entire dedicated NUMA node.
Pros: Works.
Cons: The process you're starting is not the expected one (say, benchmark) but rather cmd.exe START.
Cumbersome.
Not really "Powershell-way".
Process is bound to just one NUMA node which is fine if it's not hungry for more CPU power.
</pre>
<br />
2) Using .NET System.Diagnostics.Process (<a href="https://technet.microsoft.com/en-us/library/hh849848.aspx">PS</a>, <a href="https://msdn.microsoft.com/en-us/library/e8zac0ca%28v=vs.110%29.aspx">C#</a>):
<br />
<pre> $process = Start-Process E:\test\mysql-cluster-7.5.0-winx64\bin\mysqld.exe -ArgumentList "--standalone --console
--initialize-insecure" -WindowStyle Hidden -PassThru -Wait -RedirectStandardOutput e:\test\stdout.txt
-RedirectStandardError e:\test\stderr.txt
$process.ProcessorAffinity = 70368739983360
Affinity mask means mysqld runs on NUMA node 7, 5 and part of 3 (0-based index)
<b>IF</b> ProcessorGroup is set to 1 by Kernel scheduler:
001111111111111111111111110000000000000000000000 = 70368739983360
|___________________48 CPUs____________________|
|__________||__________||__________||__________|
NUMA #7 NUMA #5 NUMA #3 NUMA #1
Settings:
Autobench.conf:
SP_SERVER_CPU=70368739983360
Sysbench.conf:
SP_BENCHMARK_CPU=211106232532992
#Run on NUMA node 7, last 2 CPUs, 110000000000000000000000000000000000000000000000b
Pros: Real "Powershell-way" of doing things.
Process can span over more than 1 NUMA node.
Good control of the process (HasExited, ExitTime, Kill, ID (PID) ...).
Cons: Late binding; i.e. process has to be up and running for you to pin it to CPUs. This presents a problem with processes
that start running immediately.
No way to control Processor group meaning there is no way to guarantee the kernel scheduler will start all of your
processes inside the desired Processor group.</pre>
<b>Note:</b> Using <b>-PassThru</b> ensures you will get Process object. Otherwise, Start-Process cmdlet has no output. Also, you can start the process and then use Get-Process -Name... to accomplish the same.<br />
<br />
Not available in Powershell AFAIK but important to understand if using MySQL Cluster:<br />
3) Hook the threads to CPUs. Since this is not available from the "outside", I will use the Cluster code to do the work for me:
<br />
<pre>config.ini
----------
NoOfFragmentLogParts=10
ThreadConfig=ldm={count=10,cpubind=88-91,100-105},tc={count=4,cpubind=94-95,106-107},send={count=2,cpubind=92-93},
recv={count=2,cpubind=98,99},main={count=1,cpubind=109},rep={count=1,cpubind=109}
sysbench.conf
-------------
#NUMA node to run sysbench on.
BENCHMARK_NODE=0
#Zero based index.
#CPUs inside selected NUMA node to run sysbench on.
BENCHMARK_CPU="111100000000"
000000001111
|__________|
|_12 CPUs__|
NUMA #0
CPU0 CPU11
autobench.conf
--------------
SP_SERVER_CPU=1048575
Affinity mask means mysqld runs on NUMA node 7, 5 and part of 3 (0-based index) IF ProcessorGroup is 1:
001111111111111111111111110000000000000000000000 = 70368739983360d
|___________________48 CPUs____________________|
|__________||__________||__________||__________|
NUMA #7 NUMA #5 NUMA #3 NUMA #1
Test image shows
000000000000000000000000000011111111111111111111 = 1048575d
|___________________48 CPUs____________________|
|__________||__________||__________||__________|
NUMA #7 NUMA #5 NUMA #3 NUMA #1
</pre>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibu7mmLPfvuGD5D3uIqoRfFXlRC5R0HnmZJFcjKFpnLVKkuS4VOGrJ7_y5-JbEtIrs1Sm51gyPCUNEnXvAv-4lctQohWPZjSwD1LX8CwrX3cvj9LeeBETyuYcqpkYKcZYOlt1pwpbUgKTE/s1600/How-to-set-up-affinity-masks.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibu7mmLPfvuGD5D3uIqoRfFXlRC5R0HnmZJFcjKFpnLVKkuS4VOGrJ7_y5-JbEtIrs1Sm51gyPCUNEnXvAv-4lctQohWPZjSwD1LX8CwrX3cvj9LeeBETyuYcqpkYKcZYOlt1pwpbUgKTE/s400/How-to-set-up-affinity-masks.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Sysbench is running on NUMA #0, last 4 CPUs.<br />
MySQLd is running on NUMA #1 and last 8 CPUs of NUMA #3.<br />
LDM threads are running on first 4 CPUS node #5 together with 2 TC, SEND and RCV threads.<br />
LDM threads are running on first 6 CPUS node #7 together with 2 TC and 1 MAIN and REPL<br />
threads with CPUs 107 and 110(Last one) not being used.</td></tr>
</tbody></table>
<br />
<br />
Calculating ProcessorAffinity mask for process is different depending on the function accepting the input.<br />
1) For cmd.exe START, the actual number passed is in HEX notation. The binary mask is composed so that the highest index CPU comes first:
<br />
<pre>BENCHMARK_CPU="111100000000"
000000001111
|__________|
|_12 CPUs__|
NUMA #0
CPU0 CPU11</pre> It is more convenient to provide the mask in binary so I convert setting to Hex value inside the script.<br />
The NUMA node to run on is specified as decimal integer.<br />
If you have provided the NUMA node # for the process to run on, not specifying ProcessorAffinity mask means "run on all CPUs within specified node".<br />
If you provide the wrong mask, process will fail to start. For example, I have 12 CPUs per NUMA node (socket) so providing the mask like "11111111111000" will fail.<br />
The approach works only on one NUMA node.<br />
<br />
2) Start Process expects decimal integer for mask. The rightmost "1" indicates usage of CPU #0 within Processor group assigned by Kernel scheduler in Round-Robin manner.
<br />
<pre> 000000000000000000000000000011111111111111111111 = 1048575d
|___________________48 CPUs____________________|
|__________||__________||__________||__________|
NUMA #7 NUMA #5 NUMA #3 NUMA #1
or, should the scheduler pick Processor group 0:
NUMA #6 NUMA #4 NUMA #2 NUMA #0
</pre> Start process takes (and returns) decimal value for ProcessorAffinity.<br />
It uses <b>late binding</b> so Process has to be up and running before assigning Affinity mask to it.<br />
You have no control over ProcessorGroup meaning Kernel scheduler is free to pick any NUMA node in Round-Robin fashion.<br />
<br />
3) Doing things "properly" (binding threads to CPUs). Or, how to calculate ThreadConfig for MySQL Cluster:<br />
<code>ThreadConfig=ldm={count=10,cpubind=88-91,100-105},tc={count=4,cpubind=94-95,106-107},send={count=2,cpubind=92-93},recv={count=2,cpubind=98,99},main={count=1,cpubind=109},rep={count=1,cpubind=109}</code> shows CPU indexes above total number of CPUs available on my test system (2x48=96). This has to do with the maximum capacity of Processor group which is 64. The designer of this functionality treats each Processor group found on system as <b>full</b> meaning it occupies 64 places for CPU index. This makes sense if you are going from the box with 48 CPUs in group (like mine) to a box with 64 CPUs in group as your ThreadConfig line will continue to work as expected. However, it requires some math to come to CPU indexes:
<br /><pre><small>
Processor group 0 |Processor group 1
CPU#0 CPU#47 CPU#63CPU#64 CPU#110 CPU#127
| AVAILABLE | RESERV || AVAILABLE | RESERV |
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXRRRRRRRRRRRRRRRRXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXRRRRRRRRRRRRRRRR</small></pre>
Now my ThreadConfig line makes sense:<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibu7mmLPfvuGD5D3uIqoRfFXlRC5R0HnmZJFcjKFpnLVKkuS4VOGrJ7_y5-JbEtIrs1Sm51gyPCUNEnXvAv-4lctQohWPZjSwD1LX8CwrX3cvj9LeeBETyuYcqpkYKcZYOlt1pwpbUgKTE/s1600/How-to-set-up-affinity-masks.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEibu7mmLPfvuGD5D3uIqoRfFXlRC5R0HnmZJFcjKFpnLVKkuS4VOGrJ7_y5-JbEtIrs1Sm51gyPCUNEnXvAv-4lctQohWPZjSwD1LX8CwrX3cvj9LeeBETyuYcqpkYKcZYOlt1pwpbUgKTE/s400/How-to-set-up-affinity-masks.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">LDM threads are running on first 4 CPUS node #5 (88-91) together with 2 TC (94,95), SEND (92,93) and RCV (98,99) threads.<br />
LDM threads are running on first 6 CPUS node #7 (100-105) together with 2 TC (106,107) and 1 MAIN and REPL (109)<br />
threads with CPUs 107 and 110(Last one) not being used.</td></tr>
</tbody></table>
<br />
<br />
<h3>
Conclusion:</h3>
o Windows use notion of Processor group. Machines with less than 64 CPUs have 1 Processor group thus your application runs exactly as before.<br />
o <b>Bug 1:</b> Affinity mask is only 64-bit wide so there is no way to have continuous index of CPUs inside the big box such as mine.<br />
o .NET System.Diagnostics.Process has no get/set of Processor group. At least a getter function was expected and a member of System.Diagnostics.Process disclosing this information.<br />
o <b>Bug 2:</b> Information on CPU Affinity mask obtained from .NET System.Diagnostics.Process is ambiguous.<br />
o 1 + 2, <b>bug 3:</b> There is no way I found to script pinning to individual CPUs that is complete.<br />
o <b>Feature request 1:</b> .NET System.Diagnostics.Process allows only for late binding of Affinity mask. Move Affinity to .NET System.Diagnostics.ProcessStartInfo.<br />
o <b>Feature request 2</b>, consolidate: The various approaches taken by Microsoft seem uncoordinated and incomplete. Even using START command requires decimal number for NUMA node index and hexadecimal number for Affinity mask. cmd.exe START and creation of thread objects allow for early binding of CPU Affinity mask while .NET System.Diagnostics.Process allows only late binding. And so on.<br />
o <b>Feature request 3</b>, give us TASKSET complement: Given all of the above, it is impossible to script the replacement for Linux TASKSET.<br />
o What will happen once single processors with more than 64 CPUs are available?<br />
o Mysql Cluster counts CPUs as if every existing Processor group is complete (has 64 CPUs).<br />
<br />
<br />
<br />
<br />
Tončihttp://www.blogger.com/profile/10650361408804728027noreply@blogger.com0tag:blogger.com,1999:blog-7850846940198540592.post-38432096294456616852016-02-29T13:16:00.000+01:002016-02-29T13:16:14.217+01:00Linux top command on Windows, further investigationsIn my <a href="http://toncigrgin.blogspot.com/2016/01/windows-perf-counters-blog8.html">previous post</a>, I spoke of "Normalized" and "Non-Normalized" CPU utilization values:<br />
--<br>
<small><h4>TABLE:</h4>
Foreword: Windows is not "process" based OS (like Linux) but rather "thread" based so all of the numbers relating to CPU usage are approximations. I did made a "proper" CPU per Process looping and summing up Threads counter (https://msdn.microsoft.com/en-us/library/aa394279%28v=vs.85%29.aspx) based on PID but that proved too slow given I have ~1 sec to deal with everything. CPU utilization using RAW counters with 1s delay between samples proved to produce a bit more reliable result than just reading Formatted counters but, again, too slow for my 1s ticks (collect sample, wait 1s, collect sample, do the math takes longer than 1s). Thus I use PerfFormatted counters in version 0.9RC.
<pre>
<small>Win32_PerfRawData_PerfProc_Process; <a href="https://msdn.microsoft.com/en-us/library/aa394277%28v=vs.85%29.aspx">Win32_PerfFormattedData_PerfProc_Process</a>.
<b>_PID_</b> Unique identified of the process.
<b>PPID</b> Unique identifier of the process that started this one.
<b>PrioB</b> Base priority.
<b>Name</b> Name of the process.
<b>CPUpt_(N)</b> % of CPU used by process. On machines with multiple CPUs,
this number can be over 100% unless you see _CPUpt_N caption which
means "Normalized" (i.e. CPUutilization / # of CPUs).
Toggle Normal/Normalized display by pressing the "p" key.
<b>Thds</b> # of threads spawned by the process.
<b>Hndl</b> # of handles opened by the process.
<b>WS(MB)</b> Total RAM used by the process. Working Set is, basically,
the set of memory pages touched recently by the threads belonging to
the process.
<b>VM(MB)</b> Size of the virtual address space in use by the process.
<b>PM(MB)</b> The current amount of VM that this process has reserved
for use in the paging files.</small>
</pre></small>However, my approach for displaying "Non-Normalized" CPU utilization didn't work :-/<br> <br>
Proper functionality of this feature is rather important for my job. Looking at "Normalized" CPU utilization values for a process does not tell you much. Say a process has CPU utilization of 100%. This just tells you there is at least 1 CPU that's fully utilized by the process but it does not tell you the overall utilization. "Non-Normalized" value sums CPU utilization over all CPUs that process uses. In my case, the test box has 8 Xeon processors with 6 physical and 6 virtual cores each totaling at 96 CPUs. The system is configured as such that NUMA node corresponds to 1 Xeon processor (socket). Thus, when my process utilizes entire NUMA node (socket) to the fullest, the CPU utilization for that process should be <b>number of CPUs per Numa/Socket</b> (12) x 100% which is <b>1200%</b>:<br>
<br>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhxGJnxO1atK1yZ5wxl1weQdxc2WxpBPR8LvxhdPSTm3Hyy09O8jlhwMz2OHGd41EXVy9343vHQnCvR62OD4OxIm4qMWBhzoPRT-ci2hs04osYxDHIV4IRTQ1tbRGRr8_p6DwZs4Et3wCeZ/s1600/Normal.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhxGJnxO1atK1yZ5wxl1weQdxc2WxpBPR8LvxhdPSTm3Hyy09O8jlhwMz2OHGd41EXVy9343vHQnCvR62OD4OxIm4qMWBhzoPRT-ci2hs04osYxDHIV4IRTQ1tbRGRr8_p6DwZs4Et3wCeZ/s320/Normal.png" /></a></div>
<br>
If the process scales correctly, I will see more NUMA nodes/Sockets light up while increasing the load:
<br>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZrY6C6EewPKdo0W6pV92_dJXMTlroafR_uRvC4uK3ZfzfiUfmZKj6EChsHMPe-52uuZvf29JUceJb1AWc3zMRWYudc-eIlygaUB_4ze5GUYlW7eCrfsprY4TXpWo3RRK7TDdae8ejsjhc/s1600/Many-nodes.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZrY6C6EewPKdo0W6pV92_dJXMTlroafR_uRvC4uK3ZfzfiUfmZKj6EChsHMPe-52uuZvf29JUceJb1AWc3zMRWYudc-eIlygaUB_4ze5GUYlW7eCrfsprY4TXpWo3RRK7TDdae8ejsjhc/s320/Many-nodes.png" /></a></div>
<br>
However, this does not tell me it is <b>my process of interest</b> that is using the CPUs. To confirm it, I need TOP script showing CPU utilization of above 1200%:
<br>
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHbDbhunUwru7XZh5NSvTa5jrcBX0FSL_lR5Nl1tIAUYhJAUc-2G-Y0StbBh9Ub9idWBcB18Vo2q1Irhgiy_SZlaoINu3RcptUujpbydaVT7nEZJPIVUo8joMpsngJiHG8wbu0cNm1NQq3/s1600/Top-fixed.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHbDbhunUwru7XZh5NSvTa5jrcBX0FSL_lR5Nl1tIAUYhJAUc-2G-Y0StbBh9Ub9idWBcB18Vo2q1Irhgiy_SZlaoINu3RcptUujpbydaVT7nEZJPIVUo8joMpsngJiHG8wbu0cNm1NQq3/s1600/Top-fixed.png" /></a></div>
<br>
This guarantees me mysqld process is running on more than 2 sockets (sysbench is taking up ~7 CPUs and I bet mydesktopservice is the one lighting up 3rd CPU in 2nd row).
<br>
<br>
<h4>How to make it work:</h4>
Heavy rework of <code>#region Tasks job</code> which is starting the "Processes" monitoring job was in order. First, I had to remove all of the below code:<small><code><pre>
<small><#
if ($CPUDSw) {
Get-CimInstance Win32_PerfFormattedData_PerfProc_Process |
select @{Name='_PID_'; Expression={$_.IDProcess}},
@{Name='PPID'; Expression={$_.CreatingProcessID}},
@{Name='PrioB'; Expression={$_.PriorityBase}},
@{Name='Name '; Expression={(($_.Name).PadRight(22)).substring
(0, [System.Math]::Min(22, ($_.Name).Length))}},
@{Name='_CPUpt__'; Expression={($_.PercentProcessorTime).ToString("0.00").PadLeft(8)}},
@{Name='Thds'; Expression={$_.ThreadCount}},
@{Name='Hndl'; Expression={$_.HandleCount}},
@{Name='WS(MB)'; Expression={[math]::Truncate($_.WorkingSet/1MB)}},
@{Name='VM(MB)'; Expression = {[math]::Truncate($_.VirtualBytes/1MB)}},
@{Name='PM(MB)'; Expression={[math]::Truncate($_.PageFileBytes/1MB)}} |
where { $_._PID_ -gt 0} | &$sb |
Select-Object -First $procToDisp | FT * -Auto 1> $pth
} else {
Get-CimInstance Win32_PerfFormattedData_PerfProc_Process |
select @{Name='_PID_'; Expression={$_.IDProcess}},
@{Name='PPID'; Expression={$_.CreatingProcessID}},
@{Name='PrioB'; Expression={$_.PriorityBase}},
@{Name='Name '; Expression={(($_.Name).PadRight(22)).substring
(0, [System.Math]::Min(22, ($_.Name).Length))}},
@{Name='_CPUpt_N'; Expression={"{0,8:N2}" -f ($_.PercentProcessorTime / $TotProc)}},
@{Name='Thds'; Expression={$_.ThreadCount}},
@{Name='Hndl'; Expression={$_.HandleCount}},
@{Name='WS(MB)'; Expression={[math]::Truncate($_.WorkingSet/1MB)}},
@{Name='VM(MB)'; Expression = {[math]::Truncate($_.VirtualBytes/1MB)}},
@{Name='PM(MB)'; Expression={[math]::Truncate($_.PageFileBytes/1MB)}} |
where { $_._PID_ -gt 0} | &$sb |
Select-Object -First $procToDisp | FT * -Auto 1> $pth
}
#></small>
</pre></code></small>and replace it with Get-Counter version:
<code><pre> $processes = Get-CimInstance Win32_PerfFormattedData_PerfProc_Process |
Select @{Name='_PID_'; Expression={$_.IDProcess}},
@{Name='PPID'; Expression={$_.CreatingProcessID}},
ElapsedTime,
@{Name='PrioB'; Expression={$_.PriorityBase}},
@{Name='Name'; Expression={($_.Name).ToLower()}},
@{Name='Thds'; Expression={$_.ThreadCount}},
@{Name='Hndl'; Expression={$_.HandleCount}},
@{Name='WS(MB)'; Expression={[math]::Truncate($_.WorkingSet/1MB)}},
@{Name='VM(MB)'; Expression = {[math]::Truncate($_.VirtualBytes/1MB)}},
@{Name='PM(MB)'; Expression={[math]::Truncate($_.PageFileBytes/1MB)}},
PoolNonpagedBytes, PoolPagedBytes, PercentProcessorTime |
Where { $_._PID_ -gt 0}
$Samples = (Get-Counter “\Process(*)\% Processor Time”).CounterSamples</pre></code><br>
Just noting <a href="https://technet.microsoft.com/en-us/library/hh849685%28v=wps.630%29.aspx">Get-Counter</a> example: <small><code><pre><small>PS:511 [HEL01]> (Get-Counter “\Process(*)\% Processor Time”).CounterSamples | FL *
...
Path : \\hel01\process(system)\% processor time
InstanceName : system
CookedValue : 0
RawValue : 3434062500
SecondValue : 131012088253272040
MultipleCount : 1
CounterType : Timer100Ns
Timestamp : 29.02.16 09:40:25
Timestamp100NSec : 131012124253270000
Status : 0
DefaultScale : 0
TimeBase : 10000000</small></pre></code></small>
Then I had to change the way of putting it all together:
<code><pre> if ($CPUDSw) {
$pcts = $Samples | Select @{Name=”IName"; Expression={($_.InstanceName).ToLower()}},
@{Name=”CPUU”;Expression={[Decimal]::Round(($_.CookedValue), 2)}}
$processes | select '_PID_', 'PPID', 'PrioB',
@{Name='Name '; Expression=
{
(($_.Name).PadRight(22)).substring(0, [System.Math]::Min(22, ($_.Name).Length))
}
},
@{Name='_CPUpt__'; Expression=
{
if ($pcts.IName.IndexOf($_.Name) -ge 0) {
($pcts.CPUU[[array]::IndexOf($pcts.IName, $_.Name)]).ToString("0.00").PadLeft(8)
}
}
},
'Thds', 'Hndl', 'WS(MB)', 'VM(MB)', 'PM(MB)' | &$sb | Select-Object -First $procToDisp | FT * -Auto 1> $pth
} else {
$pcts = $Samples | Select @{Name=”IName"; Expression={($_.InstanceName).ToLower()}},
@{Name=”CPUU”;Expression={[Decimal]::Round(($_.CookedValue / $TotProc), 2)}}
$processes | select '_PID_', 'PPID', 'PrioB',
@{Name='Name '; Expression=
{
(($_.Name).PadRight(22)).substring(0, [System.Math]::Min(22, ($_.Name).Length))
}
},
@{Name='_CPUpt_N'; Expression=
{
if ($pcts.IName.IndexOf($_.Name) -ge 0) {
($pcts.CPUU[[array]::IndexOf($pcts.IName, $_.Name)]).ToString("0.00").PadLeft(8)
} else {
#Not found (yet). Take what you have :-/
($_.PercentProcessorTime).ToString("0.00").PadLeft(8)
}
}
},
'Thds', 'Hndl', 'WS(MB)', 'VM(MB)', 'PM(MB)' | &$sb | Select-Object -First $procToDisp | FT * -Auto 1> $pth
}</pre></code>
Since Get-Counter, by default, takes samples 1 second apart:
<small><code><pre><small>PS:507 [HEL01]> Measure-Command{(Get-Counter “\Process(*)\% Processor Time”).CounterSamples}
Days : 0
Hours : 0
Minutes : 0
Seconds : 1
Milliseconds : 18
Ticks : 10189709
TotalDays : 1.17936446759259E-05
TotalHours : 0.000283047472222222
TotalMinutes : 0.0169828483333333
TotalSeconds : 1.0189709
TotalMilliseconds : 1018.9709</small></pre></code></small> I also abandoned all of the code relating to Timer:
<small><code><pre><small> #$sw = New-Object Diagnostics.Stopwatch
do {
#$sw.Start()
...
}
#$sw.Stop()
#if ($sw.ElapsedMilliseconds -lt 1000) {
# Start-Sleep -Milliseconds (1000-$sw.ElapsedMilliseconds)
#}
#$sw.Reset()
} while ($true)
#$sw = $null</small></pre></code></small>
<br>
So now it works! I do not know right now when I will be able to release the new version so stay tuned.<br><br>
<h4>Final thoughts:</h4>
I have hit many many problems in Windows during this testing. Just note, for example, the use of ToLower() in <code>($_.InstanceName).ToLower()</code> but this is something for the new blog post. This one is about TOP script.<br>
<br>Tončihttp://www.blogger.com/profile/10650361408804728027noreply@blogger.com0tag:blogger.com,1999:blog-7850846940198540592.post-57113905709864080382016-01-21T08:32:00.002+01:002016-01-21T08:32:21.782+01:00Using Powershell to implement Linux top command on WindowsWelcome to the final blog in Windows PerfCounters and Powershell series and sorry for the delay. The purpose of this blog is to explain the inner workings of top-script.ps1 script and practical usage of Performance counters on Windows through Powershell. It is intended for people who want Linux top - like tool on Windows.<br />
<br />
The script is a part of and available in our existing <a href="https://dev.mysql.com/downloads/benchmarks.html">benchmarking package (dbt2-0.37.50.10)</a> developed by Mikael Ronstrom.<br />
<br />
<h3>
On Top:</h3>
If you ever did benchmarking on Linux or simply wondered "where did all my resources go", top is your best friend. Since this post is not about Linux, you can google "Linux top explained" for more details.<br />
<br />
<br />
<h3>
On Performance counters:</h3>
To learn about Windows PerfCounters, please refer to my previous <a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog1.html">blog</a> entries in this series. I will be addressing <code>System.Diagnostics</code> class as just <code>Diagnostics</code>.<br />
<br />
<br />
<h3>
On Powershell:</h3>
For ages, Windows users were looking at bash wondering why they do not have anything similar to it. After much trial and error, Microsoft delivered Powershell. In my humble opinion, Powershell is simply great! As per difference between Powershell and bash I would mention just one; Powershell pipe passes objects while bash pipe passes plain text.<br />
<br />
<br />
<h3>
On script itself:</h3>
Type "perfmon deleting files" in Google and you'll see why I made this script ;-) Joke aside, we have a mature testing/benchmarking framework written in bash and wanted the same look and feel on Windows. Top script is just the latest piece of that effort.<br />
I undertook this work since I firmly believe in native tools when dealing with performance issues. If I was just after measuring performance delatas between versions, some generic tool written for some other platform, such as in Perl or similar, would have been good enough. But, IMO, it would not have been fair to non-native OS.<br />
Also, studying native tools is an integral part of studying the OS itself which is something you can not tackle performance issues without.<br />
<br />
The script will evolve naturally to cover for the information we need in our everyday work.<br />
<br />
Using Windows performance counters through PowerShell CIM classes it is possible to gather stats on computer performance. The script functions like this:<br />
o Main code starts 2 background jobs; one for collecting details for header table ("Top_Header_Job") and one for processes table ("Top_Processes_Job").<br />
o Each of the jobs then collects stats and writes them into a file ("header" job to TempDir\headerres.txt file, "tasks" job to TempDir\thrprocstats.txt) which is, in turn, read by script main code. Main code uses TempDir\topcfg.txt to pass info to tasks (currently, the field to sort the table by). All of the files are overwritten each time so there is not much data in them.<br />
o After the files are read by main code, the data is displayed. To be able to properly position the output on screen, I used various <code>[console]</code> functions not available in PowerShell_ISE.<br />
<br />
<br />
<h3>
Requirements:</h3>
The script can not be run in _ISE. Use Powershell console.<br />
The script requires PS 3+.<br />
The script requires at least 50 x 80 console window.<br />
The script might work with .NET FW older than 4 but it was tested only with .NET 4.5.x.<br />
<br />
<br />
<h3>
Getting started:</h3>
1) Put script somewhere.<br />
2) Start PowerShell (NOT PS_ISE)<br />
3) cd to "somewhere" directory<br />
4) .\top-script.ps1<br />
a) Get-Help .\top-script.ps1<br />
b) Get-Help .\top-script.ps1 -Examples<br />
5) While script is running you can use (single key) shortcuts:<br />
q - Quit<br />
m - Sort by process WS (occupied RAM) DESC, CPUpt DESC<br />
p - Sort by process CPU utilization DESC, WS(MB) DESC; Non-Normalized/Normalized.<br />
<b>Note</b>: Script is started with CPU utilization by process as de-normalized (i.e. on multi-CPU boxes, this value can be well over 100%). To display normalized values (i.e. "non-normalized" value / # of CPUs), just press "p" again. IF non-normalized value is the source for data, the column title will be '_CPUpt__'. Normalized CPU utilization value (CPUpt / # of CPUs) will display as '_CPUpt_N'.<br />
n - Sort by process Name ASC<br />
r - Sort by process PID ASC, CPUpt DESC<br />
+ - Display individual CPU's. Comma separated list of values (i.e. 0,1,2).<br />
1 - Display individual Processors (Sockets). Comma separated list of values (i.e. 0,1,2).<br />
<b>Note</b>: Script displays either Socket load or CPU load.<br />
- - Cancel displaying individual CPUs/Sockets.<br />
<br />
<br />
<h3>
The output:</h3>
<pre><samp>09:21:17, Uptime:00d:00h:36m, Users: 1, # thds ready but queued for CPU: 0
-------------------------------------------------------------------------------
| RUNNING | CPU | RAM[MB] |
-------------------------------------------------------------------------------
| services: 104 | Sys: 50.78%(P 4.30%/U 46.48%)| Installed: 8192 |
| processes: 126 | Idle:49.22% | HW reserv: 320.77 |
| threads: 1483 | HWint: 7087/0.38% | Visible: 7871.23 |
| handles: 32696 | SWint: 278/0.38% | Available: 5136 |
| CoSw/s: 6351 | High-prio thd exec: 50.38% | Modified: 117.07 |
| | Total # of cores: 4 | Standby: 3936.44 |
| | | PagesIn/Read ps: 2 |
-------------------------------------------------------------------------------
_PID_ PPID PrioB Name CPUpt Thds Hndl WS(MB) VM(MB) PM(MB)
----- ---- ----- ---- ----- ---- ---- ------ ------ ------
3916 852 8 mcshield 10.14 53 490 46 225 100
1836 5452 8 powershell#2 1.09 15 377 67 616 46
7436 5452 8 powershell#1 0.72 17 556 85 624 67
3984 156 8 WmiPrvSE 0.72 9 303 14 56 9
7024 156 8 WmiPrvSE#2 0.72 7 201 10 52 7
5452 5148 8 powershell 0.36 18 454 90 628 79
2292 852 8 FireSvc 0.36 28 539 10 156 36
7864 5148 8 thunderbird 0.00 52 630 293 656 264
2880 5148 8 powershell_ise 0.00 12 427 181 869 164
5148 3860 8 explorer 0.00 25 810 81 267 53
7028 6648 8 googledrivesync#1 0.00 29 712 76 193 64
1124 852 8 svchost#5 0.00 45 1560 46 187 31
6508 5148 8 sidebar 0.00 20 433 39 195 20
816 796 13 csrss#1 0.00 10 765 35 126 3
6592 5148 8 iCloudServices 0.00 16 442 32 167 18
6868 6764 8 pcee4 0.00 7 202 32 612 32
1976 1052 13 dwm 0.00 5 135 31 140 26
3400 156 8 WmiPrvSE#1 0.00 12 297 28 91 22
1088 852 8 svchost#4 0.00 21 584 27 126 14
3648 852 8 dataserv 0.00 10 510 24 224 21
980 852 8 svchost#2 0.00 25 575 23 119 26
2404 852 8 PresentationFontCache 0.00 6 149 21 506 28
...
</samp></pre>
<h4>HEADER data</h4>
Current time, uptime, # of active users, # of threads per CPU that are ready for execution but can't get CPU cycles (obviously, you want to keep this as low as possible (<= 2)).<pre>
<b>RUNNING section</b>
# of services in Started state
# of user processes
# of threads spawned
# of handles open
# of context switches per second
<b>CPU section</b>
% of CPU used (% used by privileged instr. / % used by user instr.)
% of CPU consumed by Idle process.
# of HW interrupts per sec./% of CPU used to service HW interrupts
# of SW interrupts queued for servicing per sec./
% of CPU used to service SW interrupts
% of CPU consumed by high-priority threads execution.
# of phys. and virt. cores. Here, 4 is Dual-Core with HT enabled.
<b>RAM[MB] section</b>
Installed RAM.
RAM reserved by Windows for HW.
Amount of RAM user actually sees.
Amount of available RAM for user processes.
Amount of RAM marked as "Modified".
Amount of RAM marked as "Standby" (cached).
Ratio between Memory\Pages Input/sec and Memory\Page Reads/sec.
Number of pages per disk read. Should keep below 5.
</pre>
<h4>TABLE data</h4><pre>
_PID_ Unique identified of the process.
PPID Unique identifier of the process that started this one.
PrioB Base priority.
Name Name of the process.
CPUpt % of CPU used by process.
Thds # of threads spawned by the process.
Hndl # of handles opened by the process.
WS(MB) Total RAM used by the process. Working Set is, basically,
the set of memory pages touched recently by the threads belonging
to the process.
VM(MB) Size of the virtual address space in use by the process.
PM(MB) The current amount of VM that this process has reserved for
use in the paging files.
</pre>
<br />
<h3>Longer explanation of the values:</h3>
<h4>HEADER:</h4>
Foreword: Since Windows is not "process" based OS (like Linux) it is impossible to calculate the "System load". The next best thing is CPU queue length (see below).<pre>
<b>Uptime</b>: <small>Diagnostics.PerformanceCounter("System", "System Up Time")</small>
<b>Users</b>: <small>WMI query using query.exe tool which should be a part of your Windows.
query user /server:localhost
Number of users currently logged in. If no query.exe, the value is -1.</small>
<b># thds ready but queued for CPU</b>:
<small>Diagnostics.PerformanceCounter("System", "Processor Queue Length")
How many threads are in the processor queue ready to be executed but not
currently able to use cycles. Windows OS has single queue length counter
thus the value displayed is counter value divided with number of CPU's.
<a href="https://technet.microsoft.com/en-us/library/cc940375.aspx">Link.</a></small>
</pre><br />
<b>RUNNING section:</b><pre>
<b>services</b>:
<small>(Get-Service | Where-Object {$_.Status -ne 'Stopped'} | Measure-Object).Count
Total # of services actually running.</small>
<b>processes</b>:
<small>Diagnostics.PerformanceCounter("System", "Processes")
Total number of user processes running.
<a href="https://msdn.microsoft.com/en-us/library/aa394277%28v=vs.85%29.aspx">Link.</a></small>
<b>threads</b>:
<small>Diagnostics.PerformanceCounter("System", "Threads")
Total # of threads spawned.</small>
<b>handles</b>:
<small>Diagnostics.PerformanceCounter("Process", "Handle Count")
Total # of open handles.</small>
<b>CoSw/s</b>:
<small>Diagnostics.PerformanceCounter("System", "Context Switches/sec")
Context switching happens when a higher priority thread pre-empts a lower
priority thread that is currently running or when a high priority thread
blocks. High levels of context switching can occur when many threads share
the same priority level. This often indicates that there are too many
threads competing for the processors on the system. If you do not see much
processor utilization and you see very low levels of context switching, it
could indicate that threads are blocked.
<a href="https://msdn.microsoft.com/en-us/library/aa394279%28v=vs.85%29.aspx">Link.</a></small>
</pre><br />
<b>CPU section</b>:<br />
Foreword: Windows OS has special thread called "Idle" which consumes free CPU cycles thus these counters return values relating to this one. Also, Windows are not "process" based but rather "thread" based so all of these numbers are approximations. This is even more important in the TABLE which shows CPU utilization per process (see explanation there). Most of these counters are multi-instance so instance name is '_Total' (ie. CPU utilization in total as opposed to per NUMA node, Core, CPU...).<pre>
<b>Sys: nn.nn%(P mm.mm%/U zz.zz%)</b>:
<small>Diagnostics.PerformanceCounter("Processor Information","% Processor Time"),
Diagnostics.PerformanceCounter("Processor Information","% Privileged Time"),
Diagnostics.PerformanceCounter("Processor Information","% User Time").
First number shows, effectively, % of cycles CPU(s) didn't spend running the
Idle thread. Second number is the time CPU(s) spent on executing Privileged
instructions while third is time CPU(s) spent executing user-mode instructions.
For example, when your application calls operating system functions (say to
perform file or network I/O or to allocate memory), these operating system
functions are executed in Privileged mode.
<a href="https://msdn.microsoft.com/en-us/library/aa394271%28v=vs.85%29.aspx">Link.</a></small>
<b>Idle</b>:
<small>Diagnostics.PerformanceCounter("Processor Information", "% Idle Time")
<a href="https://msdn.microsoft.com/en-us/library/aa394271%28v=vs.85%29.aspx">Link.</a></small>
<b>HWint</b>:
<small>Diagnostics.PerformanceCounter("Processor Information","Interrupts/sec"),
Diagnostics.PerformanceCounter("Processor Information","% Interrupt Time").
Rate of hardware interrupts per second and a percent of CPU time this takes.
<a href="https://msdn.microsoft.com/en-us/library/aa394271%28v=vs.85%29.aspx">Link.</a></small>
<b>SWint</b>:
<small>Diagnostics.PerformanceCounter("Processor Information","DPCs Queued/sec"),
Diagnostics.PerformanceCounter("Processor Information","% DPC Time").
Rate at which software interrupts are queued for execution and a % of CPU time
this takes.
<a href="https://msdn.microsoft.com/en-us/library/aa394271%28v=vs.85%29.aspx">Link.</a></small>
<b>High-prio thd exec</b>:
<small>Diagnostics.PerformanceCounter("Processor Information","% Priority Time").
CPU utilization by high priority threads.
Link: Can't find any links in MSDN...</small>
<b>Total # of cores</b>:
<small>(Get-CimInstance Win32_ComputerSystem).NumberOfLogicalProcessors
Number of physical and virtual cores present.</small>
</pre><br />
<b>RAM[MB] section</b>:<pre>
<b>Installed</b>:
<small>(GCIM -class "cim_physicalmemory" | Measure-Object Capacity -Sum).Sum/1024/1024</small>
<b>HW reserv</b>:
<small>Installed - Visible ;-)</small>
<b>Visible</b>:
<small>(Get-CimInstance win32_operatingsystem).TotalVisibleMemorySize</small>
<b>Available</b>:
<small>Diagnostics.PerformanceCounter("Memory","Available MBytes")</small>
<b>Modified</b>:
<small>Diagnostics.PerformanceCounter("Memory","Modified Page List Bytes")</small>
<b>Standby</b>:
<small>Diagnostics.PerformanceCounter("Memory","Standby Cache Core Bytes") +
Diagnostics.PerformanceCounter("Memory",
"Standby Cache Normal Priority Bytes") +
Diagnostics.PerformanceCounter("Memory","Standby Cache Reserve Bytes")
Basically, cache memory.</small>
<b> PagesIn/Read ps</b>:
<small>Diagnostics.PerformanceCounter("Memory","Pages Input/sec")/
Diagnostics.PerformanceCounter("Memory","Page Reads/sec")
Ratio between Memory\Pages Input/sec and Memory\Page Reads/sec which
is number of pages per disk read. Should keep below 5.</small>
</pre>
<br />
<h4>TABLE:</h4>
Foreword: Windows is not "process" based OS (like Linux) but rather "thread" based so all of the numbers relating to CPU usage are approximations. I did made a "proper" CPU per Process looping and summing up Threads counter (https://msdn.microsoft.com/en-us/library/aa394279%28v=vs.85%29.aspx) based on PID but that proved too slow given I have ~1 sec to deal with everything. CPU utilization using RAW counters with 1s delay between samples proved to produce a bit more reliable result than just reading Formatted counters but, again, too slow for my 1s ticks (collect sample, wait 1s, collect sample, do the math takes longer than 1s). Thus I use PerfFormatted counters in version 0.9RC.
<pre>
Win32_PerfRawData_PerfProc_Process; Win32_PerfFormattedData_PerfProc_Process
<a href="https://msdn.microsoft.com/en-us/library/aa394277%28v=vs.85%29.aspx">Link.</a>
<b>_PID_</b> Unique identified of the process.
<b>PPID</b> Unique identifier of the process that started this one.
<b>PrioB</b> Base priority.
<b>Name</b> Name of the process.
<b>CPUpt_(N)</b> % of CPU used by process. On machines with multiple CPUs,
this number can be over 100% unless you see _CPUpt_N caption which
means "Normalized" (i.e. CPUutilization / # of CPUs).
Toggle Normal/Normalized display by pressing the "p" key.
<b>Thds</b> # of threads spawned by the process.
<b>Hndl</b> # of handles opened by the process.
<b>WS(MB)</b> Total RAM used by the process. Working Set is, basically,
the set of memory pages touched recently by the threads belonging to
the process.
<b>VM(MB)</b> Size of the virtual address space in use by the process.
<b>PM(MB)</b> The current amount of VM that this process has reserved
for use in the paging files.
</pre>
Note that it is possible to display CPU/Socket data for chosen HW by pressing + or 1 keys, entering 0-based index and separating multiple values by ,:
<pre><samp>
User Priv Idle HWin SWIn User Priv Idle HWin SWIn
------------------------------------- -------------------------------------
%CPU 0: 47, 5, 47, 0, 0 %CPU 1: 0, 0, 100, 0, 0
%CPU 2: 35, 11, 52, 0, 0 %CPU 3: 5, 0, 94, 0, 0
</samp></pre>
The input here was 0,1,2,3 thus displaying data about first 4 cores. The CPU/Socket data is displayed between the
Header and the Table areas reducing the number of visible processes. To remove this information from screen, just press
"-" key.
<br />
<br />
<h3>
INNER WORKINGS:</h3>
In general, script output comprises of Header part and Table part showing details on processes. In-between the two, you can show Processor/Core info. There are two background jobs started to accomplish this; "Top_Header_Job" & "Top_Processes_Job". The data about individual processors/cores is calculated in main script body.<br />
<br />
Script starts with my usual checks, proceeds to variable declaration part where I initialize some of the performance counters (which takes time) and then starts Header and Processes jobs. The jobs itself follow the same logic. I.e. I first start perfcounter instances (which takes time) and then loop through values passing them back in <a href="http://toncigrgin.blogspot.hr/2015/10/communication-between-powershell-script.html">file</a>.<br />
<br />
Main script body collects the data from files refreshing the display. Also, main script is in charge of displaying individual processor/core data as well as monitoring the keyboard input. This means CTRL+C will <a href="http://toncigrgin.blogspot.hr/2015/10/handling-keyboard-input-and-ctrlc-in.html">NOT work</a> but you can still stop the script with CTRL+BREAK:<br />
<code>[console]::TreatControlCAsInput = $true</code><br />
Regular way to exit is pressing the "q" key.<br />
After you press the "q" key, cleanup code is executed, stopping the background jobs and removing temporary files used for communication. It's worth noting that cleanup code does not throw any errors. This is because nothing bad can happen. Files are less than 1kB in total while background jobs can be stopped either via trick described below or simply by exiting Powershell console.<br />
<br />
Lets go deeper into the regions of code now. First region is <b>Check</b> which I described in <a href="http://toncigrgin.blogspot.hr/2015/10/regular-checks-before-running.html">October 2015 blog</a> so no need to repeat myself. Next is <b>Variable Declarations</b> region where I gather one-time top-level data, mainly related to CPU topology using tricks described in <a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog3.html">Blog 3</a> and <a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog4.html">Blog 4</a> by manipulating Instances as described in <a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog1.html">Blog 1</a> of this series. Executing this part takes couple of seconds.<br />
<br />
Next thing is to start the Header job. It takes argument (total number of cores) from the call and proceeds with initializing various counters. As with all initializations, this also takes couple of seconds. Main DO loop starts the timer to ensure samples are collected in 1 second intervals. Also, it checks if you have query.exe tool installed and determines the number of active users, if the tool exists, or displays -1 if it doesn't. There are other ways of determining number of logged users but they are all too slow for 1s tick. After forming the resulting lines, I use <code>[System.IO.StreamWriter]</code> to record them to Env:\TEMP headerres.txt file. The control is then returned to main script which waits for Env:\TEMP headerres.txt file (or 20s, whichever comes first).<br />
<br />
Next step is to start the Tasks job which will collect data about running processes. As opposed to Task manager, I show background processes (ie. services) too. Worth noting is that, due to timing issues, I use Process (Win32_PerfFormattedData_PerfProc_Process) and not Thread (win32_PerfFormattedData_PerfProc_Thread) counters.<br />
Since Windows is *thread* based (meaning a Process is just a container for Threads doing the work) this actually means scarifying some of the accuracy (for example CPU utilization data) in favour of faster and smoother execution:<br />
<div style='overflow-x:auto;overflow-y:hidden;white-space: nowrap;'>
<pre 100="" 18px="" 1em="" 1px="" auto="" consolas="" courier="" font-size:="" font-style:="" font-weight:="" itstream="" line-height:="" mono="" monospace="" new="" normal="" overflow:="" padding:="" sans="" style="background-color: #0f192a; border: 1px dashed #999999; color: #cfecec;" vera="" width:="">
#(Active) Code when using Process counter:
<code style="color: #b43d3d;">Get-CimInstance</code> Win32_PerfFormattedData_PerfProc_Process |
<code style="color: #b43d3d;">Select</code> @{Name=<code style="color: #1dc116;">'_PID_'</code>; Expression={<code style="color: #AA7700;">$_.IDProcess</code>}},
@{Name=<code style="color: #1dc116;">'PPID'</code>; Expression={<code style="color: #AA7700;">$_.CreatingProcessID</code>}},
@{Name=<code style="color: #1dc116;">'PrioB'</code>; Expression={<code style="color: #AA7700;">$_.PriorityBase</code>}},
@{Name=<code style="color: #1dc116;">'Name '</code>; Expression={
((<code style="color: #AA7700;">$_.Name</code>).PadRight(22)).substring(0, [System.Math]::Min(22, (<code style="color: #AA7700;">$_.Name</code>).Length))
}},
@{Name=<code style="color: #1dc116;">'_CPUpt__'</code>; Expression={(<code style="color: #AA7700;">$_.PercentProcessorTime</code>).ToString(<code style="color: #1dc116;">"0.00"</code>).PadLeft(8)}},
@{Name=<code style="color: #1dc116;">'Thds'</code>; Expression={<code style="color: #AA7700;">$_.ThreadCount</code>}},
@{Name=<code style="color: #1dc116;">'Hndl'</code>; Expression={<code style="color: #AA7700;">$_.HandleCount</code>}},
@{Name=<code style="color: #1dc116;">'WS(MB)'</code>; Expression={[math]::Truncate(<code style="color: #AA7700;">$_.WorkingSet</code>/1MB)}},
@{Name=<code style="color: #1dc116;">'VM(MB)'</code>; Expression = {[math]::Truncate(<code style="color: #AA7700;">$_.VirtualBytes</code>/1MB)}},
@{Name=<code style="color: #1dc116;">'PM(MB)'</code>; Expression={[math]::Truncate(<code style="color: #AA7700;">$_.PageFileBytes</code>/1MB)}} | #,
<code style="color: #b43d3d;">Where</code> { <code style="color: #AA7700;">$_._PID_</code> -gt 0} | &<code style="color: #AA7700;">$sb</code> |
<code style="color: #b43d3d;">Select</code> -First <code style="color: #AA7700;">$procToDisp</code> | <code style="color: #b43d3d;">FT</code> * -Auto 1> <code style="color: #AA7700;">$ToFile</code> </pre></div>
Note: Script-block $sb is used just for sorting the resultset depending on keyboard input.<br />
Note: "Name=" is the same as writing "Label=". Both can be abbreviated so the expression becomes @{L=...";"E={...}}.<br />
<br />
<div style='overflow-x:auto;overflow-y:hidden;white-space: nowrap;'>
<pre 100="" 18px="" 1em="" 1px="" auto="" consolas="" courier="" font-size:="" font-style:="" font-weight:="" itstream="" line-height:="" mono="" monospace="" new="" normal="" overflow:="" padding:="" sans="" style="background-color: #0f192a; border: 1px dashed #999999; color: #cfecec;" vera="" width:="">
#(More precise but slower) Code when scanning recursively the Thread counter:
#Get the CPU utilization percentages by summing up threads over particular process
<code style="color: #AA7700;">$pcts</code> = <code style="color: #b43d3d;">Get-CimInstance</code> win32_perfformatteddata_perfproc_thread -Property IDProcess,
PercentProcessorTime | <code style="color: #b43d3d;">Group</code> -Property IDProcess | <code style="color: #b43d3d;">Foreach</code> {
<code style="color: #b43d3d;">New-Object</code> PSObject -Property @{
PID = (<code style="color: #AA7700;">$_.Group.IDProcess</code> | Select -First 1)
CPUpt = <code style="color: #1dc116">"{0,5:N2}"</code> -f ((<code style="color: #AA7700;">$_.Group</code> | <code style="color: #b43d3d;">Measure-Object</code> -Property PercentProcessorTime -Sum).Sum)
}
}
#Pair with Process data:
<code style="color: #b43d3d;">Get-CimInstance</code> Win32_PerfFormattedData_PerfProc_Process |
<code style="color: #b43d3d;">Select</code> @{Name=<code style="color: #1dc116">'_PID_'</code>; Expression={<code style="color: #AA7700;">$_.IDProcess</code>}},
@{Name=<code style="color: #1dc116">'PPID'</code>; Expression={<code style="color: #AA7700;">$_.CreatingProcessID</code>}},
@{Name=<code style="color: #1dc116">'PrioB'</code>; Expression={<code style="color: #AA7700;">$_.PriorityBase</code>}},
@{Name=<code style="color: #1dc116">'Name'</code>; Expression={(<code style="color: #AA7700;">$_.Name</code>).PadRight(25)}},
@{Name=<code style="color: #1dc116">'CPUpt'</code>; Expression={<code style="color: #AA7700;">$pcts.CPUpt</code>[[array]::IndexOf(<code style="color: #AA7700;">$pcts.PID</code>, <code style="color: #AA7700;">$_.IDProcess</code>)]}},
@{Name=<code style="color: #1dc116">'Thds'</code>; Expression={<code style="color: #AA7700;">$_.ThreadCount</code>}},
@{Name=<code style="color: #1dc116">'Hndl'</code>; Expression={<code style="color: #AA7700;">$_.HandleCount</code>}},
@{Name=<code style="color: #1dc116">'WS(MB)'</code>; Expression={[int](<code style="color: #AA7700;">$_.WorkingSet</code>/1MB)}},
@{Name=<code style="color: #1dc116">'VM(MB)'</code>; Expression = {[int](<code style="color: #AA7700;">$_.VirtualBytes</code>/1MB)}},
@{Name=<code style="color: #1dc116">'PM(MB)'</code>; Expression={[int](<code style="color: #AA7700;">$_.PageFileBytes</code>/1MB)}} |
<code style="color: #b43d3d;">Where</code> { <code style="color: #AA7700;">$_.Name</code> -notmatch <code style="color: #1dc116">"_Total"</code> -and <code style="color: #AA7700;">$_.Name</code> -notmatch <code style="color: #1dc116">"Idle"</code>} | &<code style="color: #AA7700;">$sb</code> |
<code style="color: #b43d3d;">Select</code> -First <code style="color: #AA7700;">$procToDisp</code> | FT * -Auto 1> <code style="color: #AA7700;">$ToFile </code></pre></div>
Note: Script-block $sb is used just for sorting the resultset depending on keyboard input.<br />
<br />
There is one more way of doing this and that is by expanding Process perf object. I use this approach when checking for congestion on thread level (<a href="https://msdn.microsoft.com/en-us/library/windows/desktop/aa394494%28v=vs.85%29.aspx">MSDN</a>):
<div style='overflow-x:auto;overflow-y:hidden;white-space: nowrap;'>
<pre 100="" 18px="" 1em="" 1px="" auto="" consolas="" courier="" font-size:="" font-style:="" font-weight:="" itstream="" line-height:="" mono="" monospace="" new="" normal="" overflow:="" padding:="" sans="" style="background-color: #0f192a; border: 1px dashed #999999; color: #cfecec;" vera="" width:="">
#Run once:
#Header row, initialize output file:
<code style="color: #1dc116">"PID,Process,ThdID,CPU time (s),PctUser,PctPriv,State,WaitR,PrioLvL,PrioShift,IdealProc,ProcAff"</code> |
<code style="color: #b43d3d;">Out-File</code> E:\test\thds.csv
#PIDs of interest to me:
<code style="color: #AA7700;">$Processes</code> = <code style="color: #b43d3d;">Get-Process</code> |
<code style="color: #b43d3d;">Where</code> {(<code style="color: #AA7700;">$_.ProcessName</code> -match <code style="color: #1dc116">"mysql"</code>) -or (<code style="color: #AA7700;">$_.ProcessName</code> -match <code style="color: #1dc116">"ndb"</code>) -or (<code style="color: #AA7700;">$_.ProcessName</code> -match <code style="color: #1dc116">"sysben"</code>)} |
<code style="color: #b43d3d;">Sort</code> -Property ID
</code></pre></div>
Note: If you check the value of <code>$Processes</code> variable here, you will notice something like
<pre><code>Id : 1996
...
Threads : {2000, 2012, 2016, 2040...}
...</code></pre>meaning Threads member is actually an object and can be expanded to show more data:
<pre><code>PS > $Processes.Threads
BasePriority : 8
CurrentPriority : 9
Id : 1972
IdealProcessor :
PriorityBoostEnabled :
PriorityLevel :
PrivilegedProcessorTime :
StartAddress : 2006300688
StartTime :
ThreadState : Wait
TotalProcessorTime :
UserProcessorTime :
WaitReason : UserRequest
ProcessorAffinity :
Site :
Container :
...</code></pre>
<div style='overflow-x:auto;overflow-y:hidden;white-space: nowrap;'>
<pre 100="" 18px="" 1em="" 1px="" auto="" consolas="" courier="" font-size:="" font-style:="" font-weight:="" itstream="" line-height:="" mono="" monospace="" new="" normal="" overflow:="" padding:="" sans="" style="background-color: #0f192a; border: 1px dashed #999999; color: #cfecec;" vera="" width:="">
#Run following in loop, append result to file
#Threads belonging to PIDs of interest.
<code style="color: #b43d3d;">Foreach</code> (<code style="color: #AA7700;">$Process</code> in <code style="color: #AA7700;">$Processes</code>) {
<code style="color: #AA7700;">$ProcessThds</code> = <code style="color: #AA7700;">$Process</code> | <code style="color: #b43d3d;">Select</code> -ExpandProperty Threads | <code style="color: #b43d3d;">Sort</code> -Property ID
<code style="color: #b43d3d;">Foreach</code> (<code style="color: #AA7700;">$ProcessThd</code> in <code style="color: #AA7700;">$ProcessThds</code>) {
<code style="color: #AA7700;">$ProcName</code> = @{L=<code style="color: #1dc116">"Name"</code>;E={ <code style="color: #AA7700;">$Process.ProcessName</code>}}
<code style="color: #AA7700;">$ProcID</code> = @{L=<code style="color: #1dc116">"PID"</code>;E={ <code style="color: #AA7700;">$Process.Id</code>}}
<code style="color: #AA7700;">$ThdID</code> = @{L=<code style="color: #1dc116">"ThreadID"</code>;E={ <code style="color: #AA7700;">$ProcessThd.Id</code> }}
<code style="color: #AA7700;">$CPUTime</code> = @{L=<code style="color: #1dc116">"CPU Time (Sec)"</code>;E={ [math]::round(<code style="color: #AA7700;">$ProcessThd.TotalProcessorTime.TotalSeconds</code>,2) }}
<code style="color: #AA7700;">$UsrCPUTime</code> = @{L=<code style="color: #1dc116">"User CPU Time (%)"</code>;E={ [math]::round(((<code style="color: #AA7700;">$ProcessThd.UserProcessorTime.ticks</code> /
<code style="color: #AA7700;">$ProcessThd.TotalProcessorTime.ticks</code>)*100),1) }}
<code style="color: #AA7700;">$State</code> = @{L=<code style="color: #1dc116">"State"</code>;E={ <code style="color: #AA7700;">$ProcessThd.ThreadState</code> }}
<code style="color: #AA7700;">$WR</code> = @{L=<code style="color: #1dc116">"WaitR"</code>;E={ <code style="color: #AA7700;">$ProcessThd.WaitReason</code>}}
<code style="color: #AA7700;">$PrioDelta</code> = @{L=<code style="color: #1dc116">"PrioSh"</code>;E={ <code style="color: #AA7700;">$ProcessThd.CurrentPriority</code> - <code style="color: #AA7700;">$ProcessThd.BasePriority</code>}}
<code style="color: #AA7700;">$IdProc</code> = @{L=<code style="color: #1dc116">"Ideal proc"</code>;E={ <code style="color: #AA7700;">$ProcessThd.IdealProcessor</code>}}
<code style="color: #AA7700;">$ProcAf</code> = @{L=<code style="color: #1dc116">"Proc affinity"</code>;E={ <code style="color: #AA7700;">$ProcessThd.ProcessorAffinity</code>}}
<code style="color: #AA7700;">$PrioLvL</code> = @{L=<code style="color: #1dc116">"Prio level"</code>;E={ <code style="color: #AA7700;">$ProcessThd.PriorityLevel</code>}}
<code style="color: #AA7700;">$PrivCPU</code> = @{L=<code style="color: #1dc116">"Privil CPU"</code>;E={ [math]::round(((<code style="color: #AA7700;">$ProcessThd.PrivilegedProcessorTime.ticks</code> /
<code style="color: #AA7700;">$ProcessThd.TotalProcessorTime.ticks</code>)*100),1) }}
<code style="color: #AA7700;">$ProcessThd</code> | <code style="color: #b43d3d;">Select</code> -Property <code style="color: #AA7700;">$ProcName</code>, <code style="color: #AA7700;">$ProcID</code>, <code style="color: #AA7700;">$ThdID</code>, StartTime, <code style="color: #AA7700;">$CPUTime</code>, <code style="color: #AA7700;">$UsrCPUTime</code>, <code style="color: #AA7700;">$PrivCPU</code>,
<code style="color: #AA7700;">$State</code>, <code style="color: #AA7700;">$WR</code>, <code style="color: #AA7700;">$PrioLvL</code>, <code style="color: #AA7700;">$PrioDelta</code>, <code style="color: #AA7700;">$IdProc</code>, <code style="color: #AA7700;">$ProcAf</code> |
%{<code style="color: #1dc116">'{0},{1},{2},{3},{4},{5},{6},{7},{8},{9},{10},{11}'</code> -f <code style="color: #AA7700;">$_.PID</code>,<code style="color: #AA7700;">$_.Name</code>, <code style="color: #AA7700;">$_.ThreadID</code>,<code style="color: #AA7700;">$_."CPU Time (Sec)"</code>,
<code style="color: #AA7700;">$_."User CPU Time (%)"</code>,<code style="color: #AA7700;">$_."Privil CPU"</code>,<code style="color: #AA7700;">$_.State</code>,<code style="color: #AA7700;">$_.WaitR</code>,<code style="color: #AA7700;">$_."Prio level"</code>,<code style="color: #AA7700;">$_.PrioSh</code>,<code style="color: #AA7700;">$_."Ideal proc"</code>,
<code style="color: #AA7700;">$_."Proc affinity"</code>} | <code style="color: #b43d3d;">Out-File</code> E:\test\thds.csv -Append
}
}</pre></div>
This leaves me with neat little CSV file which I then import to Excel and group by Process ID for further analysis.<br />
<br />
<br />
Back to main script, region <b>Main-start</b>, where I wait for Processes job to start producing data before proceeding. If there is no data generated, the script will stop the jobs and exit.<br />
Next is the neat trick to reduce the flicker while clearing up the screen:<br />
<code>[System.Console]::Clear()</code><br />
and positioning the cursor at top left corner:<br />
<code>$saveYH = [console]::CursorTop<br />
$saveXH = [console]::CursorLeft</code><br />
Worth noting here, in terms of reduced flicker, is hiding the cursor itself:<br />
<code>[console]::CursorVisible = $false</code><br />
<br />
After that, you enter region <b>Main-loop</b> which is the main code for the script. If there is fresh header data to be displayed, I move cursor to (0,0) and write it out. Otherwise, I skip this and check if I should display Core/Socket data. The problem here is that user can specify any number of cores/sockets to display data for and I display two of them in each line. Thus I need an array where user input is mapped to absolute index of the requested piece of HW in perf counter. The array is created in key-press handler. For the sake of performance, both core and socket counters were initialized at the start of the script:<br />
<code>#Just the individual CPUs.<br />
$CPUdata = Get-CimInstance Win32_PerfFormattedData_Counters_ProcessorInformation | Where {$_.Name -match "^(\d{1}),(\d{1})"}<br />
#Just the individual Sockets.<br />
$Socketdata = Get-CimInstance Win32_PerfFormattedData_Counters_ProcessorInformation | Where {$_.Name -match "^(\d{1}),_Total"}</code><br />
Then, if there is fresh data provided by Top_Processes_Job, I display it.<br />
<br />
Next comes the keyboard handling routine. First, check that there is something to handle:<br />
<code> if ($Host.UI.RawUI.KeyAvailable) {</code><br />
If there is, put it into variable:<br />
<code> $k = $Host.UI.RawUI.ReadKey("AllowCtrlC,IncludeKeyDown,IncludeKeyUp,NoEcho").Character</code><br />
Once the keypress is processed, clear the input buffer:</br>
<pre><code> if ("p" -eq $k) {
'CPUpt' > $conf
$HOST.UI.RawUI.Flushinputbuffer()</code></pre><br />
"+" and "1" keys process input of CPUs/Sockets to display data for, while "-" key stops displaying that data.<br />
Pressing "c" key will clear the screen in case it becomes garbled.<br />
Pressing "q" key moves you to region <b>Cleanup</b> ending the script run.<br />
<br />
<br />
<h3>TIPS & TRICKS</h3>
As opposed to Windows TaskManager, I show background processes too (ie. "services").<br />
<br />
In an effort to achieve smoother display of data, I am truncating CPU/Socket info to their integer values. Also, I do not use Thread counters but rather Process ones. Due to delay while displaying the data, there will always be some discrepancy between data displayed. I.e. Total CPU utilization in Header will rarely match sum of CPU utilization by processes in table. I can live with that.<br />
<br />
Script is started in non-normalized CPU utilization mode which means CPU utilization per process can go well over 100% on modern boxes. Let's say you have Quad core box (8 CPUs) and a process taking 50% of Core0, 60% of Core1, 30% of Core2 and 20% of Core3 then the non-normalized CPU utilization for such process would be 160% while normalized CPU utilization would be 20% (160/8). I did it as such to confirm that process actually uses more than one CPU. To toggle between non-normalized and normalized view, use "p" key.<br />
<br />
If, for any reason, display becomes garbled, press the "c" key.<br />
<br />
Number of processes to display is controlled by $procToDisp variable which is, atm, hard-coded to 25.<br />
<br />
Initial sort order is defined by $procSortBy variable. Default is CPU% ($procSortBy = 'CPUpt').<br />
<br />
IF by any chance script does not terminate normally:<br />
- First type Get-Job<br />
- Check that Name has "Top_Header_Job" & "Top_Processes_Job". Remember the Id (or use Name parameter).<br />
Say Id's are 14 and 16.<br />
- Type commands (text after # is just a comment):<br />
<code>[console]::CursorVisible = $true #reclaims the cursor<br />
[console]::TreatControlCAsInput = $false #reverts CTRL+C processing to default value<br />
receive-job -id 14<br />
receive-job -id 16<br />
stop-job -id 16<br />
stop-job -id 14<br />
remove-job -id 14<br />
remove-job -id 16</code><br />
or just exit the Powershell window.<br />
<br />
<br />
Hope you'll find this script useful in your work!<br />
<br />
<br />
This is all from me for this series. Next, I will start new series of blogs describing script used as testing/benchmarking framework on Windows which is also available in the package.<br />
<br />Tončihttp://www.blogger.com/profile/10650361408804728027noreply@blogger.com0tag:blogger.com,1999:blog-7850846940198540592.post-54864408096127912172015-12-14T10:14:00.003+01:002015-12-14T10:14:28.628+01:00Windows PerfCounters and Powershell - Network and Contention perf dataIn <a href="https://www.blogger.com/windows-perf-counters-blog6">previous</a> blog, I covered DISK/IO counters. This blog will briefly touch on Network, Threading and Contention.<br />
<br />
<h2>
Other counters:</h2>
<br />
<h3>
Network I/O</h3>
<b>COUNTER</b>: <u>Network Interface\Bytes Total/sec</u><br />
<b>TYPE</b>: Instantaneous<br />
<b>USAGE</b>:<br />
<div style='overflow-x:auto;overflow-y:hidden;white-space: nowrap;'>
<pre 100="" 18px="" 1em="" 1px="" auto="" consolas="" courier="" font-size:="" font-style:="" font-weight:="" itstream="" line-height:="" mono="" monospace="" new="" normal="" padding:="" sans="" style="background-color: #0f192a; border: 1px dashed #999999; color: #cfecec;" vera="" width:="">
<code style="color: #008200;">#Get Instances</code>
PS > (<code style="color: #b43d3d;">New-Object</code> <code style="color: deeppink;">Diagnostics.PerformanceCounterCategory</code>(<code style="color: #1dc116;">"Network Interface"</code>)).<code style="color: deeppink;">GetInstanceNames</code>(<code style="color: #1dc116;">""</code>)
Intel[R] Centrino[R] Advanced-N 6205
Microsoft Virtual WiFi Miniport Adapter _2
Microsoft Virtual WiFi Miniport Adapter
Intel[R] 82579LM Gigabit Network Connection
PS > <code style="color: #b43d3d;">New-Object</code> <code style="color: deeppink;">Diagnostics.PerformanceCounter</code>(<code style="color: #1dc116;">"Network Interface",
"Bytes Total/sec", "Intel[R] 82579LM Gigabit Network Connection"</code>)
CategoryName : Network Interface
CounterHelp : Bytes Total/sec is the rate at which bytes are sent and received over each network adapter,
including framing characters. Network Interface\Bytes Total/sec is a sum of Network Interface\Bytes Received/sec
and Network Interface\Bytes Sent/sec.
CounterName : Bytes Total/sec
CounterType : RateOfCountsPerSecond64
InstanceLifetime : Global
InstanceName : Intel[R] 82579LM Gigabit Network Connection
ReadOnly : True
MachineName : .
RawValue : 0
Site :
Container :
PS > (<code style="color: #b43d3d;">New-Object</code> <code style="color: deeppink;">Diagnostics.PerformanceCounter</code>(<code style="color: #1dc116;">"Network Interface",
"Bytes Total/sec", "Intel[R] 82579LM Gigabit Network Connection"</code>)).<code style="color: deeppink;">NextValue</code>(<code style="color: #1dc116;">""</code>)
0</pre></div><br />
<b>MEANING</b>:This counter indicates the rate at which bytes are sent and received over each network adapter. It helps you know whether the traffic at your network adapter is saturated and if you need to add another network adapter. How quickly you can identify a problem depends on the type of network you have as well as whether you share bandwidth with other applications.<br />
<b>THRESHOLD</b>:Sustained values of more than 80 percent of network bandwidth.<br />
<br />
<b>COUNTER</b>: <u>Network Interface\Bytes Received/sec</u><br />
<b>TYPE</b>: Instantaneous<br />
<b>USAGE</b>: See above.<br />
<b>MEANING</b>:This counter indicates the rate at which bytes are received over each network adapter. You can calculate the rate of incoming data as a part of total bandwidth. This will help you know that you need to optimize on the incoming data from the client or that you need to add another network adapter to handle the incoming traffic.<br />
<b>THRESHOLD</b>:No specific value.<br />
<br />
<b>COUNTER</b>: <u>Network Interface\Bytes Sent/sec</u><br />
<b>TYPE</b>: Instantaneous<br />
<b>USAGE</b>: See above.<br />
<b>MEANING</b>:This counter indicates the rate at which bytes are sent over each network adapter. You can calculate the rate of incoming data as a part of total bandwidth. This will help you know that you need to optimize on the data being sent to the client or you need to add another network adapter to handle the outbound traffic.<br />
<b>THRESHOLD</b>:No specific value.<br />
<br />
<br />
<h3>
Threading and Contention</h3>
<b>COUNTER</b>: <u>.NET CLR LocksAndThreads\Contention Rate / sec</u><br />
<b>TYPE</b>: Instantaneous<br />
<b>USAGE</b>:<br />
<div style='overflow-x:auto;overflow-y:hidden;white-space: nowrap;'>
<pre 100="" 18px="" 1em="" 1px="" auto="" consolas="" courier="" font-size:="" font-style:="" font-weight:="" itstream="" line-height:="" mono="" monospace="" new="" normal="" padding:="" sans="" style="background-color: #0f192a; border: 1px dashed #999999; color: #cfecec;" vera="" width:="">
<code style="color: #008200;">#Get Instances</code>
PS > (<code style="color: #b43d3d;">New-Object</code> <code style="color: deeppink;">Diagnostics.PerformanceCounterCategory</code>(<code style="color: #1dc116;">".NET CLR LocksAndThreads"</code>)).<code style="color: deeppink;">GetInstanceNames</code>(<code style="color: #1dc116;">""</code>)
_Global_
powershell_ise
PresentationFontCache
dataserv
pcee4
PS > (<code style="color: #b43d3d;">New-Object</code> <code style="color: deeppink;">Diagnostics.PerformanceCounter</code>(<code style="color: #1dc116;">".NET CLR LocksAndThreads",
"Contention Rate / sec", "_Global_"</code>)).<code style="color: deeppink;">NextSample</code>(<code style="color: #1dc116;">""</code>)
RawValue : 310
BaseValue : 0
SystemFrequency : 2533369
CounterFrequency : 0
CounterTimeStamp : 0
TimeStamp : 47774751876
TimeStamp100nSec : 130927572465737721
CounterType : RateOfCountsPerSecond32
PS > (<code style="color: #b43d3d;">New-Object</code> <code style="color: deeppink;">Diagnostics.PerformanceCounter</code>(<code style="color: #1dc116;">".NET CLR LocksAndThreads",
"Contention Rate / sec", "_Global_"</code>)).<code style="color: deeppink;">NextValue</code>(<code style="color: #1dc116;">""</code>)
0
PS > (<code style="color: #b43d3d;">New-Object</code> <code style="color: deeppink;">Diagnostics.PerformanceCounter</code>(<code style="color: #1dc116;">".NET CLR LocksAndThreads",
"Contention Rate / sec", "powershell_ise"</code>)).<code style="color: deeppink;">NextValue</code>(<code style="color: #1dc116;">""</code>)
0</pre></div><br />
<b>MEANING</b>:This counter displays the rate at which the runtime attempts to acquire a <b>managed</b> lock but without a success. Sustained non-zero values may be a cause of concern. You may want to run dedicated tests for a particular piece of code to identify the contention rate for the particular code path.<br />
<b>THRESHOLD</b>:No specific value.<br />
<br />
<b>COUNTER</b>: <u>.NET CLR LocksAndThreads\Current Queue Length</u><br />
<b>TYPE</b>: Instantaneous<br />
<b>USAGE</b>: See above.<br />
<b>MEANING</b>:This counter displays the last recorded number of threads currently waiting to acquire a *managed* lock in an application. You may want to run dedicated tests for a particular piece of code to identify the average queue length for the particular code path. This helps you identify inefficient synchronization mechanisms.<br />
<b>THRESHOLD</b>:No specific value.<br />
<br />
<b>COUNTER</b>: <u>Thread\% Processor Time</u><br />
<b>TYPE</b>: Instantaneous<br />
<b>USAGE</b>:<br />
<div style='overflow-x:auto;overflow-y:hidden;white-space: nowrap;'>
<pre 100="" 18px="" 1em="" 1px="" auto="" consolas="" courier="" font-size:="" font-style:="" font-weight:="" itstream="" line-height:="" mono="" monospace="" new="" normal="" padding:="" sans="" style="background-color: #0f192a; border: 1px dashed #999999; color: #cfecec;" vera="" width:="">PS > <code style="color: #b43d3d;">Get-CimInstance</code> win32_perfformatteddata_perfproc_thread | <code style="color: #b43d3d;">Select</code> IDProcess, PercentProcessorTime |
<code style="color: #b43d3d;">Sort</code> PercentProcessorTime -Descending | <code style="color: #b43d3d;">Group</code> -Property PercentProcessorTime |
<code style="color: #b43d3d;">Select</code> -ExpandProperty Group | <code style="color: #b43d3d;">Select</code> -First 5
IDProcess PercentProcessorTime
--------- --------------------
0 100
3820 96
0 84
0 84
0 46
</pre></div>
<b>MEANING</b>:This counter gives you the idea as to which thread is actually taking the maximum processor time. If you see idle CPU and low throughput, threads could be waiting or deadlocked. You can take a stack dump of the process and compare the thread IDs from test data with the dump information to identify threads that are waiting or blocked. Or examine Thread State and Thread Wait Reason counters.<br />
<b>THRESHOLD</b>:No specific value.<br />
<br />
<br />
<div style='overflow-x:auto;overflow-y:hidden;white-space: nowrap;'>
<pre 100="" 18px="" 1em="" 1px="" auto="" consolas="" courier="" font-size:="" font-style:="" font-weight:="" itstream="" line-height:="" mono="" monospace="" new="" normal="" padding:="" sans="" style="background-color: #0f192a; border: 1px dashed #999999; color: #cfecec;" vera="" width:="">PS > <code style="color: #b43d3d;">Get-CimInstance</code> win32_perfformatteddata_perfproc_thread | <code style="color: #b43d3d;">Select</code> -First 1 | <code style="color: #b43d3d;">FL</code> *
Caption :
Description :
Name : Idle/0
Frequency_Object :
Frequency_PerfTime :
Frequency_Sys100NS :
Timestamp_Object :
Timestamp_PerfTime :
Timestamp_Sys100NS :
ContextSwitchesPersec : 0
ElapsedTime : 13092759167
IDProcess : 0
IDThread : 0
PercentPrivilegedTime : 0
PercentProcessorTime : 0
PercentUserTime : 0
PriorityBase : 0
PriorityCurrent : 0
StartAddress : 59492592
ThreadState : 2
ThreadWaitReason : 0
PSComputerName :
CimClass : root/cimv2:Win32_PerfFormattedData_PerfProc_Thread
CimInstanceProperties : {Caption, Description, Name, Frequency_Object...}
CimSystemProperties : Microsoft.Management.Infrastructure.CimSystemProperties
PS > (<code style="color: #b43d3d;">New-Object</code> <code style="color: deeppink;">Diagnostics.PerformanceCounterCategory</code>(<code style="color: #1dc116;">"Thread"</code>)).<code style="color: deeppink;">GetCounters</code>(<code style="color: #1dc116;">""</code>) |
<code style="color: #b43d3d;">Select</code> CounterName | <code style="color: #b43d3d;">Sort</code> CounterName
CounterName
-----------
% Privileged Time
% Processor Time
% User Time
Context Switches/sec
Elapsed Time
ID Process
ID Thread
Priority Base
Priority Current
Start Address
ThreadState
Thread Wait Reason
PS > (<code style="color: #b43d3d;">New-Object</code> <code style="color: deeppink;">Diagnostics.PerformanceCounterCategory</code>(<code style="color: #1dc116;">".NET CLR LocksAndThreads"</code>)).<code style="color: deeppink;">GetCounters</code>(<code style="color: #1dc116;">""</code>) |
<code style="color: #b43d3d;">Select</code> CounterName | <code style="color: #b43d3d;">Sort</code> CounterName
CounterName
-----------
# of current logical Threads
# of current physical Threads
# of current recognized threads
# of total recognized threads
Contention Rate / sec
Current Queue Length
Queue Length / sec
Queue Length Peak
rate of recognized threads / sec
Total # of Contentions
</pre>
</div>
<br />
<br />
Next blog will be the last in the Windows PerfCounters series where I will put all of this to work writing Top script for Windows.<br />
<br />
In this series:<br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog1.html">BLOG 1: PerfCounters infrastructure</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog2.html">BLOG 2: PerfCounters Raw vs. Formatted values</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog3.html">BLOG 3: PerfCounters, fetching the values </a><br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog4.html">BLOG 4: PerfCounters, CPU perf data</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog5.html">BLOG 5: PerfCounters, Memory perf data</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/12/windows-perf-counters-blog6.html">BLOG 6: PerfCounters, Disk/IO perf data</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/12/windows-perf-counters-blog7.html">BLOG 7: PerfCounters, Network and Contention perf data</a><br />Tončihttp://www.blogger.com/profile/10650361408804728027noreply@blogger.com0tag:blogger.com,1999:blog-7850846940198540592.post-46871715546004475682015-12-07T10:19:00.002+01:002015-12-14T10:16:45.213+01:00Windows PerfCounters and Powershell - Disk & IO perf dataThis post is the hardest for me to write as I generally pay little attention to disks. When they prove too slow, I replace them with faster ones. So now I am writing this on laptop with two SSDs. That said, Disk subsystem could be a major system performance bottleneck and thus there are numerous counters covering this area (<code>Get-CimClass *disk* | Select CimClassName</code>). I would also like to turn your attention to old yet excellent article <a href="http://archive.oreilly.com/pub/a/network/2002/01/18/diskperf.html">Top Six FAQs on Windows 2000 Disk Performance</a> if you're interested in subject.<br />
<br />
<h3>Disk counters:</h3>
Note: Microsoft recommends that "when attempting to analyse disk performance bottlenecks, you should always use physical disk counters. However, if you use software RAID, you should use logical disk counters. As for Logical Disk and Physical Disk Counters, the same values are available in each of these counter objects. Logical disk data is tracked by the volume manager(s), and physical disk data is tracked by the partition manager."<br />
<br />
The one I look into the most is <a href="https://msdn.microsoft.com/en-us/library/aa394262%28v=vs.85%29.aspx">Disk Queue Length</a> which comes in two flavours; Average and Current.<br />
<b>COUNTER</b>: <u>Win32_PerfFormattedData_PerfDisk_PhysicalDisk\AvgDiskQueueLength</u> (<u>AvgDiskReadQueueLength</u>)<br />
<b>TYPE</b>: Sample, Instance<br />
<div style='overflow-x:auto;overflow-y:hidden;white-space: nowrap;'>
<b>USAGE</b>:<pre 100="" 18px="" 1em="" 1px="" auto="" consolas="" courier="" font-size:="" font-style:="" font-weight:="" itstream="" line-height:="" mono="" monospace="" new="" normal="" ourier="" overflow:="" padding:="" sans="" style="background-color: #0f192a; border: 1px dashed #999999; color: #cfecec;" vera="" width:="">
PS > <code style="color: #b43d3d;">Get-CimInstance</code> Win32_PerfFormattedData_PerfDisk_PhysicalDisk | <code style="color: #b43d3d;">Where</code> {<code style="color: #aa7700;">$_</code>.Name <code style="color: #ff1493;">-eq</code> <code style="color: #1dc116;">'_Total'</code>} |
<code style="color: #b43d3d;">Select</code> AvgDiskQueueLength, CurrentDiskQueueLength | <code style="color: #b43d3d;">FL</code>
AvgDiskQueueLength : 0
CurrentDiskQueueLength : 0</pre></div>
<b>MEANING</b>: Average number of both read and write requests that were queued and waiting for the selected disk during the sample interval as well as requests in service. Since I used "_Total" instance, this means I need to divide the value with number of physical disks on the system. PerfMon shows this value per logical disk.
<div style='overflow-x:auto;overflow-y:hidden;white-space: nowrap;'>
<pre 100="" 18px="" 1em="" 1px="" auto="" consolas="" courier="" font-size:="" font-style:="" font-weight:="" itstream="" line-height:="" mono="" monospace="" new="" normal="" ourier="" overflow:="" padding:="" sans="" style="background-color: #0f192a; border: 1px dashed #999999; color: #cfecec;" vera="" width:="">
PS > <code style="color: #b43d3d;">Get-CimInstance</code> Win32_PerfFormattedData_PerfDisk_LogicalDisk |
<code style="color: #b43d3d;">Select</code> Name, AvgDiskQueueLength, CurrentDiskQueueLength | <code style="color: #b43d3d;">FL</code>
Name : HarddiskVolume1 #Boot image on Physical disk 1
AvgDiskQueueLength : 0
CurrentDiskQueueLength : 0
Name : C: #Boot partition on Physical disk 1
AvgDiskQueueLength : 0
CurrentDiskQueueLength : 0
Name : D: #Partition on Physical disk 1
AvgDiskQueueLength : 0
CurrentDiskQueueLength : 0
Name : E: #Partition on Physical disk 2
AvgDiskQueueLength : 0
CurrentDiskQueueLength : 0
Name : G: #Partition on Physical disk 2
AvgDiskQueueLength : 0
CurrentDiskQueueLength : 0
Name : _Total
AvgDiskQueueLength : 0
CurrentDiskQueueLength : 0 </pre></div>
<b>GOTCHA</b>: Since both "pending" and "in service" requests are counted, this counter might overstate the activity.<br />
<b>THRESHOLD</b>: If more than 2 requests are continuously waiting on a single disk, the disk might be a bottleneck. To analyse queue length data further, use it's components; AvgDiskReadQueueLength and AvgDiskWriteQueueLength.<br />
<br />
<b>COUNTER</b>: <u>Win32_PerfFormattedData_PerfDisk_PhysicalDisk\CurrentDiskQueueLength</u><br />
<b>TYPE</b>: Instantaneous, Instance<br />
<div style='overflow-x:auto;overflow-y:hidden;white-space: nowrap;'>
<b>USAGE</b>:<pre 100="" 18px="" 1em="" 1px="" auto="" consolas="" courier="" font-size:="" font-style:="" font-weight:="" itstream="" line-height:="" mono="" monospace="" new="" normal="" ourier="" overflow:="" padding:="" sans="" style="background-color: #0f192a; border: 1px dashed #999999; color: #cfecec;" vera="" width:="">
PS > <code style="color: #b43d3d;">Get-CimInstance</code> Win32_PerfFormattedData_PerfDisk_PhysicalDisk | <code style="color: #b43d3d;">Where</code> {<code style="color: #aa7700;">$_</code>.Name <code style="color: #ff1493;">-eq</code> <code style="color: #1dc116;">'_Total'</code>} |
<code style="color: #b43d3d;">Select</code> AvgDiskQueueLength, CurrentDiskQueueLength | <code style="color: #b43d3d;">FL</code>
AvgDiskQueueLength : 0
CurrentDiskQueueLength : 0</pre></div>
<b>MEANING</b>: Number of requests outstanding on the disk at the time the performance data is collected. It includes requests being serviced at the time of data collection. The value represents an instantaneous length, not an average over a time interval. Multispindle disk devices can have multiple requests active at one time, but other concurrent requests await service. This property may reflect a transitory high or low queue length. If the disk drive has a sustained load, the value will be consistently high. Requests experience delays proportional to the length of the queue minus the number of spindles on the disks. This difference should average less than two for good performance.<br />
<b>GOTCHA</b>: <br />
<b>THRESHOLD</b>: 2 requests in queue for prolonged period of time for single disk (spindle).<br />
<br />
<h3>Inner workings of measurement collection:</h3>
Values are mostly derived by the <b>diskperf</b> filter driver that provides disk performance statistics. Diskperf is a layer of software sitting in the disk driver stack. As I/O Request packets (IRPs) pass through this layer, diskperf keeps track of the time I/O's start and the time they finish. On the way to the device, diskperf records a timestamp for the IRP. On the way back from the device, the completion time is recorded. The difference is the duration of the I/O request. Averaged over the collection interval, this becomes the Avg. Disk sec/Transfer, a direct measure of disk response time from the point of view of the device driver. Diskperf also maintains byte counts and separate counters for reads and writes, at both the Logical and Physical disk level allowing Avg. Disk sec/Transfer to be broken out into reads and writes. This layer does add to latency but not significantly (up to 5%).
Now that we know the mechanics, back to PhysicalDisk\Avg. Disk Queue Length and why we gather both queued and in-service requests in a bunch.<br />
So, AvgDiskQueueLength counter is useful for gathering concurrency data, including data bursts and peak loads. These values represent the number of requests in flight <b>below</b> the driver taking the statistics. This means the requests are not necessarily queued but could actually be in service or completed and on the way back up the path. Possible in-flight locations include the following:<br />
<ul>
<li>SCSIport or Storport queue</li>
<li>OEM driver queue</li>
<li>Disk controller queue</li>
<li>Hard disk queue</li>
<li>Actively receiving from a hard disk</li>
</ul><br />
<h3>Brief account of some other counters:</h3>
<b>COUNTER</b>: <u>PhysicalDisk\Disk Writes/sec</u><br />
<b>MEANING</b>: This counter indicates the rate of write operations on the disk.<br />
<b>THRESHOLD</b>: Depends on manufacturer’s specifications.<br />
<br />
<b>COUNTER</b>: <u>PhysicalDisk\Split IO/sec</u><br />
<b>MEANING</b>: Reports the rate at which the operating system divides I/O requests to the disk into multiple requests. A split I/O request might occur if the program requests data in a size that is too large to fit into a single request or if the disk is fragmented. Factors that influence the size of an I/O request can include application design, the file system, or drivers. A high rate of split I/O might not, in itself, represent a problem. However, on single-disk systems, a high rate for this counter tends to indicate disk fragmentation.<br />
More info in <a href="https://msdn.microsoft.com/en-us/library/ms804035.aspx">MSDN</a>.<br />
<br />
<h3>Disk and Memory:</h3>
Because memory is cached to disk as physical memory becomes limited, make sure that you have a sufficient amount of memory available. When memory is scarce, more pages are written to disk, resulting in increased disk activity. Also, make sure to set the paging file to an appropriate size. Additional disk memory cache will help offset peaks in disk I/O requests. However, it should be noted that a large disk memory cache seldom solves the problem of not having enough spindles, and having enough spindles can negate the need for a large disk memory cache.<br />
<br />
<b>COUNTER</b>: <u>PhysicalDisk\Avg. Disk sec/Transfer</u><br />
<b>MEANING</b>: This counter indicates the time, in seconds, of the average disk transfer. This may indicate a large amount of disk fragmentation, slow disks, or disk failures.<br />
<b>GOTCHA</b>: Multiply the values of the Physical Disk\Avg. Disk sec/Transfer and Memory\Pages/sec counters. If the product of these counters exceeds 0.1, paging is taking more than 10% of disk access time, so you need more physical memory available.<br />
<b>THRESHOLD</b>: Should not be more than 18 milliseconds.<br />
<br />
<b>COUNTER</b>: <u>Memory\Pages/sec</u><br />
<b>MEANING</b>: This counter indicates the rate at which pages are read from or written to disk to resolve hard page faults. Multiply the values of the Physical Disk\Avg. Disk sec/Transfer and Memory\Pages/sec performance counters. If the product of these values exceeds 0.1, paging is utilizing more than 10 percent of disk access time, which indicates that insufficient physical memory is available.<br />
<b>GOTCHA</b>: A high value for the performance counter could indicate excessive paging which will increase disk I/0. If this occurs, consider adding physical memory to reduce disk I/O and increase performance.<br />
<b>THRESHOLD</b>: A sustained value of more than 5 indicates a bottleneck.<br />
<br />
<br />
Next I will talk briefly of other counter categories such as Network and Processes.<br />
<br />
In this series:<br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog1.html">BLOG 1: PerfCounters infrastructure</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog2.html">BLOG 2: PerfCounters Raw vs. Formatted values</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog3.html">BLOG 3: PerfCounters, fetching the values </a><br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog4.html">BLOG 4: PerfCounters, CPU perf data</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog5.html">BLOG 5: PerfCounters, Memory perf data</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/12/windows-perf-counters-blog6.html">BLOG 6: PerfCounters, Disk/IO perf data</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/12/windows-perf-counters-blog7.html">BLOG 7: PerfCounters, Network and Contention perf data</a><br />Tončihttp://www.blogger.com/profile/10650361408804728027noreply@blogger.com0tag:blogger.com,1999:blog-7850846940198540592.post-20918342742288181532015-11-30T09:18:00.000+01:002015-12-14T10:16:24.660+01:00Windows PerfCounters and Powershell - Memory perf data
In the <a href="http://toncigrgin.blogspot.com/2015/11/windows-perf-counters-blog4.html">last blog</a> I spoke of <b>CPU</b> counters. Now, I'll talk of <b>Memory</b> counters.<br />
<br />
<h3>
MEMORY Counters (<a href="https://msdn.microsoft.com/en-us/library/aa387968(v=vs.85).aspx">CIM_PhysicalMemory class</a>, <a href="https://msdn.microsoft.com/en-us/library/aa394268%28v=vs.85%29.aspx">Win32_PerfFormattedData_PerfOS_Memory class</a>, <a href="https://msdn.microsoft.com/en-us/library/windows/desktop/aa965225%28v=vs.85%29.aspx">Memory Performance Information</a> ...):</h3>
Note: I introduced the notion of samples and how to fetch them using NextValue() so I will occasionally omit $var.NextValue() going forward.<br />
<br />
Let me note here that if you thought previously described performance classes were complicated, you are now entering the realm of black magic ;-) There is a good series of <a href="http://blogs.technet.com/b/markrussinovich/archive/2008/07/21/3092070.aspx">blogs</a> on subject of Memory by Mark Russinovich worth reading although quite old.<br />
<br />
Memory is a key resource for any machine so I will look at the most of the values available on Windows. In Resource monitor, Memory tab, you find a bar with Hardware reserved, In use, Modified, Standby and Free values. There are also Available, Cached, Total and Installed values. Let's start with the biggest number, Installed RAM.<br />
<br />
<h3>
In-depth description of Memory Counters important for my use-case:</h3>
<b>COUNTER</b>: <u>cim_physicalmemory\Capacity</u><br />
<b>TYPE</b>: Instantaneous<br />
<b>USAGE</b>: <code>(Get-Ciminstance -class "cim_physicalmemory" | Measure-Object Capacity -Sum).Sum / 1024 / 1024 #MB</code><br />
<b>MEANING</b>: Total capacity of the physical memory, in bytes. Refers to "<b>Installed</b>".<br />
<b>GOTCHA</b>: You will find tips to use TotalPhysicalMemory but, according to <a href=" https://msdn.microsoft.com/en-us/library/aa394181%28v=vs.85%29.aspx">MSDN</a>, it's been deprecated. Also, that page recommends using TotalVisualMemorySize property in the CIM_OperatingSystem class instead but this is wrong as there is no TotalVisualMemorySize property and, even if there was, we need installed memory size.<br />
<b>THRESHOLD</b>:<br />
<br />
Intermediate step; how much of the installed memory is <a href="https://msdn.microsoft.com/en-us/library/aa394239%28v=vs.85%29.aspx">available to OS</a>:<br />
<b>COUNTER</b>: <u>win32_operatingsystem\TotalVisibleMemorySize</u><br />
<b>TYPE</b>: Instantaneous<br />
<b>USAGE</b>: <code>[math]::Round((Get-CimInstance win32_operatingsystem).TotalVisibleMemorySize / 1024,2)</code><br />
<b>MEANING</b>: Total amount of RAM available to OS. Refers to "<b>Total</b>".<br />
<b>GOTCHA</b>: <br />
<b>THRESHOLD</b>:<br />
<br />
Subtracting TotalVisibleMemorySize from Capacity gives us HW reserved RAM, i.e. RAM taken by various HW such as video card. Check <a href="http://blogs.technet.com/b/askperf/archive/2013/02/08/incorrect-amount-of-ram-listed.aspx">this post</a> for details.<br />
<b>COUNTER</b>: <u>HW reserved</u><br />
<b>TYPE</b>: Calculated<br />
<b>USAGE</b>: cim_physicalmemory\Capacity (Installed) - win32_operatingsystem\TotalVisibleMemorySize (Total)<br />
<b>MEANING</b>: Size of RAM not available to OS although installed on the system. Refers to "<b>Hardware reserved</b>".<br />
<b>GOTCHA</b>: Depends on HW and BIOS settings, not something "fixable" in Windows.<br />
<b>THRESHOLD</b>:<br />
<br />
<b>COUNTER</b>: <u>win32_operatingsystem\FreePhysicalMemory (Bytes), Memory\Available MBytes</u><br />
<b>TYPE</b>: Instantaneous<br />
<b>USAGE</b>:<br />
<code>(Get-WmiObject win32_operatingsystem).FreePhysicalMemory<br />
$Memory_AvailMB = New-Object Diagnostics.PerformanceCounter("Memory", "Available MBytes")<br />
(New-Object Diagnostics.PerformanceCounter("Memory", "Available MBytes")).RawValue</code><br />
<b>MEANING</b>: Total amount of RAM available to processes. Equal to the sum of memory assigned to the standby (cached), free and zero page lists. Refers to "<b>Available</b>".<br />
<b>GOTCHA</b>: <br />
<b>THRESHOLD</b>: A consistent value of less than 20% of installed RAM. In such situations, consult additional counters, such as Win32_PerfFormattedData_PerfOS_Memory\PagesPerSec to determine if System memory is adequate for the workload.<br />
<br />
<b>COUNTER</b>: <u>In use memory</u><br />
<b>TYPE</b>: Calculated<br />
<b>USAGE</b>: win32_operatingsystem\TotalVisibleMemorySize (Total) - Memory\Available MBytes (Available)<br />
<b>MEANING</b>: Amount of RAM in use by processes running on the box.<br />
<b>GOTCHA</b>: <br />
<b>THRESHOLD</b>:<br />
<br />
<b>COUNTER</b>: <u>Memory\Modified Page List Bytes (Win32_PerfFormattedData_PerfOS_Memory)</u><br />
<b>TYPE</b>: Instantaneous<br />
<b>USAGE</b>: <code>$Memory_ModPLBy = New-Object System.Diagnostics.PerformanceCounter("Memory", "Modified Page List Bytes")</code><br />
<b>MEANING</b>: The amount of RAM taken by the pages previously belonging to a working set but removed. However, the pages were modified while in use and their current contents haven’t yet been written to storage. The Page Table Entry still refers to the physical page(s) but is marked invalid and in transition. It must be written to the backing store before the physical page can be reused.<br />
<b>GOTCHA</b>: No description in <a href="https://msdn.microsoft.com/en-us/library/aa394268%28v=vs.85%29.aspx">MSDN</a>!?<br />
<b>THRESHOLD</b>: Keep as low as possible.<br />
<br />
<b>COUNTER</b>: <u>Win32_PerfFormattedData_PerfOS_Memory\FreeAndZeroPageListBytes</u><br />
<b>TYPE</b>: Instantaneous<br />
<b>USAGE</b>: <code>(get-wmiobject -computername localhost -Namespace root\CIMV2 -Query "Select * from Win32_PerfFormattedData_PerfOS_Memory").FreeAndZeroPageListBytes / 1024 / 1024 #MB</code><br />
<b>MEANING</b>: The amount of physical memory, in bytes, that is assigned to the free and zero page lists thus immediately available for allocation to a process or for system use since it does not contain any data. Refers to "<b>Free</b>".<br />
<b>GOTCHA</b>: There is a big difference between Free and Available memory. This is due to most of the pages considered available being in some sort of transition state (i.e. waiting to be written to disk) or have not yet met all of the OS requirements (i.e. page is not considered secure until it's zeroed out).<br />
<b>THRESHOLD</b>: Keep as high as possible.<br />
<br />
<b>COUNTER</b>: <u>Standby</u><br />
<b>TYPE</b>: Calculated<br />
<b>USAGE</b>:<br />
<code>$Memory_SBCCBy = New-Object Diagnostics.PerformanceCounter("Memory", "Standby Cache Core Bytes")<br />
$Memory_SBCNPBy = New-Object Diagnostics.PerformanceCounter("Memory", "Standby Cache Normal Priority Bytes")<br />
$Memory_SBCRBy = New-Object Diagnostics.PerformanceCounter("Memory", "Standby Cache Reserve Bytes")<br />
[math]::Round($Memory_SBCCBy.NextValue()/1024/1024 + $Memory_SBCNPBy.NextValue()/1024/1024+$Memory_SBCRBy.NextValue()/1024/1024,2)</code><br />
<b>MEANING</b>: The amount of RAM in pages previously belonging to a working set but removed (or marshaled directly into the standby list). The pages weren’t modified since last written to disk. The Page Table Entry still refers to the physical pages but are marked invalid and in transition. Or, simpler explanation, memory that has been removed from a process's working set (its physical memory) en route to disk but is still available to be recalled.<br />
<b>GOTCHA</b>: Please see the explanation of the factors in <a href="https://msdn.microsoft.com/en-us/library/aa394268%28v=vs.85%29.aspx">Win32_PerfFormattedData_PerfOS_Memory</a> or <a href="https://msdn.microsoft.com/en-us/library/ms804008.aspx">Memory Object</a> MSDN pages.<br />
<b>THRESHOLD</b>:<br />
<br />
<b>COUNTER</b>: <u>Cached</u><br />
<b>TYPE</b>: Calculated<br />
<b>USAGE</b>: <br />
<b>MEANING</b>: This number represents the sum of the system working set, standby list and modified page list. So, Memory\Cache Bytes, Memory\Modified Page List Bytes, Memory\Standby Cache Core Bytes, Memory\Standby Cache Normal Priority Bytes and Memory\Standby Cache Reserve Bytes. In this case, Memory\Cache Bytes + Memory\Modified Page List Bytes + Standby.<br />
<b>GOTCHA</b>: Presented here for the sake of completeness.<br />
<b>THRESHOLD</b>:<br />
<br />
<h3>More counters of significance:</h3>
<b>Win32_PerfFormattedData_PerfOS_Memory\CacheBytes</b> - Number of bytes currently being used by the file system cache. The file system cache is an area of physical memory that stores recently used pages of data for applications. The operating system continually adjusts the size of the cache, making it as large as it can while still preserving the minimum required number of available bytes for processes. This property displays the last observed value only; it is not an average.
See also <b>SystemCacheResidentBytes</b> and relatives.<br />
Simpler explanation would be that the memory pages that the System uses are counted in two main counters, <b>Cache Bytes</b> and <b>Pool Nonpaged Bytes</b>. The <b>Cache Bytes</b> counter value is the amount of resident pages allocated in RAM that the Kernel threads can address without causing a Page Fault. This counter includes the <b>Pool Paged Resident Bytes</b>, the <b>System Cache Resident Bytes</b>, the <b>System Code Resident Bytes</b> and the <b>System Driver Resident Bytes</b>.<br />
<br />
Note: If Memory\Pool Nonpaged Bytes value is 10% or more higher than its value at system startup, there is probably a leak.<br />
<br />
<b>Win32_PerfFormattedData_PerfOS_Memory\CacheFaultsPerSec</b> - Number of faults which occur when a page is not found in the file system cache and must be retrieved from elsewhere in memory (a soft fault) or from disk (a hard fault). The file system cache is an area of physical memory that stores recently used pages of data for applications. Cache activity is a reliable indicator of most application I/O operations. This property counts the number of faults without regard for the number of pages faulted in each operation.<br />
<br />
There is a whole set of Paging counters and they do require our attention since we can deduce Memory shortages on Windows by using them. Some of the key counters I will describe below. Dealing with Windows Paging you have to keep in mind that paging occurs for various operations within OS and excessive paging doesn’t automatically indicate a memory shortage. For instance, many network card drivers utilize the Pagefile (sometimes excessively) and this can be misread as a memory shortage.<br />
<br />
<b>Win32_PerfFormattedData_PerfOS_Memory\PagesPerSec</b> (and relatives) - A sustained value of over 20 should be closely monitored and a System with a sustained value of over 50 is probably lacking in System Memory. Again, it is normal for this value to spike occasionally, especially if the other Memory counters do not show a lack of System Memory.<br />
<br />
<b>COUNTER</b>: <u>Pages Input per second / Page Reads per second</u><br />
<b>TYPE</b>: Calculated<br />
<b>USAGE</b>:<br />
<code>$Memory_PIps = New-Object Diagnostics.PerformanceCounter("Memory", "Pages Input/sec")<br />
$Memory_PRps = New-Object Diagnostics.PerformanceCounter("Memory", "Page Reads/sec")<br />
[math]::Round ($Memory_PIps.NextValue() / $Memory_PRps.NextValue(),2)</code><br />
<b>MEANING</b>: The average of Memory\Pages Input/sec divided by average of Memory\Page Reads/sec gives the number of pages per disk read. This value should not generally exceed five pages per second. A value greater than five indicates that the system is spending too much time paging and requires more memory (assuming that the application has been optimized).<br />
<b>GOTCHA</b>: <br />
<b>THRESHOLD</b>: Sustained value of 5 or more.<br />
<br />
<h3>Some other interesting counters I will not be covering in detail:</h3>
<b>Memory\Page Reads/sec</b><br />
<b>Memory\Page Writes/sec</b><br />
<b>Paging File(_total)\% Usage</b><br />
and so on.<br />
<br />
In the next blog I will cover Disk counters.<br />
<br />
In this series:<br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog1.html">BLOG 1: PerfCounters infrastructure</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog2.html">BLOG 2: PerfCounters Raw vs. Formatted values</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog3.html">BLOG 3: PerfCounters, fetching the values </a><br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog4.html">BLOG 4: PerfCounters, CPU perf data</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog5.html">BLOG 5: PerfCounters, Memory perf data</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/12/windows-perf-counters-blog6.html">BLOG 6: PerfCounters, Disk/IO perf data</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/12/windows-perf-counters-blog7.html">BLOG 7: PerfCounters, Network and Contention perf data</a><br />Tončihttp://www.blogger.com/profile/10650361408804728027noreply@blogger.com0tag:blogger.com,1999:blog-7850846940198540592.post-84518666176264864072015-11-23T09:52:00.001+01:002015-12-14T10:16:05.125+01:00Windows PerfCounters and Powershell - CPU perf dataSo far, I talked of WMI, CIM, WQL, System.Diagnostics.PerformanceCounterCategory, perf-counter data organization and flavour. Now it's time to look at some performance counters I deem important for my use-case more closely.<br />
Note: List of available Counters for Get-Counter command<br />
<code>Get-Counter -ListSet * | Sort-Object CounterSetName | Format-Table CounterSetName</code><br />
<br />
<h3>
Basic concepts:</h3>
I will introduce basic concepts of Processor, Core and CPU now to help you follow the text. Let us use this convention:<br />
<ul>
<li> "Processor" is a piece of hardware you connect to a slot on the motherboard.</li>
<li> "Physical Core" is a physical computing unit built into the "Processor".</li>
<li> "Virtual Core" is a virtual computing unit built on top of "Physical Core" (i.e. HT is ON).</li>
<li> "CPU" is a computing unit inside the "Processor", either physical or virtual.</li>
</ul>
<br />
<br />
<h3>
Putting concepts to work</h3>
Now lets calculate number of CPUs for my laptop:<br />
<samp>PS > ((Get-CimInstance -Namespace root/CIMV2 -ClassName CIM_Processor).NumberOfLogicalProcessors | Measure-Object -Sum).Sum<br /><br />
4</samp><br />
Note: <b>Many</b> other counters fail for some HW configuration and/or OS! Be sure to check.<br />
Note: HT is ON on my dual-core laptop and no cores are parked so to get number of Physical cores:<br />
<samp>PS > ((Get-CimInstance -Namespace root/CIMV2 -ClassName CIM_Processor).NumberOfCores | Measure-Object -Sum).Sum<br /><br />
2</samp><br />
Note: There are many ways to collect this info:<br />
<samp>PS > (Get-CimInstance Win32_ComputerSystem).NumberOfLogicalProcessors</samp><br />
<samp>PS > ((New-Object Diagnostics.PerformanceCounterCategory("Processor Information")).GetInstanceNames() | ?{$_ -match "^(\d{1}),(\d{1})"} | Measure-Object -Sum).Count</samp><br />
Note: RegEx expression is matching "Number,Number" Instances only (See <a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog3.html">previous</a> blog about instances).<br />
<br />
It is not obvious when working with 1 NUMA node/Slot, but the -Sum might refer to Sum of CPUs per Slot, depending on RegEx.<br />
<br />
Before starting on Counters, let me stress that the measurements at the system, process and thread level in Windows are based on a sampling methodology thus the data gathered is subject to typical sampling errors like:<br />
<ul>
<li>accumulating a "sufficient" number of sample observations to be able to make a reliable statistical inference, i.e. the sampling size</li>
</ul>
and<br />
<ul>
<li>ensuring that there aren’t systemic sources of sampling error that causes results to be under or over-sampled as I will demonstrate shortly.</li>
</ul>
<br />
As of W2K8, the trends are changing towards event driven measurement for CPU utilization which, although more sane and accurate, poses its own set of challenges (say, a <a href="https://en.wikipedia.org/wiki/Clock_drift">clock drift</a> across multiprocessor cores when they are not resynchronized periodically and so on). To compensate for drift, new PerfMon/ResMon work by measuring CPU load in real time using event oriented measurement data gathered by the OS Scheduler each time a context switch occurs.<br />
A context switch occurs in Windows whenever the processor switches its execution context to run a different thread (see more below). Context switches also occur as a result of high priority Interrupt Service Routines (ISRs) as well as the Deferred Procedure Calls (DPCs) that ISRs schedule to complete the interrupt processing. Starting in Windows 6 (Vista/2008), the OS Scheduler began issuing RDTSC instructions to get the internal processor clock each time a context switch occurs. I will talk of context switching and DPC counters in a short while. For more details please see <a href="http://blogs.msdn.com/b/ddperf/archive/2010/04/04/measuring-processor-utilization-and-queuing-delays-in-windows-applications.aspx">this</a> excellent blog post.<br />
<br />
<h3>
System CPU counters:</h3>
First counter I want to talk about is <b>Processor Queue Length</b>. Immediately a Linux users observes that there is no "System load" counter on Windows. This is because Windows OS is Thread based as opposed to Linux which is Process based. This simply means that, in Windows, an execution thread is a basic unit of execution (thus basis for collecting usage statistics too) and a process acts as a container for threads. As simple as it may seem, this actually poses a lot of challenges since one has to start aggregating data about running processes from Threads counters and work his way up. I will talk about this in detail in final blog. So, the WMI counter mimicking Linux "System load" best is, IMO, Processor Queue Length:<br />
<div style='overflow-x:auto;overflow-y:hidden;white-space: nowrap;'>
<pre 100="" 18px="" 1em="" 1px="" auto="" consolas="" courier="" font-size:="" font-style:="" font-weight:="" itstream="" line-height:="" mono="" monospace="" new="" normal="" overflow:="" padding:="" sans="" style="background-color: #0f192a; border: 1px dashed #999999; color: #cfecec;" vera="" width:="">PS > <code style="color: #b43d3d;">Get-Counter</code> <code style="color: #1dc116;">'\System\Processor Queue Length'</code>
Timestamp CounterSamples
--------- --------------
23.10.15. 10:34:10 \\server_name\system\processor queue length : 1
</pre></div>
However, this is slooooow (although subsequent calls return much faster):<br />
<div style='overflow-x:auto;overflow-y:hidden;white-space: nowrap;'>
<pre 100="" 18px="" 1em="" 1px="" auto="" consolas="" courier="" font-size:="" font-style:="" font-weight:="" itstream="" line-height:="" mono="" monospace="" new="" normal="" overflow:="" padding:="" sans="" style="background-color: #0f192a; border: 1px dashed #999999; color: #cfecec;" vera="" width:="">PS > <code style="color: #b43d3d;">Measure-Command</code> { <code style="color: #b43d3d;">Get-Counter</code> <code style="color: #1dc116;">'\System\Processor Queue Length'</code> }
TotalSeconds : 4.2961321
PS > <code style="color: #b43d3d;">Measure-Command</code> { <code style="color: #b43d3d;">Get-Counter</code> <code style="color: #1dc116;">'\System\Processor Queue Length'</code> }
TotalSeconds : 1.007445
</pre></div>
So, as described in <a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog3.html">previous</a> blog, I use <samp>System.Diagnostics</samp> class to fetch this value:<br />
<div style='overflow-x:auto;overflow-y:hidden;white-space: nowrap;'>
<pre 100="" 18px="" 1em="" 1px="" auto="" consolas="" courier="" font-size:="" font-style:="" font-weight:="" itstream="" line-height:="" mono="" monospace="" new="" normal="" overflow:="" padding:="" sans="" style="background-color: #0f192a; border: 1px dashed #999999; color: #cfecec;" vera="" width:="">PS > <code style="color: #b43d3d;">Measure-Command</code> { <code style="color: #b43d3d;">New-Object</code> Diagnostics.PerformanceCounter(<code style="color: #1dc116;">"System"</code>, <code style="color: #1dc116;">"Processor Queue Length"</code>)}
TotalSeconds : 2.0006457
PS > <code style="color: #b43d3d;">Measure-Command</code> { <code style="color: #b43d3d;">New-Object</code> Diagnostics.PerformanceCounter(<code style="color: #1dc116;">"System"</code>, <code style="color: #1dc116;">"Processor Queue Length"</code>)}
TotalSeconds : 0.000643</pre></div>
Now, put this into a variable and simply call NextValue():<br />
<div style='overflow-x:auto;overflow-y:hidden;white-space: nowrap;'>
<pre 100="" 18px="" 1em="" 1px="" auto="" consolas="" courier="" font-size:="" font-style:="" font-weight:="" itstream="" line-height:="" mono="" monospace="" new="" normal="" overflow:="" padding:="" sans="" style="background-color: #0f192a; border: 1px dashed #999999; color: #cfecec;" vera="" width:="">PS > <code style="color: #aa7700;">$System_ProcQL</code> = <code style="color: #b43d3d;">New-Object</code> Diagnostics.PerformanceCounter(<code style="color: #1dc116;">"System"</code>, <code style="color: #1dc116;">"Processor Queue Length"</code>)
PS > <code style="color: #aa7700;">$System_ProcQL</code>.<code style="color: deeppink;">NextValue()</code>
0
PS > <code style="color: #aa7700;">$System_ProcQL</code>.<code style="color: deeppink;">NextValue()</code>
10
</pre></div>
The value obtained is for all of the CPU's so you need to calculate the number of CPU's to be your divider and obtain the real value:<br />
<code>$SystemLoad = $System_ProcQL.NextValue() / $totCPU</code><br />
<br />
<br />
<h3>
In-depth description of System Counters important for my use-case:</h3>
<b>COUNTER</b>: <u>System\Processor Queue Length</u><br />
<b>TYPE</b>: Instantaneous <br />
<b>USAGE</b>: <code>New-Object Diagnostics.PerformanceCounter("System", "Processor Queue Length") / ((Get-CimInstance -Namespace root/CIMV2 -ClassName CIM_Processor).NumberOfLogicalProcessors | Measure-Object -Sum).Sum</code><br />
<b>MEANING</b>: Number of threads per CPU that are ready for execution but can't get CPU cycles for whatever reason thus waiting in OS Scheduler queue. Since Windows have one Scheduler queue, I divide this value with total number of computation units (i.e. CPUs). The actual mechanics is that when Counter value is requested a measurement function traverses the Scheduler Ready Queue and counts the number of threads waiting for an available CPU.<br />
<b>GOTCHA</b>: Even on idle system there can be significant number of threads running on schedule that can bump this number very high. Say you have 4 CPU box and processes fetching values for 100 counters, 10 samples every 1 second. All of these sample requests will lay sleeping for 1 second (thus the Processor Queue Length value will be low) and then all will wake up at the same timer event (clock interrupt) causing Processor Queue Length to spike although there is no real load on the system. It's even worse if your thread(s) is of high priority as it will get executed sooner than the user threads thus pushing Processor Queue Length number very very high. This leads to disproportionate number of Ready Threads waiting for cycles, even (or especially) when the processor itself is not very busy overall. So tip 1 would be to check if CPUs are really busy or not.<br />
<b>THRESHOLD</b>: Pending on above, it is hard to tell what the threshold value is but most people seem to agree it's "sustained value of 2 or more" with CPU utilization of 85%+. This combination tells us we can benefit from adding more CPUs.<br />
<br />
<b>COUNTER</b>: <u>System\Context Switches/sec</u><br />
<b>TYPE</b>: Instantaneous<br />
<b>USAGE</b>:<br />
<code>$System_CSpS = New-Object Diagnostics.PerformanceCounter("System", "Context Switches/sec")<br />
$System_CSpS.NextValue()</code><br />
<b>MEANING</b>: Context switching happens when a higher priority thread pre-empts a lower priority thread that is currently running or when a high priority thread blocks. High levels of context switching can occur when many threads share the same priority level. This often indicates that there are too many threads competing for the processors on the system. If you do not see much processor utilization and you see very low levels of context switching, it could indicate that threads are blocked (<a href="https://msdn.microsoft.com/en-us/library/aa394279%28v=vs.85%29.aspx">link</a>).<br />
<b>GOTCHA</b>: The number obtained is system-wide! To report the total number of context switches generated per second by all threads use the Thread(_Total)\Context Switches/sec counter (Category((Instance)\Counter):<br />
<code>New-Object Diagnostics.PerformanceCounter("Thread", "Context Switches/sec", "_Total")</code><br />
<b>THRESHOLD</b>: Context switching rates in excess of 15,000 per second per CPU. The remedy would be to reduce the number of threads and queue more at the application level. This will cause less context switching, and less context switching is good for reducing CPU load.<br />
<br />
<br />
<h3>
In-depth description of <a href="https://www.blogger.com/(https://msdn.microsoft.com/en-us/library/aa394271%28v=vs.85%29.aspx)">CPU Counters</a> important for my use-case:</h3>
Note: "Processor Information" category, besides overall _Total, has instances for Slot/NUMA node (0,_Total, n,_Total) while "Processor" category gives just _Total for all CPUs as defined above.<br />
<b>Gotcha:</b> On single slot machines, "Processor" category will give info for all the CPUs while on machines with multiple slots, it will give info on just the Physical cores :-/<br />
Thus, if InstanceName is _Total, both yield the same value.<br />
<br />
<b>COUNTER</b>: <u>Processor Information(_Total)\% Processor Time</u>, <u>Processor(_Total)\% Processor Time</u><br />
<b>TYPE</b>: Sample, Instance<br />
<b>USAGE</b>:<br />
<code>$InstanceName = "_Total"<br />
$PI_PT = New-Object Diagnostics.PerformanceCounter("Processor Information", "% Processor Time")<br />
$PI_PT.InstanceName = $InstanceName<br />
$null = $PI_PT.NextValue()<br />
--or--<br />
Get-Counter -Counter "\Processor Information(_Total)\% Processor Time"<br />
Get-Counter -Counter "\Processor(_Total)\% Processor Time"</code><br />
<b>MEANING</b>: Primary indicator of CPU activity. High values many not necessarily be bad. However, if the other processor-related counters are increasing linearly such as Processor\% Privileged Time or System\Processor Queue Length, high CPU utilization may be worth investigating.<br />
<b>GOTCHA</b>: If this counter is around threshold value, starting new processes will only lead to increased value of Processor Queue Length but the work done will remain the same. Look for some more counters that I'm about to describe in relation to this one.<br />
<b>THRESHOLD</b>: Folks seem to agree on ~85%. Low CPU utilization with sustained Processor Queue Length value of 2 or higher is indicator that requests for CPU time arrive randomly and threads demand irregular amounts of time from the CPU. This means that the processor power is not a bottleneck but that the application threading logic should be improved.<br />
<br />
<b>COUNTER</b>: <u>Processor Information(_Total)\% Privileged Time</u><br />
<b>TYPE</b>: Sample, Instance<br />
<b>USAGE</b>:<br />
<code>$InstanceName = "_Total"<br />
$PI_PPT = New-Object Diagnostics.PerformanceCounter("Processor Information", "% Privileged Time")<br />
$PI_PPT.InstanceName = $InstanceName<br />
$null = $PI_PPT.NextValue()</code><br />
<b>MEANING</b>: Counter indicates the percentage of non-idle CPU time spent in privileged mode, i.e. calls to OS functions (file or network I/O, memory allocation...). Basically, this is unrestricted mode allowing direct access to hardware and all memory.<br />
<b>GOTCHA</b>:<br />
<b>THRESHOLD</b>: Folks seem to agree on consistently being over 75%.<br />
<br />
<b>COUNTER</b>: <u>Processor Information(_Total)\% User Time</u><br />
<b>TYPE</b>: Sample, Instance<br />
<b>USAGE</b>:<br />
<code>$InstanceName = "_Total"<br />
$PI_PUT = New-Object Diagnostics.PerformanceCounter("Processor Information", "% User Time")<br />
$PI_PUT.InstanceName = $InstanceName<br />
$null = $PI_PUT.NextValue()</code><br />
<b>MEANING</b>: Percentage of non-idle CPU time spent in user mode. User mode is a restricted processing mode designed for applications, environment subsystems, and integral subsystems.<br />
<b>GOTCHA</b>: Processor Information(_Total)\% Privileged Time + <br />Processor Information(_Total)\% User Time = Processor Information(_Total)\% Processor Time.<br />
<b>THRESHOLD</b>: Depends on previous two counters.<br />
<br />
<b>COUNTER</b>: <u>Processor Information(_Total)\% Idle Time</u><br />
<b>TYPE</b>: Sample, Instance<br />
<b>USAGE</b>:<br />
<code>$InstanceName = "_Total"<br />
$PI_PIT = New-Object Diagnostics.PerformanceCounter("Processor Information", "% Idle Time")<br />
$PI_PIT.InstanceName = $InstanceName<br />
$null = $PI_PIT.NextValue()</code><br />
<b>MEANING</b>: Counter indicates the percentage of time OS idle thread was consuming cycles. On Windows, there is a special Kernel thread that consumes cycles when CPU is idling. Counting cycles consumed by this thread gives Idle CPU time.<br />
<b>GOTCHA</b>: Processor Information(_Total)\% Processor Time + Processor Information(_Total)\% Idle Time = 100%<br />
<b>THRESHOLD</b>:<br />
<br />
<b>COUNTER</b>: <u>Processor Information(_Total)\% Priority Time</u><br />
<b>TYPE</b>: Sample, Instance<br />
<b>USAGE</b>:<br />
<code>$InstanceName = "_Total"<br />
$PI_PPRIOT = New-Object Diagnostics.PerformanceCounter("Processor Information", "% Priority Time")<br />
$PI_PPRIOT.InstanceName = $InstanceName<br />
$null = $PI_PPRIOT.NextValue()</code><br />
<b>MEANING</b>: CPU utilization by high priority threads.<br />
<b>GOTCHA</b>: Kernel scheduler can, on occasion, wake up low priority threads sleeping for "long" time assigning them much more slices on CPU than one would expect given the (low)priority. This, in turn, blocks high-priority threads from execution which is never an expected behaviour. I would look at this value in relation to Context switches/second to determine what's going on.<br />
<b>THRESHOLD</b>:<br />
<br />
<b>COUNTER</b>: <u>Processor Information\Interrupts/sec</u><br />
<b>TYPE</b>: Sample, Instance<br />
<b>USAGE</b>:<br />
<code>$InstanceName = "_Total"<br />
$PI_INTPS = New-Object Diagnostics.PerformanceCounter("Processor Information", "Interrupts/sec")<br />
$PI_INTPS.InstanceName = $InstanceName<br />
$null = $PI_INTPS.NextValue()</code><br />
<b>MEANING</b>: Number of hardware interrupts per second. This value is the indicator of the activity of devices that generate interrupts, such as network adapters.<br />
<b>GOTCHA</b>: See next counter.<br />
<b>THRESHOLD</b>:<br />
<br />
<b>COUNTER</b>: <u>Processor Information\% Interrupt Time</u><br />
<b>TYPE</b>: Sample, Instance<br />
<b>USAGE</b>:<br />
<code>$InstanceName = "_Total"<br />
$PI_PINTT = New-Object Diagnostics.PerformanceCounter("Processor Information", "% Interrupt Time")<br />
$PI_PINTT.InstanceName = $InstanceName<br />
$null = $PI_PINTT.NextValue()</code><br />
<b>MEANING</b>: The value indicates the percentage of time CPUs spend receiving and servicing hardware interrupts. This value is an indirect indicator of the activity of devices that generate interrupts, such as network adapters.<br />
<b>GOTCHA</b>: Mass increase in Processor Information\Interrupts/sec and Processor Information\% Interrupt Time indicates potential hardware problems.<br />
<b>THRESHOLD</b>:<br />
<br />
<b>COUNTER</b>: <u>Processor Information\DPCs Queued/sec</u><br />
<b>TYPE</b>: Sample, Instance<br />
<b>USAGE</b>:<br />
<code>$InstanceName = "_Total"<br />
$PI_DPCQPS = New-Object Diagnostics.PerformanceCounter("Processor Information", "DPCs Queued/sec")<br />
$PI_DPCQPS.InstanceName = $InstanceName<br />
$null = $PI_DPCQPS.NextValue()</code><br />
<b>MEANING</b>: Overall rate at which deferred procedure calls ("SW interrupts") are added to the processor's DPC queue. This property measures the rate at which DPCs are added to the queue, not the number of DPCs in the queue.<br />
<b>GOTCHA</b>: This is NOT the number of SW interrupts in the queue!<br />
<b>THRESHOLD</b>:<br />
<br />
<b>COUNTER</b>: <u>Processor Information\DPC Time</u><br />
<b>TYPE</b>: Sample, Instance<br />
<b>USAGE</b>:<br />
<code>$InstanceName = "_Total"<br />
$PI_PDPCT = New-Object Diagnostics.PerformanceCounter("Processor Information", "% DPC Time")<br />
$PI_PDPCT.InstanceName = $InstanceName<br />
$null = $PI_PDPCT.NextValue()</code><br />
<b>MEANING</b>: Percentage of time that the processor spent receiving and servicing deferred procedure calls (SW interrupts) during the sample interval. They are counted separately and are not a component of the interrupt counters.<br />
<b>GOTCHA</b>: This property is a component of PercentPrivilegedTime because DPCs are executed in privileged mode.<br />
<b>THRESHOLD</b>:<br />
<br />
Other useful counters I would look into in case of trouble are <b>C1/C2/C3TransitionsPerSec</b>. There is a huge penalty waking up CPU from C3 low power state to C2 low power state and considerable penalty transitioning from C2 to C1. So if box is choking and CPUs are idling, look here. And make sure ParkingStatus for each CPU is 0 ;-)<br />
Example: (Physical) CPU 9 in Slot 7 was asleep:<br />
<div style='overflow-x:auto;overflow-y:hidden;white-space: nowrap;'>
<pre 100="" 18px="" 1em="" 1px="" auto="" consolas="" courier="" font-size:="" font-style:="" font-weight:="" itstream="" line-height:="" mono="" monospace="" new="" normal="" overflow:="" padding:="" sans="" style="background-color: #0f192a; border: 1px dashed #999999; color: #cfecec;" vera="" width:="">PS ><code style="color: #b43d3d;"> GCim</code> Win32_PerfFormattedData_Counters_ProcessorInformation
...
Name : 7,9
AverageIdleTime : 100
C3TransitionsPersec : 64
ClockInterruptsPersec : 64
IdleBreakEventsPersec : 64
InterruptsPersec : 64
PercentC3Time : 99
...</pre></div>
Basically, only processing timer events.<br />
<br />
There are also combinations of counters that can point out problems like <b>Processor\% DPC Time</b>, <b>% Interrupt Time</b> and <b>% Privileged Time</b>.
If Interrupt Time and DPC time are a large portion of Privileged Time, the kernel is spending significant amount of time processing (most likely) I/O requests. In some cases performance can be improved by configuring interrupts and DPC affinity to a small number of CPUs on a multiprocessor system, which improves cache locality. In other cases, it works best to distribute the interrupts and DPCs among many CPUs, so as to keep the interrupt and DPC activity from becoming a bottleneck.<br />
<br />
In the next blog I will cover Memory performance counters.<br />
<br />
In this series:<br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog1.html">BLOG 1: PerfCounters infrastructure</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog2.html">BLOG 2: PerfCounters Raw vs. Formatted values</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog3.html">BLOG 3: PerfCounters, fetching the values </a><br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog4.html">BLOG 4: PerfCounters, CPU perf data</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog5.html">BLOG 5: PerfCounters, Memory perf data</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/12/windows-perf-counters-blog6.html">BLOG 6: PerfCounters, Disk/IO perf data</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/12/windows-perf-counters-blog7.html">BLOG 7: PerfCounters, Network and Contention perf data</a><br />Tončihttp://www.blogger.com/profile/10650361408804728027noreply@blogger.com0tag:blogger.com,1999:blog-7850846940198540592.post-83785443866053133272015-11-16T09:02:00.001+01:002015-12-14T10:15:46.485+01:00Windows PerfCounters and Powershell - Fetching the values<h4>
Summary from <a href="http://toncigrgin.blogspot.com/2015/11/windows-perf-counters-blog2.html">last blog</a>:</h4>
<ul>
<li><b>Tip:</b> An alias for Get-CimInstance is GCim, for Select-Object it's Select and for Get-WmiObject is GWmi.</li>
<li>There are <b>Raw</b> and <b>Formatted</b> counters. Watch out for formula converting Raw samples to Formatted values.</li>
</ul>
<br />
<br />
<h3>
NAMESPACE organization</h3>
The general organization of namespaces is as follows:<br />
Category (Class if you prefer)<br />
Counter(s)<br />
Instance(s)<br />
Every Category has Counters but not all of the Counters have Instances. The full path to the desired value is called a __PATH:
<br />
<div style='overflow-x:auto;overflow-y:hidden;white-space: nowrap;'>
<pre 100="" 18px="" 1em="" 1px="" auto="" consolas="" courier="" font-size:="" font-style:="" font-weight:="" itstream="" line-height:="" mono="" monospace="" new="" normal="" ourier="" overflow:="" padding:="" sans="" style="background-color: #0f192a; border: 1px dashed #999999; color: #cfecec;" vera="" width:="">PS ><code style="color: #b43d3d;"> GWmi </code>Win32_PerfFormattedData_PerfOS_Processor | <code style="color: #b43d3d;">Select</code> __Path
__PATH
------
namespace Category/Class InstanceName
\\localhost\root\cimv2:Win32_PerfFormattedData_PerfOS_Processor.Name="0"
...
\\.\root\cimv2:Win32_PerfFormattedData_PerfOS_Processor.Name="_Total"</pre></div>
Note: "." and "localhost" are synonyms.<br />
Note: Knowing paths will allow us to move faster around as well as writing WMI queries as I will demonstrate later.<br />
<br />
<br />
<h3>
Putting .NET to work</h3>
Using <code>System.Diagnostics.PerformanceCounterCategory</code> class (which I prefer for fetching single values that update often) lets take two Categories as an example.<br />
Note: I saw a lot of questions regarding WMIRefresher that exists in VB and has no apparent counterpart in Powershell and this is it, IMO, since variable pointing to counter object holds (lightweight) connection to path reducing the overhead when fetching next value. Another good example of refreshing WMI data is described in <a href="http://blogs.msdn.com/b/powershell/archive/2013/08/19/cim-cmdlets-some-tips-amp-tricks.aspx">Tip 10/c</a>. This "trick" will do the fetch of entire <b>Processor Information</b> category in approximately 0.3s on my laptop which is about as fast as it gets. Remember that the first execution usually takes a while (couple of seconds).
<br />
<div style='overflow-x:auto;overflow-y:hidden;white-space: nowrap;'>
<pre 100="" 18px="" 1em="" 1px="" auto="" consolas="" courier="" font-size:="" font-style:="" font-weight:="" itstream="" line-height:="" mono="" monospace="" new="" normal="" ourier="" overflow:="" padding:="" sans="" style="background-color: #0f192a; border: 1px dashed #999999; color: #cfecec;" vera="" width:="">PS > [System.Diagnostics.PerformanceCounterCategory]::<code style="color: deeppink;">GetCategories()</code> | <code style="color: #b43d3d;"> Select </code>CategoryName | <code style="color: #b43d3d;">Sort</code> CategoryName
CategoryName
------------
.NET CLR Data
.NET CLR Security
.NET Data Provider for Oracle
Cache
Memory
Network Interface
Objects
PhysicalDisk
Process
Thread</pre></div>
.... there are 91 in total.<br />
<div style='overflow-x:auto;overflow-y:hidden;white-space: nowrap;'>
<pre 100="" 18px="" 1em="" 1px="" auto="" consolas="" courier="" font-size:="" font-style:="" font-weight:="" itstream="" line-height:="" mono="" monospace="" new="" normal="" ourier="" overflow:="" padding:="" sans="" style="background-color: #0f192a; border: 1px dashed #999999; color: #cfecec;" vera="" width:="">PS > <code style="color: #aa7700;">$Category</code> = <code style="color: #1dc116;">"Memory"</code>
PS > (<code style="color: #b43d3d;">New-Object</code> Diagnostics.PerformanceCounterCategory(<code style="color: #aa7700;">$Category</code>)).<code style="color: deeppink;">GetCounters("")</code> |
<code style="color: #b43d3d;"> Select </code>CounterName | <code style="color: #b43d3d;">Sort</code> CounterName
CounterName
-----------
Available Bytes
Cache Bytes
Cache Faults/sec
Modified Page List Bytes
Page Faults/sec
Pages/sec
Pool Nonpaged Bytes
Standby Cache Reserve Bytes
System Driver Resident Bytes
Write Copies/sec</pre></div>
.... there are 35 in total.<br />
<div style='overflow-x:auto;overflow-y:hidden;white-space: nowrap;'>
<pre 100="" 18px="" 1em="" 1px="" auto="" consolas="" courier="" font-size:="" font-style:="" font-weight:="" itstream="" line-height:="" mono="" monospace="" new="" normal="" ourier="" overflow:="" padding:="" sans="" style="background-color: #0f192a; border: 1px dashed #999999; color: #cfecec;" vera="" width:="">PS > (<code style="color: #b43d3d;">New-Object</code> Diagnostics.PerformanceCounterCategory(<code style="color: #aa7700;">$Category</code>)).<code style="color: deeppink;">GetInstanceNames()</code>
PS ></pre></div>
So, <b>Memory</b> category has 35 Counters and no Instances which means you can fetch values directly:<br />
<div style='overflow-x:auto;overflow-y:hidden;white-space: nowrap;'>
<pre 100="" 18px="" 1em="" 1px="" auto="" consolas="" courier="" font-size:="" font-style:="" font-weight:="" itstream="" line-height:="" mono="" monospace="" new="" normal="" ourier="" overflow:="" padding:="" sans="" style="background-color: #0f192a; border: 1px dashed #999999; color: #cfecec;" vera="" width:="">PS > <code style="color: #aa7700;">$tmp</code> = (<code style="color: #b43d3d;">New-Object</code> Diagnostics.PerformanceCounter(<code style="color: #aa7700;">$Category</code>, <code style="color: #1dc116;">"Available MBytes"</code>)
PS > <code style="color: #aa7700;">$tmp</code>.<code style="color: deeppink;">NextValue()</code>
5009</pre></div>
Note: <samp>Available MBytes</samp> is ever updating value not dependant on number of samples. Thus calling <samp><b>NextValue()</b></samp> was unnecessary but good practice.<br />
<br />
Fetching data from <b>Processor Information</b> category is a bit different:<br />
<div style='overflow-x:auto;overflow-y:hidden;white-space: nowrap;'>
<pre 100="" 18px="" 1em="" 1px="" auto="" consolas="" courier="" font-size:="" font-style:="" font-weight:="" itstream="" line-height:="" mono="" monospace="" new="" normal="" ourier="" overflow:="" padding:="" sans="" style="background-color: #0f192a; border: 1px dashed #999999; color: #cfecec;" vera="" width:="">PS > <code style="color: #aa7700;">$Category</code> = <code style="color: #1dc116;">"Processor Information"</code>
PS > (<code style="color: #b43d3d;">New-Object</code> Diagnostics.PerformanceCounterCategory(<code style="color: #aa7700;">$Category</code>)).<code style="color: deeppink;">GetInstanceNames()</code> | <code style="color: #b43d3d;">Sort</code>
_Total
0,_Total
0,0
0,1
0,2
0,3</pre></div>
<b>Processor Information</b> has 6 instances which means I am writing this on dual-core laptop with HT enabled (2 physical CPUs, 2 logical CPU's and the 2 Totals). It is rather interesting to play with these counters on proper servers. For now, important thing is to notice that "_Total" stands for entire box, "N, _Total" represents "Socket N" instance while "N,M" stands for "Socket,CPU" instance.<br />
<br />
To fetch particular perf-counter value, I have to provide <b>InstanceName</b>:<br />
<div style='overflow-x:auto;overflow-y:hidden;white-space: nowrap;'>
<pre 100="" 18px="" 1em="" 1px="" auto="" consolas="" courier="" font-size:="" font-style:="" font-weight:="" itstream="" line-height:="" mono="" monospace="" new="" normal="" ourier="" overflow:="" padding:="" sans="" style="background-color: #0f192a; border: 1px dashed #999999; color: #cfecec;" vera="" width:="">PS > <code style="color: #aa7700;">$Category</code> = <code style="color: #1dc116;">"Processor Information"</code>
PS > <code style="color: #aa7700;">$InstanceName</code> = <code style="color: #1dc116;">"_Total"</code>
PS > <code style="color: #aa7700;">$tmp</code> = <code style="color: #b43d3d;">New-Object</code> Diagnostics.PerformanceCounter(<code style="color: #aa7700;">$Category</code>, <code style="color: #1dc116;">"% Processor Time"</code>)
PS > <code style="color: #aa7700;">$tmp</code>.InstanceName = <code style="color: #aa7700;">$InstanceName</code>
PS > <code style="color: #aa7700;">$tmp</code>.<code style="color: deeppink;">NextValue()</code>
21.97821</pre></div>
<b>Tip:</b>There is an overload allowing you to write <samp>New-Object Diagnostics.PerformanceCounter("Processor Information", "% Processor Time", "_Total")</samp> instead of providing <b>Category/InstanceName</b> members via variables.<br />
<br />
Let's check on New-Object members:
<br />
<div style='overflow-x:auto;overflow-y:hidden;white-space: nowrap;'>
<pre 100="" 18px="" 1em="" 1px="" auto="" consolas="" courier="" font-size:="" font-style:="" font-weight:="" itstream="" line-height:="" mono="" monospace="" new="" normal="" ourier="" overflow:="" padding:="" sans="" style="background-color: #0f192a; border: 1px dashed #999999; color: #cfecec;" vera="" width:="">PS > <code style="color: #aa7700;">$tmp</code>
<b>CategoryName :</b> Processor Information
<b>CounterHelp :</b> % Processor Time is the percentage of elapsed
time that the processor spends to execute a non-Idle thread. It is
calculated by measuring the percentage of time that the processor spends
executing the idle thread and ...
<b>CounterName :</b> % Processor Time
<b>CounterType :</b> Timer100NsInverse
<b>InstanceLifetime :</b> Global
<b>InstanceName :</b> Total
<b>ReadOnly :</b> True
<b>MachineName :</b>
<b>RawValue :</b> 131585400020
<b>Site :</b>
<b>Container :</b> </pre></div>
Note: It is always a good practice to read <b>CounterHelp</b> and make note of <b>CounterType.</b> This will tell you a lot about values obtained.<br />
<div style='overflow-x:auto;overflow-y:hidden;white-space: nowrap;'>
<pre 100="" 18px="" 1em="" 1px="" auto="" consolas="" courier="" font-size:="" font-style:="" font-weight:="" itstream="" line-height:="" mono="" monospace="" new="" normal="" ourier="" overflow:="" padding:="" sans="" style="background-color: #0f192a; border: 1px dashed #999999; color: #cfecec;" vera="" width:="">PS > <code style="color: #aa7700;">$tmp</code>.RawValue
131585502870
PS > <code style="color: #aa7700;">$tmp</code>.<code style="color: deeppink;">NextSample()</code>
RawValue : 131585517490
BaseValue : 0
SystemFrequency : 2533388
CounterFrequency : 2533388
CounterTimeStamp : 117273907751
TimeStamp : 44097511588
TimeStamp100nSec : 130899951977398973
CounterType : Timer100NsInverse</pre></div>
Note: Please check on <a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog2.html">previous blog</a>, paragraph about Raw/Formatted counters and sampling.<br />
Note: There is a plenitude of counters and you are free to explore them in search for one that suits your needs best. I.e. if you do not want to bother with Instances here, you can use Win32_PerfFormattedData_PerfOS_Processor Class which uses absolute Index to each CPU (see below):
<br />
<div style='overflow-x:auto;overflow-y:hidden;white-space: nowrap;'>
<pre 100="" 18px="" 1em="" 1px="" auto="" consolas="" courier="" font-size:="" font-style:="" font-weight:="" itstream="" line-height:="" mono="" monospace="" new="" normal="" ourier="" overflow:="" padding:="" sans="" style="background-color: #0f192a; border: 1px dashed #999999; color: #cfecec;" vera="" width:="">PS ><code style="color: #b43d3d;"> GCim </code>Win32_PerfFormattedData_PerfOS_Processor | <code style="color: #b43d3d;">Select</code> Name
Name
----
0
1
...
_Total</pre></div>
<br />
<h4>
So, how do we tell if Counter has Instances?</h4>
<div style='overflow-x:auto;overflow-y:hidden;white-space: nowrap;'>
<pre 100="" 18px="" 1em="" 1px="" auto="" consolas="" courier="" font-size:="" font-style:="" font-weight:="" itstream="" line-height:="" mono="" monospace="" new="" normal="" ourier="" overflow:="" padding:="" sans="" style="background-color: #0f192a; border: 1px dashed #999999; color: #cfecec;" vera="" width:="">PS > <code style="color: #aa7700;">$Category</code> = <code style="color: #1dc116;">"Processor Information"</code>
PS > <code style="color: #b43d3d;">New-Object</code> Diagnostics.PerformanceCounterCategory(<code style="color: #aa7700;">$Category</code>)
<table>
<tbody>
<tr>
<td>CategoryName </td><td>CategoryHelp </td><td>CategoryType </td><td>MachineName</td>
</tr>
<tr>
<td>------------ </td><td>------------ </td><td>------------ </td><td>-----------</td>
</tr>
<tr>
<td>Processor Information</td><td>The Processor Information ...</td><td>MultiInstance </td><td>.</td>
</tr>
</tbody></table>
PS > <code style="color: #aa7700;">$Category</code> = <code style="color: #1dc116;">"Memory"</code>
PS > <code style="color: #b43d3d;">New-Object</code> Diagnostics.PerformanceCounterCategory(<code style="color: #aa7700;">$Category</code>)
<table>
<tbody>
<tr>
<td>CategoryName </td><td>CategoryHelp </td><td>CategoryType </td><td>MachineName</td>
</tr>
<tr>
<td>------------ </td><td>------------ </td><td>------------ </td><td>-----------</td>
</tr>
<tr>
<td>Memory </td><td>The Memory performance obj...</td><td>SingleInstance </td><td>.</td>
</tr>
</tbody></table>
</pre></div>
The answer is obviously in <b>CategoryType</b> member of <b>PerformanceCounterCategory</b> class which you should check while iterating.<br />
Note: The dot in <b>MachineName</b> stands for localhost.<br />
<br />
You can check instance values directly with WMI too using their individual instance paths. Remember WMI classes:
<samp>GWmi -List | Select Name | Where {$_.Name -match "Win32_PerfForm"}</samp><br />
<br />
Let's take <b>Win32_PerfFormattedData_PerfOS_Processor</b> for the example:<br />
<div style='overflow-x:auto;overflow-y:hidden;white-space: nowrap;'>
<pre 100="" 18px="" 1em="" 1px="" auto="" consolas="" courier="" font-size:="" font-style:="" font-weight:="" itstream="" line-height:="" mono="" monospace="" new="" normal="" ourier="" overflow:="" padding:="" sans="" style="background-color: #0f192a; border: 1px dashed #999999; color: #cfecec;" vera="" width:="">PS ><code style="color: #b43d3d;"> GWmi</code> Win32_PerfFormattedData_PerfOS_Processor | <code style="color: #b43d3d;">Select</code> __Path
__PATH
------
\\localhost\root\cimv2:Win32_PerfFormattedData_PerfOS_Processor.Name="0"
...
\\.\root\cimv2:Win32_PerfFormattedData_PerfOS_Processor.Name="_Total"</pre></div>
Note: Remember that <b>Path</b> property begins with two underscores.<br />
<br />
Now that you know the path to a WMI instance, you can access it directly by converting the WMI path to a WMI object:<br />
<div style='overflow-x:auto;overflow-y:hidden;white-space: nowrap;'>
<pre 100="" 18px="" 1em="" 1px="" auto="" consolas="" courier="" font-size:="" font-style:="" font-weight:="" itstream="" line-height:="" mono="" monospace="" new="" normal="" ourier="" overflow:="" padding:="" sans="" style="background-color: #0f192a; border: 1px dashed #999999; color: #cfecec;" vera="" width:="">PS > [WMI]<code style="color: #1dc116;">'Win32_PerfFormattedData_PerfOS_Processor.Name="0"'</code>
__GENUS : 2
__CLASS : Win32_PerfFormattedData_PerfOS_Processor
__SUPERCLASS : Win32_PerfFormattedData
__DYNASTY : CIM_StatisticalInformation
__RELPATH : Win32_PerfFormattedData_PerfOS_Processor.Name="0"
__PROPERTY_COUNT : 24
__DERIVATION : {Win32_PerfFormattedData, Win32_Perf, CIM_StatisticalInformation}
__SERVER : ...
__NAMESPACE : root\cimv2
__PATH : \\...\root\cimv2:Win32_PerfFormattedData_PerfOS_Processor.Name="0"
C1TransitionsPersec : 263
DPCRate : 0
DPCsQueuedPersec : 27
InterruptsPersec : 1054
Name : 0
PercentDPCTime : 0
PercentIdleTime : 99
PercentInterruptTime : 0
PercentPrivilegedTime : 0
PercentProcessorTime : 0
PercentUserTime : 0</pre></div>
Note: <samp>__PROPERTY_COUNT : 24</samp> means I trimmed some lines.<br />
Pick one counter:
<br />
<div style='overflow-x:auto;overflow-y:hidden;white-space: nowrap;'>
<pre 100="" 18px="" 1em="" 1px="" auto="" consolas="" courier="" font-size:="" font-style:="" font-weight:="" itstream="" line-height:="" mono="" monospace="" new="" normal="" ourier="" overflow:="" padding:="" sans="" style="background-color: #0f192a; border: 1px dashed #999999; color: #cfecec;" vera="" width:="">PS > ([WMI]<code style="color: #1dc116;">'Win32_PerfFormattedData_PerfOS_Processor.Name="0"'</code>).PercentProcessorTime
68
PS > [WMI]<code style="color: #1dc116;">'Win32_Service.Name="RemoteAccess"'</code>
ExitCode : 1077
Name : RemoteAccess
ProcessId : 0
StartMode : Disabled
State : Stopped
Status : OK
PS > [WMI]<code style="color: #1dc116;">'\\.\root\cimv2:Win32_LogicalDisk.DeviceID="C:"'</code>
DeviceID : C:
DriveType : 3
ProviderName :
FreeSpace : 68152655872
Size : 168037445632
VolumeName : System</pre></div>
Note that '.' stands for 'localhost'. You can provide Server name here.<br />
Note: You can also specify the full WMI path, including a machine name to access WMI objects on remote systems (provided you have sufficient access rights).
<br />
<br />
<b>Tip:</b> There is a hidden object property called "PSTypeNames" which will tell you the object type as well as the inheritance chain:<br />
<div style='overflow-x:auto;overflow-y:hidden;white-space: nowrap;'>
<pre 100="" 18px="" 1em="" 1px="" auto="" consolas="" courier="" font-size:="" font-style:="" font-weight:="" itstream="" line-height:="" mono="" monospace="" new="" normal="" ourier="" overflow:="" padding:="" sans="" style="background-color: #0f192a; border: 1px dashed #999999; color: #cfecec;" vera="" width:="">PS > (<code style="color: #b43d3d;">GWmi</code> Win32_PhysicalMemory).PSTypeNames
System.Management.ManagementObject#root\cimv2\Win32_PhysicalMemory
System.Management.ManagementObject#root\cimv2\CIM_PhysicalMemory
System.Management.ManagementObject#root\cimv2\CIM_Chip
System.Management.ManagementObject#root\cimv2\CIM_PhysicalComponent
System.Management.ManagementObject#root\cimv2\CIM_PhysicalElement
System.Management.ManagementObject#root\cimv2\CIM_ManagedSystemElement
System.Management.ManagementObject#Win32_PhysicalMemory
System.Management.ManagementObject#CIM_PhysicalMemory
System.Management.ManagementObject#CIM_Chip
System.Management.ManagementObject#CIM_PhysicalComponent
System.Management.ManagementObject#CIM_PhysicalElement
System.Management.ManagementObject#CIM_ManagedSystemElement
System.Management.ManagementObject
System.Management.ManagementBaseObject
System.ComponentModel.Component
System.MarshalByRefObject
System.Object</pre></div>
Of course, the type listed at the top is telling you the most:<br />
<div style='overflow-x:auto;overflow-y:hidden;white-space: nowrap;'>
<pre 100="" 18px="" 1em="" 1px="" auto="" consolas="" courier="" font-size:="" font-style:="" font-weight:="" itstream="" line-height:="" mono="" monospace="" new="" normal="" ourier="" overflow:="" padding:="" sans="" style="background-color: #0f192a; border: 1px dashed #999999; color: #cfecec;" vera="" width:="">PS > (<code style="color: #b43d3d;">GWmi</code> Win32_PhysicalMemory).PSTypeNames[0]
System.Management.ManagementObject#root\cimv2\Win32_PhysicalMemory</pre></div>
<b>PSTypeNames</b> will work for all objects and might come handy navigating namespaces.<br />
<br />
Note: You can do things with WMI objects, not just read counters. Check, for example, Win32_LogicalDisk device <b>Chkdsk</b> method:<br />
<div style='overflow-x:auto;overflow-y:hidden;white-space: nowrap;'>
<pre 100="" 18px="" 1em="" 1px="" auto="" consolas="" courier="" font-size:="" font-style:="" font-weight:="" itstream="" line-height:="" mono="" monospace="" new="" normal="" ourier="" overflow:="" padding:="" sans="" style="background-color: #0f192a; border: 1px dashed #999999; color: #cfecec;" vera="" width:="">PS > ([WMI]<code style="color: #1dc116;">'\\.\root\cimv2:Win32_LogicalDisk.DeviceID="C:"'</code>).Chkdsk
OverloadDefinitions
-------------------
System.Management.ManagementBaseObject Chkdsk(System.Boolean FixErrors,
System.Boolean VigorousIndexCheck, System.Boolean SkipFolderCycle,
System.Boolean ForceDismount, System.Boolean RecoverBadSectors,
System.Boolean OkToRunAtBootUp)</pre></div>
This functionality as well as accessing remote machines is beyond scope of the document and mentioned here just for the sake of completeness.<br />
<br />
It is also worth noting you can call WMI methods with CIM cmdlets. Please see this <a href="http://powershell.com/cs/blogs/tips/archive/2013/03/07/calling-wmi-methods-with-cim-cmdlets.aspx">POWERTIP</a> for details if you're interested.<br />
<br />
<h4>
Useful WMI links:</h4>
<a href="http://powershell.com/cs/blogs/tips/archive/2013/01/28/auto-discovering-online-help-for-wmi.aspx">Powertip 1, auto-discovering online help for wmi</a><br />
<a href="http://powershell.com/cs/blogs/tips/archive/2010/07/30/getting-help-on-wmi-methods.aspx">Powertip 2, getting help on wmi methods</a><br />
<br />
<br />
<h3>
WQL</h3>
I mentioned earlier you can write your own WQL queries to fetch data from WMI objects. <a href="https://msdn.microsoft.com/en-us/library/windows/desktop/aa394606%28v=vs.85%29.aspx">WQL</a> is the WMI Query Language, a subset of the ANSI SQL with minor semantic changes.<br />
<samp>GWmi Win32_Process -Filter "Name like ""power%.exe"""</samp> translates to WQL query <samp>'select * from Win32_Process where Name like "power%.exe"'</samp>. So, to get process owner for example:
<br />
<div style='overflow-x:auto;overflow-y:hidden;white-space: nowrap;'>
<pre 100="" 18px="" 1em="" 1px="" auto="" consolas="" courier="" font-size:="" font-style:="" font-weight:="" itstream="" line-height:="" mono="" monospace="" new="" normal="" ourier="" overflow:="" padding:="" sans="" style="background-color: #0f192a; border: 1px dashed #999999; color: #cfecec;" vera="" width:=""><code style="color: #aa7700;">$processes</code> = <code style="color: #b43d3d;">GWmi</code> -Query <code style="color: #1dc116;">'select * from Win32_Process where Name like "power%.exe"'</code>
<code style="color: #aa7700;">$extraproc</code> =
<code style="color: #b43d3d;">ForEach</code> (<code style="color: #aa7700;">$process</code> in <code style="color: #aa7700;">$processes</code>)
{
<code style="color: #b43d3d;">Add-Member</code> -MemberType NoteProperty -Name Owner
-Value ((<code style="color: #aa7700;">$process</code>.<code style="color: deeppink;">GetOwner()</code>).User)
-InputObject <code style="color: #aa7700;">$process</code> -PassThru
}
<code style="color: #aa7700;">$extraproc</code> | <code style="color: #b43d3d;">Select-Object</code> -Property Name, Owner</pre></div>
Note: Make sure Add-Member ... -PassThrough line is not broken if you want this code to work.<br />
Personally, I find WQL inadequate since it's missing aggregation functions thus I use it very rarely.<br />
<br />
<br />
<h3>
Summing it up</h3>
<h4>
get classes:</h4>
<code>GWmi -List</code><br />
or<br />
<code> Get-CimClass | Select CIMClassName</code><br />
or<br />
<code>[System.Diagnostics.PerformanceCounterCategory]::GetCategories()</code><br />
<br />
<h4>
shorten the list:</h4>
<code>GWmi -List Win32_*memory* | Select Name</code><br />
or <br />
<code> Get-CimClass | Select CIMClassName | Where {$_.CimClassName -match "memory"}</code><br />
or<br />
<code>[System.Diagnostics.PerformanceCounterCategory]::GetCategories() | Where {$_.CategoryName -match "memory"} | Select CategoryName</code><br />
<br />
<h4>
list counters:</h4>
<code>GWmi Win32_PhysicalMemory</code><br />
or<br />
<code>GCim CIM_PhysicalMemory</code><br />
or<br />
<code>(New-Object Diagnostics.PerformanceCounterCategory("Memory")).GetCounters("") | Select CounterName | Sort CounterName</code><br />
<br />
<h4>
and for WQL, write a query:</h4>
<code>GWmi -Query 'select Manufacturer from Win32_PhysicalMemory where BankLabel = "BANK 0"'</code><br />
or<br />
<code>GCim -Query 'Select Manufacturer from CIM_PhysicalMemory Where BankLabel = "BANK 0"'</code><br />
<br />
Note: Opening communication and fetching objects from WMI server might take considerable amount of time. Counting CPU's takes at least few seconds on my boxes:<br />
<code>PS > Measure-Command {((GCim -Namespace root/CIMV2 -ClassName CIM_Processor).NumberOfLogicalProcessors | Measure-Object -Sum).Sum}</code>
<samp><br />
...<br />
TotalSeconds : 2.2558991<br />
...</samp><br />
<b>Tip:</b> The fastest way to learn how many CPUs there are on the box is <code>(GCim Win32_ComputerSystem).NumberOfLogicalProcessors</code><br />
Note: In my experience, best time to fetch some value (if not cached) is about 0.3 seconds so that's what I'm aiming for always.<br />
<br />
<br />
<h3>
Conclusion:</h3>
To speed up fetching counter data, where available, I like to use <samp>System.Diagnostics</samp> .NET class:<br />
<code>$System_CSpS = New-Object Diagnostics.PerformanceCounter("System", "Context Switches/sec")<br />
$System_CSpS.NextValue()</code><br />
Since some of the performance counters are empty upon first access, calling NextValue() is a good habit. Subsequent calls to $var.NextValue() are lightning fast so put it in variable. The next thing this technique is good for are values that are always there; such as amount of (free)RAM, Context Switches per second and so on. Although possible, I do not use this mechanism for values that might disappear, such as number of threads belonging to some process as process might die.<br />
If you are not able to use <samp>System.Diagnostics</samp> or you prefer CIM approach you can always fall back to <a href="http://blogs.msdn.com/b/powershell/archive/2013/08/19/cim-cmdlets-some-tips-amp-tricks.aspx">Tip 10/c</a>:<br />
<code>PS > # Get instance of a class<br />
PS > $p = Get-CimInstance -ClassName Win32_PerfFormattedData_PerfOS_Processor<br />
PS > # Perform get again by passing the instance received earlier, and get the updated properties. The value of $p remains unchanged.<br />
PS > $p | Get-CimInstance | select PercentProcessorTime<br />
</code>
<br />
Beware that for counters with instances, you need to supply InstanceName:<br />
<code>$InstanceName = "_Total"<br />
$PI_PT = New-Object Diagnostics.PerformanceCounter("Processor Information", "% Processor Time")<br />
$PI_PT.InstanceName = $InstanceName<br />
#or $PI_PT = New-Object Diagnostics.PerformanceCounter("Processor Information", "% Processor Time", "_Total")<br />
$PI_PT.NextValue()</code><br />
If you are interested in measuring just one value, <a href="https://technet.microsoft.com/en-us/library/hh849685.aspx">Get-Counter</a> is your friend but I prefer TAB completion of CIM classes approach over iterating through paths (say, <code>(Get-Counter -listset memory).paths</code>).<br />
<br />
You can almost always accomplish the same thing using WMI and CIM cmdlest. Prefer CIM over WMI.<br />
<br />
WQL is cumbersome and lacking many commands. Avoid.<br />
<br />
Remember the hierarchy:<br />
Category (Class if you prefer)<br />
Counter(s)<br />
Instance(s)<br />
Every Category has Counters but not all of the Counters have Instances. Check before using.<br />
<br />
<br />
Next blog will deal with several specific counters and the meaning of the values obtained in terms of performance.<br />
<br />
In this series:<br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog1.html">BLOG 1: PerfCounters infrastructure</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog2.html">BLOG 2: PerfCounters Raw vs. Formatted values</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog3.html">BLOG 3: PerfCounters, fetching the values </a><br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog4.html">BLOG 4: PerfCounters, CPU perf data</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog5.html">BLOG 5: PerfCounters, Memory perf data</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/12/windows-perf-counters-blog6.html">BLOG 6: PerfCounters, Disk/IO perf data</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/12/windows-perf-counters-blog7.html">BLOG 7: PerfCounters, Network and Contention perf data</a><br />Tončihttp://www.blogger.com/profile/10650361408804728027noreply@blogger.com5tag:blogger.com,1999:blog-7850846940198540592.post-56973768051864915722015-11-09T09:09:00.000+01:002016-12-03T14:15:25.169+01:00Windows PerfCounters and Powershell - Raw vs. Formatted values<br /><h3>How to interpret Raw data from Windows performance counters.</h3><br>
<strong>Tip:</strong> An alias for Get-CimInstance is GCim and alias for Get-WmiObject is GWmi.<br>
<br>
In the <a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog1.html">first blog post</a>, I covered what WMI/CIM is and how to get info from there. Last I talked about was RawData counters:<br>
<code>Get-CimInstance -Class Win32_PerfRawData_PerfOS_Processor</code><br>
<br />
<samp>Name : _Total<br>
...<br>
PercentIdleTime : 78061457390</samp><br><br><br>
<h4>
Understanding RawData:</h4>
By itself, a RawData value is a sample but important thing is to determine what concrete sample value actually is and how to convert it to a form we understand. In this example, <a href="https://msdn.microsoft.com/en-us/library/aa394317%28v=vs.85%29.aspx">MSDN</a> tells us <samp>PercentIdleTime</samp> is a counter of type <samp>542180608</samp>:<br>
<strong>PercentIdleTime</strong><br>
<small><strong>Data type:</strong> uint64</small><br>
<small><strong>Access type:</strong> Read-only</small><br>
<small><strong>Qualifiers:</strong> DisplayName ("% Idle Time") , CounterType (542180608) , DefaultScale (0) , PerfDetail (400)</small><br><br>
Bear in mind, most of RawData counters need 2 samples to produce humanly readable result thus now we need a formula to convert Raw counter values into something meaningful. <br>Numeric-to-Name conversion (<samp>542180608 -> PERF_100NSEC_TIMER</samp>) of counter type values is listed in <a href="https://msdn.microsoft.com/en-us/library/aa389383%28v=VS.85%29.aspx">this MSDN page</a>. The actual formula is then located under entries listed <a href="https://technet.microsoft.com/en-us/library/cc785636%28v=ws.12%29.aspx">here</a> as described in <a href="https://technet.microsoft.com/en-us/library/cc728274%28v=ws.12%29.aspx">this page</a>:
<p style="text-indent: 5em;">(N1 - N0) / (D1 - D0) x 100, where the denominator (D) represents the total elapsed time of the sample interval, and the numerator (N) represents the portions of the sample interval during which the monitored components were active.</p>
This translates to:
<pre><code> $Val =
(
(PercentIdleTime_Sample[n] - PercentIdleTime_Sample[n-1])
/
(Timestamp_Sys100NSSample[n] - Timestamp_Sys100NSSample[n-1])
) *100</code></pre>
Note: Although formula is correct according to documentation, it usually summarizes result over <strong>all</strong> CPU's (say, when fetching CPU utilization per process/thread) thus the result will most likely be well over 100% on modern boxes. In such case, we need to divide samples with total number of CPU's:<br>
<samp> $Val = <br>
(<br>
(PercentIdleTime_Sample[n] - PercentIdleTime_Sample[n-1]) <br>
/( <br>
(Timestamp_Sys100NSSample[n] - Timestamp_Sys100NSSample[n-1]) <br>
*$TotProc) <br>
) *100</samp><br><br>
With the Class <samp>Win32_PerfRawData_PerfOS_Processor</samp> it's easy. It has Instance named <samp>_Total</samp> and when you apply formula to its values you will get proper result. I will talk of this more in next blogs.<br><br>
<h3>
Conclusion:</h3>
So, why Formatted and Raw counters? After all, Formatted data is coming from Raw counters. First, we have to remember that Raw counters are used for collecting N samples as the naked number obtained is meaningless. So, let's say that in above example I asked for CPU usage by certain process that I only just started. Formatted counter will either have 0 or NaN value in it while Raw counter will produce some number given that sampling time is usually 100ns. Well, you might say, it's the same as "get formatted sample | check if it is a number | no? -> take another formatted sample" and you'd be right. But you should take into account the rounding happening in calculating Formatted values internally as well as in your script (check that data type of Formatted counters is usually UINT!) and also the latency involved in WMI provider populating formatted data counters.<br><br>
If you are set on using Raw counters, bear in mind you need formulas for transforming Samples into Values.<br><br>
All said, it is actually a question of choice whether to use one type or the other as you will see in the script I'll be describing in the final blog.<br><br>
Examples of calculating Value from Counter type and samples <a href="https://technet.microsoft.com/en-us//sysinternals/aa371891">@Sysinternals</a> and <a href="https://msdn.microsoft.com/en-us/library/windows/desktop/aa371891%28v=vs.85%29.aspx">@MSDN</a>.<br />
<br />
Next blog will deal with various ways to obtain performance data.<br />
<br />
In this series:<br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog1.html">BLOG 1: PerfCounters infrastructure</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog2.html">BLOG 2: PerfCounters Raw vs. Formatted values</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog3.html">BLOG 3: PerfCounters, fetching the values </a><br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog4.html">BLOG 4: PerfCounters, CPU perf data</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog5.html">BLOG 5: PerfCounters, Memory perf data</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/12/windows-perf-counters-blog6.html">BLOG 6: PerfCounters, Disk/IO perf data</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/12/windows-perf-counters-blog7.html">BLOG 7: PerfCounters, Network and Contention perf data</a><br />Tončihttp://www.blogger.com/profile/10650361408804728027noreply@blogger.com0tag:blogger.com,1999:blog-7850846940198540592.post-76421388047925488552015-11-03T15:29:00.000+01:002015-12-14T10:15:01.419+01:00Windows PerfCounters and Powershell - InfrastructureIn this series of blogs, I will cover Windows performance counters infrastructure, collecting and interpreting the data and, in final blog, provide the Powershell script built on this. I should note that I am very new to Powershell so take my empirical findings with grain of salt. Also, coming from Linux bash, I found Powershell confusing at first but, after getting comfortable with concept of passing Objects down the pipe, I have to say I like it a lot.
<br />
<br />
It all starts in WDM framework (<a href="https://msdn.microsoft.com/en-us/library/aa394052%28v=vs.85%29.aspx">Windows Driver Model</a>) where metrics is collected for WMI-for-WDM enabled device drivers. The classes created by the WDM provider to represent device driver data reside in the "Root\WMI" namespace. I will talk of namespaces shortly.<br />
So, the WDM provider records information about WDM operations in the <a href="https://msdn.microsoft.com/en-us/library/aa394582%28v=vs.85%29.aspx">WMI Log Files</a>. However, WMI is not just mirroring physical data provided by WDM but it also adds (logical) counters.<br />
Windows Management Instrumentation (WMI) is the Microsoft implementation of Web-Based Enterprise Management (WBEM), which is an industry initiative to develop a standard technology for accessing management information in an enterprise environment. WMI uses the Common Information Model (CIM) industry standard to represent systems, applications, networks, devices, and other managed components. CIM is developed and maintained by the Distributed Management Task Force (DMTF). I will talk of CIM cmdlets in Powershell shortly.<br />
<br />
There are many ways to collect performance data in Windows, like through <a href="https://msdn.microsoft.com/en-us/library/windows/desktop/aa373219%28v=vs.85%29.aspx">Registry functions</a>, but not all of them make sense. Since WMI is a database-like service always returning live data, it is used to find out details about physical and logical computer configurations and works locally as well as remotely (only for Windows OS). It is organized in PerfCounter classes which return PerfCounter objects. So the usual way to start exploring is to collect the Classes with <code>Get-WmiObject -List</code>.
First lines are similar to below: <br />
<samp> NameSpace: ROOT\CIMV2<br />
<br />
Name Methods Properties<br />
---- ------- ----------<br />
...</samp><br />
<br />
So what is NAMESPACE? The WMI infrastructure is a part of Microsoft Windows operating system and has two components: the WMI service (winmgmt), including the WMI Core, and the WMI repository.<br />
WMI repository is organized in WMI namespaces. The WMI service creates some namespaces such as root\default, root\cimv2, and root\subscription at system startup and preinstalls a default set of class definitions, including the Win32 Classes, the WMI System Classes, and others. The remaining namespaces found on your system are created by providers for other parts of the operating system or products.<br />
The WMI service acts as an intermediary between the providers (management applications) and WMI repository. Only static data about objects is stored in the repository, such as the classes defined by the providers meaning WMI obtains most data dynamically from the provider when a client requests it.<br />
A WMI consumer is a management application or script that interacts with the WMI infrastructure. A management application can query, enumerate data, run provider methods, or subscribe to events by calling either the COM API for WMI or the Scripting API for WMI (like WMI-enabled Powershell cmdlets). The only data or actions available for a managed object, such as a disk drive or a service, are those that a provider supplies.<br />
So, basically, NAMESPACE is a subfolder :-) The default namespace is “root\CIMV2” and you do not need to submit the namespace to a function call as long as the class you're addressing is located inside the default one. Most of us will use default namespace (ROOT\CIMV2) for most of the tasks but let's list them here for the sake of completeness:<br />
<samp>PS > Get-WmiObject -Query “Select * from __Namespace” -Namespace Root | Select-Object -ExpandProperty Name<br />
subscription<br />
DEFAULT<br />
CIMV2<br />
Cli<br />
nap<br />
SECURITY<br />
SecurityCenter2<br />
RSOP<br />
WMI<br />
IntelNCS2<br />
directory<br />
Policy<br />
Interop<br />
ServiceModel<br />
SecurityCenter<br />
MSAPPS12<br />
Microsoft<br />
aspnet</samp><br />
<br />
Now, back to <samp>Get-WmiObject -List | Select Name</samp> output. We also see "duplicate" entries like this:<br />
<samp>CIM_SoftwareElementActions<br />
Win32_SoftwareElementAction</samp><br />
So, what's CIM? The <a href="https://msdn.microsoft.com/en-us/library/aa389234%28v=vs.85%29.aspx">Common Information Model</a> is an open standard that defines how managed elements in an IT environment are represented as a common set of objects and relationships between them. The Distributed Management Task Force maintains the CIM to allow consistent management of these managed elements, independent of their manufacturer or provider. So CIM cmdlets will allow you to gather management data in heterogeneous environment as opposed to WMI cmdlets which work only on Windows. See <a href="https://en.wikipedia.org/wiki/Common_Information_Model_%28computing%29">wiki</a> for the list of operating systems and provider-specific CIM implementations. This basically means that some of the WMI classes and their objects are copied to CIMv2 namespace to produce more standard objects.<br />
So, <a href="https://msdn.microsoft.com/en-us/library/dn792216%28v=vs.85%29.aspx">CIM provider for WMI</a> is consuming data from WMI and there are many benefits of using CIM in place of WMI:<br />
<ul>
<li>Use of WSMAN for remote access – no more DCOM errors.</li>
<small>Note: You can drop back to DCOM for accessing systems with WSMAN 2 installed.</small>
<li>Use of CIM sessions for accessing multiple machines.</li>
<li>Get-CIMClass for investigating WMI classes.</li>
<li>Improved way of dealing with WMI associations.</li>
<li>Get-CimInstance do not contain any methods.</li>
<small>Note: This may appear as a drawback but GWMI will also loose this info when returned from background job or remote session due to serialization.</small>
<li>Works on other operating systems.</li>
</ul>
<br />
Note: Should you still find yourself in need to work with boxes running older Windows (such as XP), you can still use CIM cmdlets only define DCOM as protocol:<br />
<samp>$oldProt = New-CimSessionOption -Protocol DCOM<br />
$oldBoxSession = New-CimSession -ComputerName someOldXPBox -SessionOption $option<br />
Get-CimInstance -ClassName Win32_SystemDevices -CimSession $oldBoxSession</samp><br />
However, DCOM is not firewall friendly and can be unavailable on destination box (which should run Windows OS).<br />
Note: CIM cmdlets are available as of Powershell v3.<br />
<br />
So, we learned how to list classes:<br />
<samp>WMI: Get-WmiObject -List<br />
CIM: Get-CimClass | Select CIMClassName | Sort CIMClassName</samp><br />
<small>Note:From now on, I will mostly use CIM commands as they provide TAB completion.</small><br />
<br />
Next, we need actual performance objects that chosen class provides:<br />
<samp>PS > Get-CimInstance -Class Win32_Process<br />
<table>
<tbody>
<tr>
<td>ProcessId</td><td>Name </td><td>HandleCount</td><td>WorkingSetSize</td><td>VirtualSize</td>
</tr>
<tr>
<td>---------</td><td>---- </td><td>-----------</td><td>--------------</td><td>-----------</td>
</tr>
<tr>
<td>0 </td><td>System Idle P...</td><td>0 </td><td>24576 </td><td>0 </td>
</tr>
<tr>
<td>4 </td><td>System </td><td>976 </td><td>2834432 </td><td>6344704 </td>
</tr>
</tbody></table>
</samp> <br />
This is simple class but you will encounter more complex ones having Instances:<br />
<samp>PS > Get-CimInstance -Class Win32_PerfFormattedData_Counters_ProcessorInformation<br /><br />
#<b>All CPU's:</b><br />
Name : _Total<br />
...<br />
PercentIdleTime : 73<br />
<br />
#<b>Core 0, total:</b><br />
Name : 0,_Total<br />
...<br />
PercentIdleTime : 73<br />
<br />
#<b>Core 0, CPU 0:</b><br />
Name : 0,0<br />
...<br />
PercentIdleTime : 100</samp><br />
and so on.<br />
<br />
Notice also that I used <samp>PerfFormattedData</samp> class in this example. There are RawData classes too but that's for next blog.<br />
<samp>PS > Get-CimInstance -Class Win32_PerfRawData_Counters_ProcessorInformation<br />
Name : _Total<br />
...<br />
PercentIdleTime : 66682471448</samp><br />
<br />
This concludes the first blog in series. In next blog I will deal with various flavours of counter values.<br />
<br />
In this series:<br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog1.html">BLOG 1: PerfCounters infrastructure</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog2.html">BLOG 2: PerfCounters Raw vs. Formatted values</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog3.html">BLOG 3: PerfCounters, fetching the values </a><br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog4.html">BLOG 4: PerfCounters, CPU perf data</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/11/windows-perf-counters-blog5.html">BLOG 5: PerfCounters, Memory perf data</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/12/windows-perf-counters-blog6.html">BLOG 6: PerfCounters, Disk/IO perf data</a><br />
<a href="http://toncigrgin.blogspot.hr/2015/12/windows-perf-counters-blog7.html">BLOG 7: PerfCounters, Network and Contention perf data</a><br />Tončihttp://www.blogger.com/profile/10650361408804728027noreply@blogger.com0tag:blogger.com,1999:blog-7850846940198540592.post-45265451264361208212015-10-27T08:41:00.000+01:002015-11-30T15:16:20.179+01:00Communication between Powershell script and JobsI had an interesting task of collecting and displaying some runtime metrics during the benchmark. Since data was clearly divided between OS and processes, I opted for separate background jobs in which case no task will stall the other. However, this means I had to establish communication between main script code and jobs which is tricky in Powershell (see this <a href="https://connect.microsoft.com/PowerShell/feedback/details/711951/interprocess-communication-or-communication-between-jobs">bug</a> for example). Immediately, I thought of using named pipes or files (as I do have small, ever changing set of data). Further investigation revealed 4 ways of accomplishing this:
<ol>
<li><b>Using files</b></li>
<ul>
<li><b>pros:</b></li>
<ul>
<li>Tried and tested; works everywhere.</li>
<li>Intuitive.</li>
<li>File management routines are mature.</li>
</ul>
<li><b>cons:</b></li>
<ul>
<li>Resource lock/leak possible.</li>
<li>Not really fit for big amount of data.</li>
<li>Local.</li>
</ul>
</ul>
<li><b>Using named pipes</b></li>
<ul>
<li><b>pros:</b></li>
<ul>
<li>Tried and tested; works everywhere.</li>
<li>Network.</li>
</ul>
<li><b>cons:</b></li>
<ul>
<li>Complicated to get right.</li>
<li>Blocking. Pipe has to get consumed before proceeding.</li>
<li>Async. Even more complicated to manage.</li>
<li>One should ensure unique pipe name.</li>
<li>No easy way to do cleanup if things go wrong.</li>
</ul>
</ul>
<li><b>Using custom (Engine)Events</b></li>
<ul>
<li><b>pros:</b></li>
<ul>
<li>"Proper" Windows way.</li>
<li>No "weird" code in script.</li>
</ul>
<li><b>cons:</b></li>
<ul>
<li>Hard to control; not fit for all the scenarios.</li>
</ul>
</ul>
<li><b>Using UDP/IP</b></li>
<ul>
<li><b>pros:</b></li>
<ul>
<li>Fast.</li>
<li>Fit for purpose (packet loss is acceptable).</li>
</ul>
<li><b>cons:</b></li>
<ul>
<li>Getting free ephemeral port.</li>
<li>Needs additional transport for initial Port negotiation.</li>
</ul>
</ul>
</ol>
<br><h3>Using files:</h3>
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 1px; width: 100%;"> <code style="color: black; word-wrap: break-word;">
$job1 = Start-Job -ScriptBlock {
...
#Permanent value.
$job1out = (dir Env:\TEMP).Value + '\job1out.txt'
$job1in = (dir Env:\TEMP).Value + '\job1in.txt'
if (Test-Path $job1in) { #Ready to go. }
do {
#Do the work, pipe output to file:
...
Select-Object -First 20 | FT * -Auto 1> $job1out
}
$job2 = Start-Job -ScriptBlock {
#Permanent values.
$job2out = (dir Env:\TEMP).Value + '\job2out.txt'
$job2in = (dir Env:\TEMP).Value + '\job2in.txt'
if (Test-Path $job2in) { #Ready to go. }
do {
#Do the work
...
#Utilize fast FS classes to output the result:
$stream = [System.IO.StreamWriter] ($job2out)
$stream.WriteLine("")
$stream.close()
... }
}
#Consolidate in main script code:
$job1out = (dir Env:\TEMP).Value + '\job1out.txt'
$job2out = (dir Env:\TEMP).Value + '\job2out.txt'
do {
$l = Get-Content $job1out -ErrorAction SilentlyContinue
$tc = Get-Item $job1out -ErrorAction SilentlyContinue
#Make sure to display only fresh data.
if (($l) -and ($tc) -and ($tc.LastWriteTime -gt $lastTimeHeader)) {
$lastTimeHeader = Get-Date
} else { $l = $null }
$l1 = Get-Content $job2out -ErrorAction SilentlyContinue
$tc = Get-Item $job2out -ErrorAction SilentlyContinue
#Make sure to display only fresh data.
if (($l1) -and ($tc) -and ($tc.LastWriteTime -gt $lastTimeProc)) {
$lastTimeProc = Get-Date
} else { $l1 = $null }
...
}
#Do the cleanup:
Stop-Job -Job $job1
$null = Receive-Job $job1
Remove-Job $job1
Stop-Job -Job $job2
$null = Receive-Job $job2
Remove-Job $job2
</code>
</pre>
<small><strong>Gotcha:</strong> Background job is not aware of session settings (say, $PSScriptRoot) thus you need to utilize machine-wide variables, such as Env:TEMP.</small><br><br>
<h3>Using named pipes (unfit for my particular use-case):</h3>
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 1px; width: 100%;"> <code style="color: black; word-wrap: break-word;">
#Create 2 pipe servers; one for each background job.
$pipeH = new-object System.IO.Pipes.NamedPipeServerStream 'pipe1','In', 1, "Message"
$srH = new-object System.IO.StreamReader $pipeH
#--2nd pipe has 2-way communication
$pipeT = new-object System.IO.Pipes.NamedPipeServerStream 'pipe2','InOut', 1, "Message"
$srT = new-object System.IO.StreamReader $pipeT
$swT = new-object System.IO.StreamWriter $pipeT
#--
#Start Job1:
$job1 = Start-Job -ScriptBlock {
...
#Permanent value.
$pipeH1 = new-object System.IO.Pipes.NamedPipeClientStream '.', 'pipe1','Out'
$pipeH1.Connect()
$swH = new-object System.IO.StreamWriter $pipeH1
$swH.AutoFlush = $true
do {
#Do the work, pipe output to StreamWriter:
...
$swH.WriteLine($ln0)
$swH.WriteLine("")
}
#In main code, wait for traffic to start:
$pipeH.WaitForConnection()
cls
#Say we read 11 lines.
$l = $srH.ReadLine()
if ($l -ne $null) {
$l
for ($i = 1; $i -le 10; $i++)
{
$l = $srH.ReadLine()
if ($l -ne $null) {
$l
}
}
}
#Start Job2:
$job2 = Start-Job -ScriptBlock {
#Permanent values.
$pipeT = new-object System.IO.Pipes.NamedPipeClientStream '.', 'pipe2','InOut'
$pipeT.Connect()
$srT = new-object System.IO.StreamReader $pipeT
$srT.AutoFlush = $true
$swT = new-object System.IO.StreamWriter $pipeT
do {
#Do the work, pipe output to StreamWriter:
...
Select-Object -First 20 | FT * -Auto 1> $swT
...
#Check communication:
if (($tmp = $srT.ReadLine()) -ne $null) {...}
}
}
#Finish setting things up:
$swT.AutoFlush = $true
$pipeT.WaitForConnection()
#Say we read 27 lines from 2nd job.
$l = $srT.ReadLine()
if ($l -ne $null) {
$l
for ($i = 1; $i -le 26; $i++)
{
$l = $srT.ReadLine()
if ($l -ne $null) {
$l
}
}
}
#And we send some info to 2nd job:
$swT.WriteLine('Something')
#You can read in loop too but make sure to allow for flow control between processes:
while (($tmp= $srT.ReadLine()) -ne 'something')
{
...
}
#Do the cleanup:
Stop-Job -Job $job1
$null = Receive-Job $job1
Remove-Job $job1
Stop-Job -Job $job2
$null = Receive-Job $job2
Remove-Job $job2
$pipeH.Close()
$pipeH.Dispose()
$srH.Close()
$srH.Dispose()
$pipeT.Close()
$pipeT.Dispose()
$srT.Close()
$srT.Dispose()
$swT.Close()
$swT.Dispose()
</code>
</pre>
<small><strong>Gotcha:</strong> NamedPipeServer/ClientStream have many constructors. Please check <a href="https://msdn.microsoft.com/en-us/library/system.io.pipes.namedpipeserverstream%28v=vs.110%29.aspx">Server</a> and <a href="https://msdn.microsoft.com/en-us/library/system.io.pipes.namedpipeclientstream%28v=vs.110%29.aspx">Client</a> documentation.</small><br>
<small>Good read on subject:<a href="https://gbegerow.wordpress.com/2012/04/09/interprocess-communication-in-powershell/">Georg Begerow</a>, <a href="http://blogs.msdn.com/b/bclteam/archive/2006/12/07/introducing-pipes-justin-van-patten.aspx">MSDN</a></small><br>
<small><strong>Gotcha:</strong> You can work without streams completely (ie. just with pipe object):</small>
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 1px; width: 100%;"> <code style="color: black; word-wrap: break-word;">
$enc = [system.Text.Encoding]::Default
$msg = $enc.GetBytes("Message");
$pipeSort.Write($msg, 0, $msg.Length)
$pipeSort.WaitForPipeDrain()
$pipeSort.Flush()
--
do
{
$SortBy += $pipeT.ReadByte().ToChar()
}
while (!($pipeT.IsMessageComplete))
$pipeT.Flush()
</code>
</pre>
<small><strong>Gotcha:</strong> I have found no way for this to work in my particular case (Async is out of question). Ie. Read/Peek... is always blocking thus preventing repeated updates. I guess this could be worked around by disposing and re-creating the client pipe but that's suboptimal.</small><br><br>
<h3>Using custom (Engine)Events:</h3>
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 1px; width: 100%;"> <code style="color: black; word-wrap: break-word;">
#Start Job1:
$job1 = Start-Job -ScriptBlock {
...
Register-EngineEvent -SourceIdentifier Job1Message -Forward
do {
#Do the work, raise Event:
...
$null = New-Event -SourceIdentifier Job1Message -MessageData $your_result
}
}
#Start Job2:
$job2 = Start-Job -ScriptBlock {
Register-EngineEvent -SourceIdentifier Job2Message -Forward
do {
#Do the work, raise Event:
...
$null = New-Event -SourceIdentifier Job2Message -MessageData $your_result
}
}
#In main code, I want to process results synchronously,
#thus a bit weird approach instead of just using -Action {} scriptblock:
do {
$OldErrorActionPreference = $ErrorActionPreference
$ErrorActionPreference = "SilentlyContinue"
$tmp = ''
$tmp = (Get-Event -SourceIdentifier Job1Message).MessageData | Select -Last 1
if ($tmp.Length) {
#Do work with $tmp
}
$tmp = ''
$tmp = (Get-Event -SourceIdentifier Job2Message).MessageData | Select -Last 1
if ($tmp.Length) {
#Do work with $tmp
}
$ErrorActionPreference = $OldErrorActionPreference
}
#and the cleanup:
$OldErrorActionPreference = $ErrorActionPreference
$ErrorActionPreference = "SilentlyContinue"
Remove-Event -SourceIdentifier "Job1Message"
Remove-Event -SourceIdentifier "Job2Message"
$ErrorActionPreference = $OldErrorActionPreference
$job1 | Stop-Job -PassThru| Remove-Job
$job2 | Stop-Job -PassThru| Remove-Job
</code>
</pre>
<small><strong>Gotcha:</strong> Unfortunately, I was unable to make Event propagate from main script to background job using same approach which makes me think there are more things to learn here and/or there is a bug in PS v3 regarding events processing.<br>
<strong>Gotcha:</strong> <samp>Register-EngineEvent ... -Action {}</samp> will start another (child)background job which will run in different runspace thus the options for communication are running scarce.</small><br><br>
<h3>Using UDP/IP:</h3>
Based on <a href="http://powershell.com/cs/blogs/tips/archive/2012/05/09/communicating-between-multiple-powershells-via-udp.aspx">PowerTip.</a>
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 1px; width: 100%;"> <code style="color: black; word-wrap: break-word;">
#To be executed in each script/job:
function Send-Text($Text='Sample Text', $Port=2500) {
$endpoint = New-Object System.Net.IPEndPoint ([IPAddress]::Loopback,$Port)
$udpclient= New-Object System.Net.Sockets.UdpClient
$bytes=[Text.Encoding]::ASCII.GetBytes($Text)
$bytesSent=$udpclient.Send($bytes,$bytes.length,$endpoint)
$udpclient.Close()
}
function Start-Listen($Port=2500) {
$endpoint = New-Object System.Net.IPEndPoint ([IPAddress]::Any,$Port)
$udpclient= New-Object System.Net.Sockets.UdpClient $Port
$content=$udpclient.Receive([ref]$endpoint)
[Text.Encoding]::ASCII.GetString($content)
}
#To determine if Port is free:
[System.Net.NetworkInformation.IPGlobalProperties]::GetIPGlobalProperties() | %{$_.GetActiveTcpListeners() } |
Select Port -Unique | Where {$_.Port -eq your_port}
#or
function IsLocalPortListening([int16] $LPort)
{
<#
.SYNOPSIS
Method to check if local port is available. This is used to determine free
port for deployment.
#>
Try
{
$connection = (New-Object Net.Sockets.TcpClient)
$connection.Connect("127.0.0.1",$LPort)
$connection.Close()
$connection = $null
return "Listening"
} Catch {
$connection.Close()
$connection = $null
return "Not listening"
}
}
</code>
</pre>
<small><strong>Gotcha:</strong> This approach is unfit for my use-case since I would need another delivery mechanism to establish Port to communicate on.</small>
<br><br>
<h3>Conclusion:</h3>
Given the task of communicating results from background jobs to main script, "writing to files" approach worked the best. Little overhead, single mechanism, common cmdlets and so on.<br>
Events approach appear to work most smoothly but for the fact that I was not able to get job to process event from main script code. Since there were no subscribers to the event, I am also concerned about the quantity of Events generated (2/sec).<br>
Named pipes proved too much for me and I never got them to work as expected while UDP/IP would require another delivery mechanism to sort out initial Port settings.
<br><br><br>Happy coding!Tončihttp://www.blogger.com/profile/10650361408804728027noreply@blogger.com0tag:blogger.com,1999:blog-7850846940198540592.post-91248449312122416302015-10-22T16:27:00.002+02:002015-11-30T15:10:34.737+01:00Regular checks before running Powershell script + Write-Error replacementOn occasion, one will produce the script that will not work in _ISE or on some particular version of Powershell. So, before allowing script to run, I always do several checks depending on the task at hand. Here's the code:<br />
<br />
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 1px; width: 100%;"> <code style="color: black; word-wrap: break-word;">
#region Check
if ($host.Name -ne 'ConsoleHost')
{
#Running in ISE
$host.UI.WriteErrorLine("`t Script can not be run in _ISE. Exiting.")
Exit 1
}
if (($PSVersionTable).PSVersion.Major -lt 3) {
$host.UI.WriteErrorLine("`t Script can not be run in PS 2. Exiting.")
Exit 2
}
$clrV =
((Get-ChildItem 'HKLM:\SOFTWARE\Microsoft\NET Framework Setup\NDP' -recurse |
Get-ItemProperty -name Version,Release -EA 0 |
Where { $_.PSChildName -match '^(?!S)\p{L}'} |
Select Version | Sort Version -Desc | Select -First 1).Version).Split('.')[0]
if ( $clrV -lt 4) {
$host.UI.WriteErrorLine("`t Script can not be run in .NET v"+$clrV+". Exiting.")
Exit 3
}
Set-StrictMode -Version Latest
Set-PSDebug -strict
#endregion
</code>
</pre>
<br />
<b>Explanation:</b><br />
<ul>
<li>Regions are great for increased readability, especially when script is big.</li>
<li>There are differences between Powershell Console and ISE (<a href="http://blogs.msdn.com/b/powershell/archive/2009/04/17/differences-between-the-ise-and-powershell-console.aspx">see points 3 and 4 for example</a>):<pre style="font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace;"><samp>PS C:\Users\user> $host.Name
ConsoleHost
PS:ISE [BOX]> $Host.Name
Windows PowerShell ISE Host</samp></pre>
</li>
<li>I have PS v3 on my laptop and PS v4 on my Labs servers thus not coding (or testing) for older versions.</li>
<li>Getting the .NET version could have been much simpler if it wasn't for the fact that <samp>[environment]::version</samp> will get deprecated soon:
<pre style="font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace;"><samp>PS C:\Users\user> [environment]::version
Major Minor Build Revision
----- ----- ----- --------
4 0 30319 34209</samp></pre>
</li>
<li><a href="https://technet.microsoft.com/en-us/library/hh849697.aspx">Set-PSDebug -strict</a> is there so that engine can throw an exception if a variable is referenced before being assigned a value.</li>
<li>Using $host.UI.WriteErrorLine produces much cleaner output, imo, than Write-Error.</li>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-ZZOIfqu1LaEqTMAMlWlQKWhTpJZEZi9Fwo5nGryLX8_rLhKaFmGyFplbU1iTCSg8O-1Bqotrkoo6-4_cTUkPAhAjO1UPqEsDS9a1mO0tn3gA3k4tNBLH7KLKFnKJWUb0rnPWUAurEWnt/s1600/Write-Error.PNG" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-ZZOIfqu1LaEqTMAMlWlQKWhTpJZEZi9Fwo5nGryLX8_rLhKaFmGyFplbU1iTCSg8O-1Bqotrkoo6-4_cTUkPAhAjO1UPqEsDS9a1mO0tn3gA3k4tNBLH7KLKFnKJWUb0rnPWUAurEWnt/s1600/Write-Error.PNG" /></a></div>
</ul>
<div class="separator" style="clear: both; text-align: left;">
<br />
Happy coding!</div>Tončihttp://www.blogger.com/profile/10650361408804728027noreply@blogger.com0tag:blogger.com,1999:blog-7850846940198540592.post-23121418848256775552015-10-19T12:14:00.000+02:002015-11-30T15:02:35.284+01:00Handling keyboard input and CTRL+C in Powershell without pausingRecently, I had a requirement to update my script console output depending on user key-press. Since calculation was done by background threads, I also wanted to prevent CTRL+C from stopping the script without proper cleanup.<br>
However, looks like seamless handling of key-press along with handling special sequences is not available in Powershell. This was to be expected given that [system.console]::readkey is designed for accepting the user input which is mostly answers to flow control questions. <a href="https://technet.microsoft.com/en-us/library/hh849967.aspx?f=255&MSPPError=-2147217396">Register-EngineEvent</a> does not help either.<br>
<br>
Anyway, let's tackle both problems one by one:<br>
<i>1) Disable CTRL+C from stopping the script:</i>
<br>
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 1px; width: 100%;"> <code style="color: black; word-wrap: normal;">
[console]::TreatControlCAsInput = $true
</code>
</pre>
Note that CTRL+BREAK will end the session entirely and do the proper cleanup thus it's not a problem.<br>
<br>
<i>2) Handle the user key-press:</i>
<br>
<pre style="background-color: #eeeeee; border: 1px dashed #999999; color: black; font-family: Andale Mono, Lucida Console, Monaco, fixed, monospace; font-size: 12px; line-height: 14px; overflow: auto; padding: 1px; width: 100%;"> <code style="color: black; word-wrap: break-word;">
if ($Host.UI.RawUI.KeyAvailable) { #Make sure there is something to handle.
#$k = [system.console]::readkey($true) -This will wait for ENTER after any key-press, thus unacceptable.
$k = $Host.UI.RawUI.ReadKey("AllowCtrlC,IncludeKeyDown,IncludeKeyUp,NoEcho").Character
if ("p" -eq $k) {
#Update the displaying
$HOST.UI.RawUI.Flushinputbuffer() #Flush the key buffer
} else {
if ("r" -eq $k) {
#Update the displaying
$HOST.UI.RawUI.Flushinputbuffer()
} else {
if ("m" -eq $k) {
#Update the displaying
$HOST.UI.RawUI.Flushinputbuffer()
} else {
if ("n" -eq $k) {
#Update the displaying
$HOST.UI.RawUI.Flushinputbuffer()
} else {
if ($k -eq "q") { #my wish was to map CTRL+C here but no way.
#Do the cleanup
Write-Host "Exiting...." -Background DarkRed
Stop-Job -Job $j
$null = Receive-Job $j
Remove-Job $j
...
break;
}
}
}
}
}
}
</code>
</pre>
The problem with proper solution is that one has to register his own functions and callbacks (<a href="https://msdn.microsoft.com/en-us/library/system.console.cancelkeypress.aspx">system.console.cancelkeypress</a>, <a href="https://msdn.microsoft.com/en-us/library/system.consolecanceleventhandler.aspx">system.console.canceleventhandler</a>, <a href="https://msdn.microsoft.com/en-us/library/ms686016.aspx">set application-defined HandlerRoutine</a>, <a href="https://msdn.microsoft.com/en-us/library/ms685049.aspx">install a control handler</a> ...) which was just too much for the scope of my task especially since the library implementing desired behavior exists (<a href="https://pseventing.codeplex.com/">PSEventing</a>).<br>
<br>
I do feel PowerShell should address this some time in the future through Register-Event cmdlets.Tončihttp://www.blogger.com/profile/10650361408804728027noreply@blogger.com0Samobor, Croatia45.771993380528734 15.69946289062545.417510880528731 15.054015890625 46.126475880528737 16.344909890625