Linux Performance Tracking

When it comes to performance tracking on a Linux server, there are tons of useful utilities; enough to where it becomes difficult to know what to use when. The purpose of this article is to offer advice and tips on how to track various performance metrics.

Top is an ubiquitous utility in the Unix admins arsenal. It is provided by the procps package and is available by default on just about every major Linux distribution. It displays process info in real-time, both about the system as a whole and about individual processes. For this particular guide, we are mostly concerned with a particular line (or lines if you have a multi-core/processor system), reading:

Cpu(s):  4.2% us,  0.6% sy,  0.0% ni, 94.9% id,  0.9% wa,  0.0% hi, 0.0% si

? us ! Percent cpu being used by userland code ? sy ! Percent cpu being used by kernelspace code ? ni ! Like “us” but related to “niced” processes ? id ! Idle percent ? wa ! CPU is idle because it waits for IO to complete ? hi ! Interrupts generated by hardware ? si ! Interrupts generated by software

Together all of these figures add up to 100%. Generally you don't need to be concerned with all of these metrics, however there are a few that are very important. Any activity that generates CPU activity will reduce the id % and depending on how the load was generated, increase one of the others. In most cases these are either us or wa. Depending on your hardware, it is normal to see anywhere from 1-100% us usage. wa, however, should ideally be 0%, or as near to 0% as possible.

As the value of wa increases, there is more and more CPU time being spent waiting for system I/O to finish (harddrives/disks). Values of even as much as 0.5% have been known to cause slowing on older SATA and PATA drives. This number will almost always be >1% when your server is using swap space (more on that later). Next we'll cover how to get more information about what in particular might cause undesirable wa percentages.