System logs are some of the most important data you can keep on your dedicated server or VPS. Chances are if something is going to go wrong, it will do it when you aren’t logged in or able to look at it live. This leaves your system log files as the only way to trace the cause of the problem and, more importantly, discover how to prevent it from occurring again. One distinct shortfall with the log files, though, is that they only log what the system is doing, not how it is performing.
This is where sar comes in. It keeps a history of the server’s performance information, similar to what you can see live using the top command, and allows you to look back over time to see how the server was performing at any given time. Installation is easy on most Linux distributions as it is part of the sysstat package which provides a number of tools for system performance monitoring, and is included in the repositories for most distributions. For example, installation on Debian and Ubuntu is as follows:
sudo apt-get update
sudo apt-get install sysstat
For Red Hat and CentOS Linux distributions you can install sysstat with the following command:
yum install sysstat
Sysstat comes with a configuration file that you will need to modify in order to meet your requirements. The file is kept in different places dependent upon the Linux distribution you are using, so to edit the file on Debian and Ubuntu:
For Red Hat and CentOS distributions you can edit the config file as follows:
Note that while I’ve used nano as the text editor in my example, other text editors are available.
Once you have the configuration file open you’ll notice there’s not a lot to configure; the main two lines we are interested in are HISTORY and COMPRESSAFTER. The HISTORY line tells sar how many days of logs to keep – Debian and Ubuntu both default to 7 days whilst CentOS and Red Hat sit at 28. What you need to set it to is unfortunately one of those “how long is a piece of string?” questions, and it really depends on you and your system how far back you may be checking the data that is logged. Personally, I find the 7 days is a pretty good measure for most systems as I’ll often be investigating a problem well within that period. The COMPRESSAFTER line tells sar after how many days to compress the log files, which is useful if you are logging for a long period of time and need to conserve space on your server, something that is likely more important on a VPS than a dedicated server. You’ll note that the default configuration has COMPRESSAFTER set to longer than sar is configured to keep a HISTORY for, and again this is very much a personal needs setting.
With those files configured sar is ready for use, although Debian and Ubuntu users do have one more step, which is to edit the /etc/default/sysstat file to enable the logging as by default it is disabled:
Next, the ENABLED line should be changed to look like this:
Save and exit the file, and the logging will begin.
The logging itself is performed through a cron script called /etc/cron.d/sysstat which will run every ten minutes to collect information about the system, then place the log files in the /var/log/sysstat or /var/log/sa directory.
On its own the sar command will display the last hour’s worth of CPU usage information, but you can use flags to change the displayed output some of which are as follows:
-r Display RAM statistics.
-b Display disk IO statistics.
-s Start time of when to display statistics in hh:mm:ss format.
-e End time of when to display statistics in hh:mm:ss format.
-f Specify a log file to be scanned, note that sar stores its logs in a log file for each day within it’s logging directory.
-n DEV Display the network information logs for the network devices.
-P ALL Display CPU statistics on a per-processor basis, handy with multi-core systems.
-S Show SWAP space usage statistics.
-q Show system queue and load statistics.
There are many more flags that can be used and all of these are explained in plenty of detail in the man pages, And I’ve cherry-picked the ones that I find are most useful for this article. Something to note is that sar will default to using the current day’s log file when displaying information, so if you want to view the logs from a different day then you will need to use the -f flag with the full path to the log file for the day that you wish to look at. I recommend having a bit of a play to familiarize yourself with sar’s workings and also having a read through the man page.