Examples of using SAR command for system monitoring in Linux

System Activity Reporter is an important tool that helps system administrators to get an overview of the server machine with status of different important metrics at different points of time.

If suppose you are having an issue with the system currently, Like some of your customers are unable to list some data from the database. The first thing that most of the Linux system administrators do is to recall the same issue when it previously occurred, and If you remember the day of its previous occurrence then you can easily compare the internal system statistics with the current statistics.

SAR is very much helpful in doing exactly that.

The first thing that we need to do is check and confirm whether you have SAR utility installed on the machine. Which can be checked by listing all rpm's and finding for this utility.

SAR is one of the utility inside sysstat. You can easily download and install it in your machine very easily through YUM. (But yeah dont worry because most of the distribution comes prepacked with sysstat tool).

[root@myvm1 ~]# yum install sysstat

Yeah but make it sure that you have epel,rpmformge repository enabled for installing. Otherwise your distribution DVD will be a nice place to look for the package.

SAR (System Activity Reporter) will Give Information about the following things:

1. System Buffer activity
2. Information about system calls
3. Block device information
4. Overall paging information
5. Semaphore and memory allocation information
6. CPU utilization and process report

The main thing that we need to understand regarding SAR is that, everything is done using a cron. By default in many Linux distribution you will have a file named /etc/cron.d/sysstat.

Lets see how really SAR works.

If we start thinking about system monitoring, then the tool must have each and every data about the system's different aspects and must cover all time intervals. Which means a monitoring system must be able to provide the statistics of the machine for a given time.

There is no way, other than taking all the metrics and statistics of the machine at a definite time interval. Reducing the time interval for collecting the statistics will increase the amount of detailed statistics we have(because we will be having more data about the system).

SAR does exactly that. sar takes the statistics of different aspects of the machine at a definite time interval. So SAR runs through CRON.

[root@myvm ~]# cat /etc/cron.d/sysstat
# run system activity accounting tool every 10 minutes
*/10 * * * * root /usr/lib64/sa/sa1 1 1
# generate a daily summary of process accounting at 23:53
53 23 * * * root /usr/lib64/sa/sa2 -A

• So it can be seen from the above cron file for SAR that its running "sa1" script located at "/usr/lib64/sa/" at every 10 minutes
• And is also running a script /usr/lib64/sa/sa2 at the end of the day at around 23.53

So the first cron entry for SAR(/usr/lib64/sa/sa1) will run every 10 minutes which inturn will call the sadc utility to collect system stats and store it in a binary file (one file for a day)

And the second cron entry will dump all the contents of that binary file into another text file, and purges data older than a particular number of days, Normally 7 days by default(which is mentioned in the following file),

[root@archive ~]# cat /etc/sysconfig/sysstat
# How long to keep log files (days), maximum is a month
HISTORY=7

So you can modify that HISTORY entry easily by editing the file.

So although the system statistics is being collected every 10 minutes through cron(modify the cron to run every 1 minute for more accurate information) If you want to see the stats, then you need to run the command as below.

The simple sar command output is as shown below.

12:00:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
12:01:01 AM     all     73.28      0.00      1.25      0.00      0.00     25.47
12:02:01 AM     all      7.83      0.00      0.44      0.00      0.00     91.73
12:03:01 AM     all     61.65      0.00      0.70      0.00      0.00     37.66
12:04:01 AM     all     57.85      0.00      0.82      0.00      0.00     41.34
12:05:01 AM     all      4.25      0.00      0.41      0.00      0.00     95.34
12:06:01 AM     all      4.20      0.00      0.22      0.00      0.00     95.58
12:07:01 AM     all      5.05      0.00      0.33      0.00      0.00     94.63
12:08:01 AM     all      4.76      0.00      0.06      0.00      0.00     95.18
12:09:01 AM     all     37.57      0.00      0.37      0.00      0.00     62.05
12:10:01 AM     all     70.04      0.00      0.80      0.00      0.00     29.16
12:11:01 AM     all      5.03      0.00      0.12      0.00      0.00     94.84

It can be seen from the output that its reporting me the output of the collected stats for every minute(which means i have my cron at 1 minute interval), and will show the details of the whole day(or will show details collected till when you typed the command).

Understanding the output of SAR command

%user: This shows the total time that the processor is spending on different process YCX5UKN5ZKEJ

%sys: this shows the percentage of time spend by the processor for operating system tasks(because the previous user shows the time spend for user end process)

%iowait: the name iowait itself suggests that its the time spend by processor waiting for devices(input and output)

%nice: Most of you guys must be knowing that a user can change the priority of a process in linux by changing the nice value in Linux. This table shows the time spend by CPU for process whose nice value has been changed.

%steal:  This column shows the amount to time spend by a CPU (which is virtualized), for resources from the physical CPU

%idle: This suggests the idle time spend by the processor.

By default sar stores all of its data under /var/log/sa/ and a days are named as shown below.

s01 - for first day of the month

s02-for second day of the month

s03,s04..and so on.

-d option in SAR command

This -d option can be used to report each and every activity related to different devices attached to the system(block devices). A typical output of the sar command with -d option is shown below.

12:00:01 AM       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
12:01:01 AM    dev3-0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Cards Typically
12:01:01 AM   dev3-64      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:01:01 AM    dev8-0     55.62      9.98   8317.87    149.72     13.68    245.96      2.61     14.52
12:01:01 AM   dev8-16      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:02:01 AM    dev3-0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:02:01 AM   dev3-64      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:02:01 AM    dev8-0      1.55      0.00     35.29     22.70      0.01      7.26      1.86      0.29
12:02:01 AM   dev8-16      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:03:01 AM    dev3-0      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
12:03:01 AM   dev3-64      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

DEV: this column names devices on the machine, according to major and minor numbers of a Linux block device. You can check this by doing an ls -l in /dev directory. as shown below.

brw-r-----  1 root disk   8,    0 Nov 16 16:29 sda
brw-r-----  1 root disk   8,    1 Nov 16 16:29 sda1
brw-r-----  1 root disk   8,    2 Nov 16 16:29 sda2

in the above shown "ls -l" outut for "sda", major number is "8",and minor number is "0"...So you can easily identify the disk mentioned in the sar command output.

tps: tps stands for transfer per second, so it shows the transfer per second to that particular device

rd_sec/s: this shows you the total number of sectors on that device which is being read

wr_sec/s: if rd_sec/s is sectors being read per second then obviously wr_sec is sectors being written per second.

avgrq-sz: this column shows the average.

await: this shows the total number of time that the processor waited for requests regarding IO

%util: this column shows the usage of cpu in percentage when the request was generated

Show Memory usage in SAR command

the -r option available in sar command is very much useful. it shows the memory,swap,cached memory etc at every interval or required time interval.

02:20:01 AM kbmemfree kbmemused  %memused kbbuffers  kbcached kbswpfree kbswpused  %swpused  kbswpcad
02:30:01 AM    609500   1487652     70.94    242420    777560   1075980       364      0.03       360
02:40:01 AM    609500   1487652     70.94    242424    777568   1075980       364      0.03       360
02:50:01 AM    609500   1487652     70.94    242424    777592   1075980       364      0.03       360
03:00:01 AM    608980   1488172     70.96    242424    777600   1075980       364      0.03       360
03:10:01 AM    608584   1488568     70.98    242424    777628   1075980       364      0.03       360
03:20:01 AM    608584   1488568     70.98    242424    777648   1075980       364      0.03       360

How to fetch metrics of a particular day using SAR in linux

[root@archive ~]# sar -f /var/log/sa/sa27
Linux 2.6.18-194.el5xen (archive.r)     11/27/2012
 
02:20:01 AM       CPU     %user     %nice   %system   %iowait    %steal     %idle
02:30:01 AM       all      2.58      0.00      0.70      1.12      0.05     95.55
02:40:01 AM       all      2.56      0.00      0.69      1.05      0.04     95.66
02:50:01 AM       all      2.64      0.00      0.65      1.15      0.05     95.50
03:00:01 AM       all      3.27      0.00      0.71      1.12      0.04     94.86

How to fetch SAR metrics for a specific time on a particular date

this can be achieved by passing another argument as shown below.

[root@archive ~]# sar -f /var/log/sa/sa27 -s 02:20:00 -e 03:20:00
Linux 2.6.18-194.el5xen (archive.r)     11/27/2012
 
02:20:01 AM       CPU     %user     %nice   %system   %iowait    %steal     %idle
02:30:01 AM       all      2.58      0.00      0.70      1.12      0.05     95.55
02:40:01 AM       all      2.56      0.00      0.69      1.05      0.04     95.66
02:50:01 AM       all      2.64      0.00      0.65      1.15      0.05     95.50
03:00:01 AM       all      3.27      0.00      0.71      1.12      0.04     94.86
03:10:01 AM       all      2.72      1.06      0.75      1.09      0.04     94.33
Average:          all      2.76      0.21      0.70      1.11      0.04     95.18

Show network statistics using sar command

sar command even shows network statistics. This can be done by using the -n DEV option in sar command.

[root@archive ~]# sar -n DEV
Linux 2.6.18-194.el5xen (archive.r)     11/27/2012
 
02:20:01 AM     IFACE   rxpck/s   txpck/s   rxbyt/s   txbyt/s   rxcmp/s   txcmp/s  rxmcst/s
02:30:01 AM        lo      0.01      0.01      0.77      0.77      0.00      0.00      0.00
02:30:01 AM      eth0     12.30      0.12   1285.06     27.59      0.00      0.00      0.00
02:30:01 AM      eth1     14.45      0.00   1399.34      0.00      0.00      0.00      0.00
02:30:01 AM      sit0      0.00      0.00      0.00      0.00      0.00      0.00      0.00
02:40:01 AM        lo      0.01      0.01      0.77      0.77      0.00      0.00      0.00
02:40:01 AM      eth0     10.65      0.12   1139.38     27.00      0.00      0.00      0.00
02:40:01 AM      eth1     13.96      0.00   1352.87      0.00      0.00      0.00      0.00

IFACE: stands for the nic card interface name

rxpck/s: this shows the total packets received per second

txpck/s:transmitted packets per second

rxcmp/s: compressed packets received

txcmp/s: compressed packets transmitted

rxmcst/s: packets multicasted per second.

Some other Metrics that can be determined using sar

-y option in sar:  can be used to determine tty details

-X option in sar to get details of a particular process. You need to pass pid as an argument to this option.

-n SOCK option in sar: this option will report all socket details.

There are couple of more options available in SAR.You can refer for the complete list of arguments in man pages..Hope this introductory article was helpful in getting started and using SAR..