Linux – Profiling System Resource Utilization – Day 1

Background

Since we are already talking about profiling Applications, let us quickly highlight some of the available tools for measuring Application Resource Utilization on a Linux box.

Terminology

  1. Memory
    • buffer
      • Amount of memory that is buffered. In simple words, buffers are used for caching of filesystem metadata (permissions, location, etc.) and tracking in-flight pages. ( Link )
    •  cache
      • Amount of memory that is cached. Cache contains data that has already been read from the disk and is kept in memory for possible future use, f.e a pdf file or web browser pages. ( Link )

Outline

  1. top ( Link )
  2. sar ( Link )
  3. free ( Link )
  4. iostat ( Link )

top

The “top” command lists the processes running on a Linux box.

top stands for “Table of Processes“.

Sample Reading

MySQL

Here are sample outputs from a MySQL Box.

Sample #1
Image

top_linux_mysql_20180812_0531PM.PNG

Output
  1. Our box has been up for 14 days
  2. CPU
    • User :- 0.2%
    • System :- 0.5%
    • Nice :- 0%
    • Idle :- 99.2%
  3. Memory
    • Mem: 1872716k total, 1598392k used, 274324k free, 204700k buffers
      • Total :- 1872716k ( 1872 MB / 1.82 GB )
      • Used :- 1598392k ( 1598 MB / 1.60 GB )
      • Free :- 274324k ( 274 MB )
  4. Swap
    • Swap: 1048572k total, 87176k used, 961396k free, 578420k cached
      • Total :-1048572K ( 1048.572 MB / 1.05 GB )
      • Used :- 87176k ( 87 MB  )
      • Free :- 961396k ( 961 MB )
  5. Processes
    • Process /MySQL
      • PID ( Process ID )
      • USER
        • MySQL
      • PR ( User Priority )
        • 20
      • NI ( Nice )
        • 0
      • VIRT ( Virtual Memory )
        • 1789 MB
          • Our database server, MYSQL, requested 1800 MB
      • RES ( Resident Memory )
        • 568 MB
          • Our database server, MYSQL, is using actual memory of 568 MB
      • SHR ( How much of Total Virtual Memory this process is sharing with others)
        • 6048
        • What is the measurement?
      • S ( Process State )
      • %CPU
      • %MEM
      • TIME+
      • COMMAND

Functionalities

Outline

  1. Order By

Order By

To review or change the ordering please press the Shift-O button.

The “Current Sort Field” window appears.

Image

top_sort_20180813_0316AM

Please review the available sort fields and press the corresponding character.

sar

Definition

The sar common stands for “System Activity Report“.

Options

Memory-rMemory free, usedsar -r

Objective Option Description Sample
Processor Cumulative -p Processor Cumulative sar -P ALL
Processor Detail -P Process All or Specific sar -P ALL
Processor Run Queue and Load Average -q Reports run queue and load average sar -q
Swap Space Utilization -S Swap space utilization sar -S
Storage I/O -b Storage IO Transactions ( Count & Bytes) sar -b
Storage I/O @ Block Device Level -d IO Transactions ( Count & Bytes) sar -d
Network -n Network sar -n DEV

Scenarios

Review current Days report.

Outline

Get Stats for current day.

Code

sar

Output

linux_db2_20180813_0338AM.PNG

Explanation
  1. CPU
    • All available CPU
  2. %user
    • User applications CPU Percentile usage
  3. %nice
    • CPU Percentile usage attributed to applications running under “nice” mode
  4. %System
    • % CPU by system applications ( Kernel, device drivers )
  5. %iowait
    • % CPU Idle waiting for I/O completion
    • References
      • Red Hat Enterprise Linux – What Is the Meaning of %iowait as Reported by Utilities Such as sar or top?
        Link

Review past reports

Outline

Get Stats for the 7th day of the current Month

Code

sar -p -f /var/log/sa/sa07

Output

linux_db2_20180812_0957PM

Review CPU Utilization full and individualized

Outline

Get Stats for current day.

Code

sar -P ALL 

Output

linux_db2_20180814_0341AM

Explanation
  1. CPU
    • All available CPU
  2. %user
    • User applications CPU Percentile usage
  3. %nice
    • CPU Percentile usage attributed to applications running under “nice” mode
  4. %System
    • % CPU by system applications ( Kernel, device drivers )
  5. %iowait
    • % CPU Idle waiting for I/O completion
    • References
      • Red Hat Enterprise Linux – What Is the Meaning of %iowait as Reported by Utilities Such as sar or top?
        Link

Review Memory Utilization

Outline

Get Stats for current day.

Code

sar -r 

Output

sar-r-20180814-0347AM.PNG

Explanation
  1. kbmemfree
    • Free Memory in KB
  2. kbmemused
    • Used Memory in KB
  3. %memused
    • Percentile ( %) of Memory Used
  4. kbbuffers
    • Buffers for disk writes
  5. kbcached
    • Used as cached for disk reads
  6. kbcommit
    • Amount of Memory needed for current workload.  Total Memory needed to ensure that we do not run out of Memory.
  7. %commit
    • % Total Memory that is actually in use per current workload

Review Swap Space Utilization

Outline

Get Swap Space for current day.

Code

sar -S 

Output

sar-S-20180814-0755AM.PNG

Explanation
  1. kbswpfree
    • Free Swap Space in KB
  2. kbswpused
    • Used Swap Space in KB
  3. %swpused
    • Percentile ( %) of Swap Space Used
  4. kbswpcad
    • Amount of cached swap memory in kilobytes. This is memory that once was swapped out, is swapped back in but still also is in the swap area (if memory is needed it doesn’t need to be swapped out again because it is already in the swap area. This saves I/O). ( Link )
  5. %swpcad
    • Percentage of cached swap memory in relation to the amount of used swap space. ( Link )

Review I/O Transfer Rates

Outline

Get I/O Transfer rates and read and writes blocks.

Code

sar -d -f /var/log/sa/sa07 

Output
sar-d_20180814_0924AM.PNG
Explanation
  1. DEV
  2.  tps
    • Total number of transfers per second that were issued to physical devices. A transfer is an I/O request to a physical device. Multiple logical requests can be combined into a single I/O request to the device. A transfer is of indeterminate size. ( Link )
  3. rd_sec/s
    • Number of read requests per second
  4. wr_sec/s
    • Number of writes per second
  5. avgrq-sz
    • The average size (in sectors) of the requests that were issued to the device. ( Link )
  6. avgqu-sz
    • Average Queue Size
    • The average queue length of the requests that were issued to the device. ( Link )
  7. await
    • The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them. ( Link )
  8. svctm
    • The average service time (in milliseconds) for I/O requests that were issued to the device. Warning! Do not trust this field any more. This field will be removed in a future sysstat version. Percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%. ( Link )
  9. %util
    • Percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this value is close to 100%. ( Link )
Outline

Get I/O Transfer rates and read and writes blocks and in print friendly mode.

  1. sar -dp
    • Option
      • d is disk
      • p is print friendly
    • Crediting
      • Community Home Servers and Operating Systems Operating System – Linux System Administration “sar -d” devices – how to map to /dev/sdNNN device…
        Link
Code

sar -dp -f /var/log/sa/sa07 

Output

sar-dp_20180814_1140AM.PNG

Review I/O Transfer Rates

Outline

Get I/O Transfer rates and read and writes blocks.

Code

sar -b 

Output
sar-b-20180814-0810AM.PNG
Explanation
  1. tps
    • Total number of transfers per second that were issued to physical devices. A transfer is an I/O request to a physical device. Multiple logical requests can be combined into a single I/O request to the device. A transfer is of indeterminate size. ( Link )
  2. rtps
    • Number of read requests per second
  3. wtps
    • Number of writes per second
  4. bread/s
    • Number of sectors read per second.  A sector is 512 bytes  ( Link )
  5. bwrtn/s
    • Number of sectors written per second.  A sector is 512 bytes ( Link )

Reports run queue and load average (sar -q)

Outline

This reports the run queue size and load average of last 1 minute, 5 minutes, and 15 minutes.

Code

sar -q 

Output

sar-q_linux_db2_20180814_1217PM.PNG

Explanation
  1. runq-sz :- Run Queue Size
  2. plist-sz :- Process List
  3. ldavg-1 :- Load Average Size taken 1 minute ago
  4. ldavg-5 :- Load Average Size taken 5 minutes ago
  5. ldavg-15 :- Load Average Size taken 15 minutes ago
  6. blocked :- Number of blocked tasks

Reports run queue and load average (sar -n)

Outline

This reports the run queue size and load average of last 1 minute, 5 minutes, and 15 minutes.

Code

sar -n DEV -f /var/log/sa/sa07 -s 18:00:00 -e 23:59:00 | grep 'IFACE\|eth0'

Output

sar-n-DEV-linux_db2_20180814_1250PM

Explanation
  1. IFACE
    • Interface
  2. rxpck/s
    • Receive packets
  3. txpck/s
    • Transmit packets
  4. rxkB/s
    • Receive Kb/s
  5. txkB/s
    • Transmit Kb/s
  6. rxcmp/s
  7. txcmp/s
  8. rxmcst/s


Free

Definition

Display amount of free and used memory in the system.

Outline

Review Memory usage.

Usage

Usage – Raw Data

Code

free

Output

free_linux_db2_20180814_04432AM.PNG

Explanation
  1. total
    • Total Memory
  2. used
    • Used Memory
  3. free
    • Free Memory
  4. shared
    • Shared Memory
  5. buffers
    • Memory allotted for writing
  6. cached
    • Memory allotted for reading

Usage – Human Readable

Code

free -h

Output

free_linux_db2_-h_20180814_0732AM.PNG

iostat

Definition

Reports IO Statistics.

Outline

Review IO Statistics on each device.

Usage

Usage – Raw Data

Code

iostat -N

Output

iostat_db2_linux_20180815_0506PM.PNG

 

Explanation
  1. Summary
    • Average CPU
      • User
        • 7.4%
      • System
        • 2.51%
  2. Detail
    • Device
      • data1-db2data–11
        • tps :- 226.42
        • Blk_read/s :- 23254.73
        • Blk_wrtn/s :- 347.39
        • Blk_read :- 35060914332
        • Blk_wrtn :- 523763058
      • data1-dbbackup–11
        • tps :- 796.31
        • Blk_read/s :- 5785.45
        • Blk_wrtn/s :- 6187.70
        • Blk_read :- 8722664698
        • Blk_wrtn :- 9329126392

References

  1. BooleanWorld
    • Top Command
      • A Guide to the Linux Top Command
        Link
  2. Lily Feng
    • Top Command
      • The difference among VIRT, RES, and SHR in top output
        Link
  3. nixCraft
    • How do I Find Out Linux CPU Utilization?
      Link
  4. Ramesh Natarajan
    • The Geek Stuff
      • 10 Useful Sar (Sysstat) Examples for UNIX / Linux Performance Monitoring
        Link
  5. Neeraj Bhatia
    • Linux Memory Utilization
      Link
  6. Lisenet
    • Tomas
      • Measure and Troubleshoot Linux Memory Resource Usage
        Link
  7. HowtoForge
    • Linux Free Command Explained for Beginners (6 Examples)
      Link
  8.  stackExchange
    • top
      • Top – what does Virtual memory size mean? …linux/ubuntu
        Link
      • Process ‘niceness’ vs. ‘priority’
        Link
  9. serverfault
    • top
      • Memory Usage Numbers In top/htop
        Link
  10. Quora
    • top
      • What is the difference between the NI and PR values in the top(1) command’s output? I know NI is the nice value, which ranges from -19 to 20, but what is the significance of PR value?
        Link

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s