Linux – Disk Utilization using du

Background

End of Year, Beginning of a new year, and end of Month are often good times to stress a computer system.

We have had our systems stressed these last couple of months.

Often it is running low on disk space, a lot of higher than normal database backup and log backups.

 

MS Windows

Microsoft Tools

On Windows I will often use Microsoft’s own Computer Management to review disks logical size and utilization.

Or File Explorer and filter by size looking for huge ones.

Space Sniffer

Or better still SpaceSniffer.

I spoke about SpaceSniffer here.

Linux

I know I am avoiding talking about Linux.

And, so let us talk about Linux a bit.

du command

Introduction

I suppose du means disk utilization.

Help

First let us get a quick overview of what du does and the options it offers.

Syntax


du --help

Output

Explanation

What does it do?

Summarize disk usage of each FILE, recursively for directories.

Options Tabulated

Option Meaning Usage
-a List files as well, not just directories Usage is du -a
-B Block Size -b 1 is default
-b 1024 means to divide the actual size by 1 K
-h Human readable  usage is du -h
–exclude=PATTERN Exclude files based on certain filename or extension du -a -B 1 -h –exclude=*.gz /dbbackup
 –max-depth  Our many folders down to go  du –max-depth=4 /dbbackup

 

Sample

Block Size ( -B)
Sample ( Block Size at 1 )

du -a -B 1 /dbbackup | grep "sql"

Output

Explanation
  1. /dbbackup/gregory/export.sql is 4096

 

Sample ( Block Size at 1024 )

du -a -B 1024 /dbbackup | grep "sql"

Output

Explanation
  1. /dbbackup/gregory/export.sql
    • was at 4096
    • But, now is 4
      • 4096/1024 => 4

 

Human Readable ( -h)
Sample ( Block Size at 1024 )

du -a -h *.sql

Output

Explanation
  1. /dbbackup/gregory/export.sql
    • was at 4096
    • But, now a more readable 4.0 K

 

exclude ( -exclude=filter)
Sample ( Exclude files already compressed via gzip )

du -a --exclude=*.gz

max-depth=N
Depth to show = 1

du --max-depth=1 -h /dbbackup

Output
 

Depth to show = 4

du --max-depth=4 -h /dbbackup

Output

Explanation
  1. In our first sample:
    • We progressed down only one folder tree from our asking node ( /dbbackup )
  2. In our second example:
    • We went down 4 levels from our root from /dbbackup

 

Use case scenario

Use case #1

Get Folder Size for our backup folder ( /dbbackup)

Baseline

Sample

du --max-depth=4 /dbbackup

Output

 

Silence Errors

There is an error for “permission denied“.

Let us silence it for now.

Sample

du --max-depth=4 /dbbackup 2>1 | grep -v 'permission denied'

Output

Explanation
  1. Using 2>1
    • We redirect errors such as permission denied to standard output
  2. Using grep -v
    • We filter out ‘permission denied’

 

Ordered Results

Let us order by size.

Sample

du --max-depth=4 /dbbackup 2>1 | grep -v 'permission denied' | sort -nr

Output

 

References

  1. StackOverflow
    • Exclude all permission denied messages from “du”
      Link

Linux – Disk Utilization [ du ]

Background

Let us quickly discuss using “Disk Utilization“, du, to list disk utilization on a Linux System.

BTW, we will be targeting folder level information.

Options

Tabulated

Option Option Short Form Option Long Form Sample
Human Readable -h –human-readable K – Kilobytes
M – Megabytes
G – Gigabytes
T – Terabytes
Total -c –total du –total
Summarize -s –summarize du -s
du –summarize

Sample

Human Readable

Command


du -h *  2>/dev/null

Output

Total

Objective

  1. List sizes for the following folders and at the tail end list aggregated size, as well
    • db2inst1/sqllib/acs
    • db2inst1/sqllib/db2tss

Command


du -c db2inst1/sqllib/acs db2inst1/sqllib/db2tss 2>/dev/null | more

Output

Summarize

Summarize for specific folders

Objective
  1. Summarize sized for the following folders
    • db2inst1/sqllib/acs
    • db2inst1/sqllib/db2tss
Command

du -s db2inst1/sqllib/acs db2inst1/sqllib/db2tss 2>/dev/null | more

Output

Summarize for all folders

Objective
  1. Summarize all sub-folders
Command

du -s * 2>/dev/null | more

Output

Summarize for all folders and Ordered

Objective
  1. Summarize all sub-folders
  2. And, order the results
Command

du -s * 2>/dev/null | sort -nr

Output

References

  1. Lifewire
    • How To Find How Much Disk Space A File Or Folder Uses In Linux
      Link
  2. Stack Exchange
    • Listing directories based on size from largest to smallest on single line
      Link

Linux – Finding Text in Log Files – Day 1

Background

Don’t really spend much time in the Linux World.

But, there are many Application Servers that sit on Linux Servers and rely on SQL Server on Windows.

 

Log Files

When error occurs a good Application Server write these errors in Log Files.

After a while needs develop to trend the type of DB Connectivity and more serious hard errors.

 

Sample Log Files

As we discussed in a recent post, we copied over SQL Server Error Log files over to Linux.

 

Text Search

grep

Let us use grep to perform simple Text Searches.

Scenario

Look for Text Matching Manufacturer

Script

grep -w -i -l -n  "Manufacturer" ERRORLOG

Output

Explanation
  1. No matches

 

Remediation

Attempted Workarounds

We tried the following workarounds

  1. Worked
    • Created a much simple text file and was able to grep against it
  2. Did Not Help
    • Modified copy over mode to binary and back to text
    • Command Issued
      • cat ERRORLOG | grep -i “manufacturer”

 

Worked

Code


cat ERRORLOG | tr -d '\000' | grep -i "manufacturer"

Output

Why does it work

  1. Cat
    • Outputs a file’s content
  2. tr – stands for translate
    • tr -d ‘\000’
      • -d means delete
      • \000 means null character

Crediting

What makes grep consider a file to be binary?
Link

When you are top 13% on stackexchange, you are one of the Big Boys and Girls.