Troubleshooting disk usage on a Linux server with ncdu

ncdu

I awoke early Sunday morning (December 4 2016) to find that this website was not reachable. I, and visitors too, was getting the dreaded (for a website owner) 500 error message.

I went to bed just after 2:00 a.m. that Sunday, so between that time and 7:00 a.m. when I woke up, something had gone wrong. I manage my own servers, so there’s no technical support to call. I had to do something, because server down means revenue will be down at the end of the month.

Based on experience, the first thing I did after logging into the server was check whether MySQL was running. It was not. A database-driven website has to have the database application up and running or there’ll be no connection. An attempt to restart MySQL failed, with an output that included a line about not being able to write to disk or something of that nature.

That pointed to a disk is full situation. Not good! Now I have to find out what application or applications chewed up whatever free disk space I had left.

That’s when I thought about the easiest tool to use when troubleshooting disk usage on a Linux server – ncdu.

If you’re not familiar with it, ncdu is an ncurses interface for du, the tool used for estimating file space usage on Linux distributions. It’s installed out of the box, but ncdu is not, so if you which to use it, first install it using one of the following commands.

# Install ncdu on Ubuntu- or Debian-based distributions

sudo apt install ncdu

# Install ncdu on Fedora or CentOS

sudo dnf install ncdu

#

So I had a problem (my website was down) that was the result of full disk space on the server. There were two solutions – one temporary (delete files not needed on the server) and another much deeper solution (find out what application was writing too much data to disk). The temporary solution was easy, and in about five minutes I had freed up almost 300 MB of disk space. That was more than enough to restart MySQL (and other services) and get the website back up.

Related Post:  How to install Linux Mint 12 KDE on a btrfs file system

That out of the way, I deployed ncdu to get at the root cause of the problem. The application can be run from any directory, including the root directory. The output will tell you the disk usage of each file and directory, with the ability to drill down into any listed directory.

So I began my troubleshooting journey by typing the following command:

# Run ncdu on a Linux distribution

ncdu /

#

The output, and the main interface of ncdu, is shown in Figure 1. It’s a very simple and easy application to use – the arrow keys, ENTER/Return key and a few other keys are all you need to navigate its interface. Just by looking at what was displayed on the screen, I knew right away where most of the disk space is being used. But what I had completely forgotten was I had assigned 4 GB of disk space to a swap file. Because DigitalOcean no longer recommends configuring swap on their servers, that’s 4 GB that I could reclaim. So installing and running ncdu has already paid dividends.

To drill down into a directory, you only have to selected it and press ENTER.

mcdu linux disk usage
Figure 1: Main interface of ncdu

Doing that on the /var directory further narrowed where most of the disk space is being used. What I saw on this screen surprised me – OSSEC is using up almost 14 GB of disk space. That’s too much, and hints at a misconfiguration of OSSEC. If OSSEC is new to you, it’s an host-based IDS. More about it here.

ncdu drill down
Figure 2: Open a selected directory on ncdu

By drilling much further down into the OSSEC directory, it was easy to see that OSSEC was the application that’s chewing up way too much disk space than it’s supposed to. So problem has been identified. I’ll have to do something about OSSEC within the next few days.

ncdu on linux
Figure 3: Drilling further down into a selected directory on ncdu

By drilling down into the directory holding the website’s data, I was able to find a few archive files that I could delete. These were files from last year that I didn’t need any more. One was a 1.2 GB archive that was neatly tucked away into in the bowels of WordPress. Now it’s gone.

ncdu
Figure 4: Still drilling further down into a selected directory on ncdu

So that’s how I spent the early hours of Sunday, December 4 (2016). It pays when you know what to do when trouble strikes. One final commentary on ncdu: Though it provides much needed information about each file and directory, it does not tell you the last access time of any resource, something you can get when calling du directly. For example, if you run du as given in the following command:

# Run du with the --time option

du -chs --time /var/www/* | sort -rn | head

#

The output should look something like the one below: The last time a file was accessed is useful when looking for signs of unauthorized access.

# Output of du with the --time option

8.0K	2015-07-13 11:13	/var/www/html
5.7G	2016-12-05 17:58	/var/www/sites
5.7G	2016-12-05 17:58	total

#

Hope you find this little piece about ncdu useful. And I hope your Sunday didn’t start out like mine.

Related Post:  How to install OpenSSH on Windows 10

Share:

Share on facebook
Facebook
Share on twitter
Twitter
Share on pinterest
Pinterest
Share on linkedin
LinkedIn

Hola! Did you notice that LinuxBSDos.com no longer run network ads?  Yep, no more ads from the usual suspects that track and annoy you across the Internet. But since I still need to pay to keep the site running, feel free to make a small donation by PayPal or your favorite cryptocurrency.

  • Bitcoin
  • Ethereum
  • Xrp
  • Bitcoin cash
  • Bitcoin sv
  • Litecoin
  • Binance coin
  • Cardano
  • Ethereum classic
Scan to Donate Bitcoin to bc1qzvlte2m224zkayhdc7fdfjkp2rsgt0l5a496ua

Donate Bitcoin to this address

Scan the QR code or copy the address below into your wallet to send some Bitcoin

Scan to Donate Ethereum to 0x0F4362DFF77F3Ba0Dc637F5f3Eba35D09a2fA60C

Donate Ethereum to this address

Scan the QR code or copy the address below into your wallet to send some Ethereum

Scan to Donate Xrp to r4ggjvL36njsMCYTkJ3S7cTHscPsMsSGQv

Donate Xrp to this address

Scan the QR code or copy the address below into your wallet to send some Xrp

Scan to Donate Bitcoin cash to qrs0dedzp9t55af3nfwypydghp29r0xguy9s20fz2k

Donate Bitcoin cash to this address

Scan the QR code or copy the address below into your wallet to send some Bitcoin cash

Scan to Donate Bitcoin sv to 15K9TLyVDBtLuG9cYvXCX9SSkq9C9oUKHK

Donate Bitcoin sv to this address

Scan the QR code or copy the address below into your wallet to send some Bitcoin sv

Scan to Donate Litecoin to LetJ9QQMb7u2LMZ9Tu6rtHwcBcQFW98fbG

Donate Litecoin to this address

Scan the QR code or copy the address below into your wallet to send some Litecoin

Scan to Donate Binance coin to bnb1ga8trq08ssqepd90v6225nzfgy448pu5pw8gxp

Donate Binance coin to this address

Scan the QR code or copy the address below into your wallet to send some Binance coin

Scan to Donate Cardano to addr1qx2354yw49etstfljpdhwja3ajjlt487lg95vu9ngy2q6vu4rf2ga2tjhqknlyzmwa9mrm997h20a7stgectxsg5p5esq5l7d9

Donate Cardano to this address

Scan the QR code or copy the address below into your wallet to send some Cardano

Scan to Donate Ethereum classic to 0xcD6CC972a2297FcafACDcfE042C55C69516a9264

Donate Ethereum classic to this address

Scan the QR code or copy the address below into your wallet to send some Ethereum classic

Subscribe for updates. Trust me, no spam!

Sponsored links

1. Attend Algorithm Conference, a top AI and ML event.
2. Reasons to use control panel for your server.
3. DHgate Computers Electronics, Cell Phones & more.

Upcoming events

One Response

Leave a Reply

Your email address will not be published. Required fields are marked *

Get the latest

On social media
Via my newsletter

Partner links

1. Attend Algorithm Conference, a top AI and ML event for 2021.
2. Reasons to use control panel for your server.
3. DHgate Computers Electronics, Cell Phones & more.
Hacking, pentesting distributions

Linux Distributions for Hacking

Experts use these Linux distributions for hacking, digital forensics, and pentesting.

Categories
Archives

The authors of these books are confirmed to speak during

Algorithm Conference

T-minus AI

Author was the first chairperson of AI for the U.S. Air Force.

The case for killer robots

Author is the Director of the Center for Natural and Artificial Intelligence.

Why greatness cannot be planned

Author works on AI safety as a Senior Research Scientist at Uber AI Labs.