Tutorials/Tips

Troubleshooting disk usage on a Linux server with ncdu

I awoke early Sunday morning (December 4 2016) to find that this website was not reachable. I, and visitors too, was getting the dreaded (for a website owner) 500 error message.

I went to bed just after 2:00 a.m. that Sunday, so between that time and 7:00 a.m. when I woke up, something had gone wrong. I manage my own servers, so there’s no technical support to call. I had to do something, because server down means revenue will be down at the end of the month.

Based on experience, the first thing I did after logging into the server was check whether MySQL was running. It was not. A database-driven website has to have the database application up and running or there’ll be no connection. An attempt to restart MySQL failed, with an output that included a line about not being able to write to disk or something of that nature.

That pointed to a disk is full situation. Not good! Now I have to find out what application or applications chewed up whatever free disk space I had left.

That’s when I thought about the easiest tool to use when troubleshooting disk usage on a Linux server – ncdu.

If you’re not familiar with it, ncdu is an ncurses interface for du, the tool used for estimating file space usage on Linux distributions. It’s installed out of the box, but ncdu is not, so if you which to use it, first install it using one of the following commands.

So I had a problem (my website was down) that was the result of full disk space on the server. There were two solutions – one temporary (delete files not needed on the server) and another much deeper solution (find out what application was writing too much data to disk). The temporary solution was easy, and in about five minutes I had freed up almost 300 MB of disk space. That was more than enough to restart MySQL (and other services) and get the website back up.

Related Post:  Download the beautiful Canopee wallpaper for KDE Plasma 5.9

That out of the way, I deployed ncdu to get at the root cause of the problem. The application can be run from any directory, including the root directory. The output will tell you the disk usage of each file and directory, with the ability to drill down into any listed directory.

So I began my troubleshooting journey by typing the following command:

The output, and the main interface of ncdu, is shown in Figure 1. It’s a very simple and easy application to use – the arrow keys, ENTER/Return key and a few other keys are all you need to navigate its interface. Just by looking at what was displayed on the screen, I knew right away where most of the disk space is being used. But what I had completely forgotten was I had assigned 4 GB of disk space to a swap file. Because DigitalOcean no longer recommends configuring swap on their servers, that’s 4 GB that I could reclaim. So installing and running ncdu has already paid dividends.

To drill down into a directory, you only have to selected it and press ENTER.

mcdu linux disk usage
Figure 1: Main interface of ncdu

Doing that on the /var directory further narrowed where most of the disk space is being used. What I saw on this screen surprised me – OSSEC is using up almost 14 GB of disk space. That’s too much, and hints at a misconfiguration of OSSEC. If OSSEC is new to you, it’s an host-based IDS. More about it here.

ncdu drill down
Figure 2: Open a selected directory on ncdu

By drilling much further down into the OSSEC directory, it was easy to see that OSSEC was the application that’s chewing up way too much disk space than it’s supposed to. So problem has been identified. I’ll have to do something about OSSEC within the next few days.

ncdu on linux
Figure 3: Drilling further down into a selected directory on ncdu

By drilling down into the directory holding the website’s data, I was able to find a few archive files that I could delete. These were files from last year that I didn’t need any more. One was a 1.2 GB archive that was neatly tucked away into in the bowels of WordPress. Now it’s gone.

ncdu
Figure 4: Still drilling further down into a selected directory on ncdu

So that’s how I spent the early hours of Sunday, December 4 (2016). It pays when you know what to do when trouble strikes. One final commentary on ncdu: Though it provides much needed information about each file and directory, it does not tell you the last access time of any resource, something you can get when calling du directly. For example, if you run du as given in the following command:

Related Post:  How to keep your Boot EFI partition clean of old Linux bootloader folders

The output should look something like the one below: The last time a file was accessed is useful when looking for signs of unauthorized access.

Hope you find this little piece about ncdu useful. And I hope your Sunday didn’t start out like mine.

LinuxBSDos needs your donation to continue!

I hope this article has saved you valuable time and effort to fix a problem that would have taken more time than is necessary. That makes me happy, and why I love doing this. But because more people than ever are reading articles like this with an adblocker, ad revenues have fallen to a level that's not enough to cover my operating costs. That's why I want to ask you a favor: To make a one-time or recurring donation to support this site and keep it going. It's a small favor, but every one counts. And you can make your donation using Patreon or directly via Paypal. Thank you for whatever donation you're able to make.

Donate via Patreon. Donate via Paypal.

Aside from donation, you may also signup to receive an email once I publish new content. Your email will not be shared or traded to anyone. And you can unsubscribe at any time.

Please share:

We Recommend These Vendors and Free Offers

Launch an SSD VPS in Europe, USA, Asia & Australia on Vultr's KVM-based Cloud platform starting at $5:00/month (15 GB SSD, 768 MB of RAM).

Deploy an SSD Cloud server in 55 seconds on DigitalOcean. Built for developers and starting at $5:00/month (20 GB SSD, 512 MB of RAM).

Want to become an expert ethical hacker and penetration tester? Request your free video training course of Online Penetration Testing and Ethical Hacking

Whether you're new to Linux or are a Linux guru, you can learn a lot more about the Linux kernel by requesting your free ebook of Linux Kernel In A Nutshell.


One Comment

  1. Bob Tennent

    I like xdiskusage, which gives a nice graphic presentation of directory sizes.

Leave a Comment

Your email address will not be published. Required fields are marked *

*