ncdu

I awoke early Sunday morning (December 4 2016) to find that this website was not reachable. I, and visitors too, was getting the dreaded (for a website owner) 500 error message.

I went to bed just after 2:00 a.m. that Sunday, so between that time and 7:00 a.m. when I woke up, something had gone wrong. I manage my own servers, so there’s no technical support to call. I had to do something, because server down means revenue will be down at the end of the month.

Based on experience, the first thing I did after logging into the server was check whether MySQL was running. It was not. A database-driven website has to have the database application up and running or there’ll be no connection. An attempt to restart MySQL failed, with an output that included a line about not being able to write to disk or something of that nature.

That pointed to a disk is full situation. Not good! Now I have to find out what application or applications chewed up whatever free disk space I had left.

That’s when I thought about the easiest tool to use when troubleshooting disk usage on a Linux server – ncdu.

If you’re not familiar with it, ncdu is an ncurses interface for du, the tool used for estimating file space usage on Linux distributions. It’s installed out of the box, but ncdu is not, so if you which to use it, first install it using one of the following commands.

# Install ncdu on Ubuntu- or Debian-based distributions

sudo apt install ncdu

# Install ncdu on Fedora or CentOS

sudo dnf install ncdu

#

So I had a problem (my website was down) that was the result of full disk space on the server. There were two solutions – one temporary (delete files not needed on the server) and another much deeper solution (find out what application was writing too much data to disk). The temporary solution was easy, and in about five minutes I had freed up almost 300 MB of disk space. That was more than enough to restart MySQL (and other services) and get the website back up.

Related Post:  How to install HotShots on Fedora 19 and Ubuntu 13.04

That out of the way, I deployed ncdu to get at the root cause of the problem. The application can be run from any directory, including the root directory. The output will tell you the disk usage of each file and directory, with the ability to drill down into any listed directory.

So I began my troubleshooting journey by typing the following command:

# Run ncdu on a Linux distribution

ncdu /

#

The output, and the main interface of ncdu, is shown in Figure 1. It’s a very simple and easy application to use – the arrow keys, ENTER/Return key and a few other keys are all you need to navigate its interface. Just by looking at what was displayed on the screen, I knew right away where most of the disk space is being used. But what I had completely forgotten was I had assigned 4 GB of disk space to a swap file. Because DigitalOcean no longer recommends configuring swap on their servers, that’s 4 GB that I could reclaim. So installing and running ncdu has already paid dividends.

To drill down into a directory, you only have to selected it and press ENTER.

mcdu linux disk usage
Figure 1: Main interface of ncdu

Doing that on the /var directory further narrowed where most of the disk space is being used. What I saw on this screen surprised me – OSSEC is using up almost 14 GB of disk space. That’s too much, and hints at a misconfiguration of OSSEC. If OSSEC is new to you, it’s an host-based IDS. More about it here.

ncdu drill down
Figure 2: Open a selected directory on ncdu

By drilling much further down into the OSSEC directory, it was easy to see that OSSEC was the application that’s chewing up way too much disk space than it’s supposed to. So problem has been identified. I’ll have to do something about OSSEC within the next few days.

ncdu on linux
Figure 3: Drilling further down into a selected directory on ncdu

By drilling down into the directory holding the website’s data, I was able to find a few archive files that I could delete. These were files from last year that I didn’t need any more. One was a 1.2 GB archive that was neatly tucked away into in the bowels of WordPress. Now it’s gone.

ncdu
Figure 4: Still drilling further down into a selected directory on ncdu

So that’s how I spent the early hours of Sunday, December 4 (2016). It pays when you know what to do when trouble strikes. One final commentary on ncdu: Though it provides much needed information about each file and directory, it does not tell you the last access time of any resource, something you can get when calling du directly. For example, if you run du as given in the following command:

# Run du with the --time option

du -chs --time /var/www/* | sort -rn | head

#

The output should look something like the one below: The last time a file was accessed is useful when looking for signs of unauthorized access.

# Output of du with the --time option

8.0K	2015-07-13 11:13	/var/www/html
5.7G	2016-12-05 17:58	/var/www/sites
5.7G	2016-12-05 17:58	total

#

Hope you find this little piece about ncdu useful. And I hope your Sunday didn’t start out like mine.

Related Post:  How to dual-boot Linux Mint Debian Edition and Windows 7

Share:

Share on facebook
Facebook
Share on twitter
Twitter
Share on pinterest
Pinterest
Share on linkedin
LinkedIn

Hola! Did you notice that LinuxBSDos.com no longer runs network ads?  Yep, no more ads from the usual suspects that track you across the Internet.  But since  I still need to pay to keep the site running, feel free to make a small donation by PayPal.

Subscribe for updates. Trust me, no spam!

Mailchimp Signup Form

Sponsored links

1. Attend Algorithm Conference, a top AI and ML event for 2020.
2. Reasons to use control panel for your server.
3. DHgate Computers Electronics, Cell Phones & more.

One Response

Leave a Reply

Your email address will not be published. Required fields are marked *

Get the latest

On social media
Via my newsletter
Mailchimp Signup Form

Partner links

1. Attend Algorithm Conference, a top AI and ML event for 2021.
2. Reasons to use control panel for your server.
3. DHgate Computers Electronics, Cell Phones & more.
Hacking, pentesting distributions

Linux Distributions for Hacking

Experts use these Linux distributions for hacking, digital forensics, and pentesting.

Categories
Archives

The authors of these books are confirmed to speak during

Algorithm Conference

T-minus AI

Author was the first chairperson of AI for the U.S. Air Force.

The case for killer robots

Author is the Director of the Center for Natural and Artificial Intelligence.

Why greatness cannot be planned

Author works on AI safety as a Senior Research Scientist at Uber AI Labs.

Anastasia Marchenkova

An invitation from Anastasia Marchenkova

Hya, after stints as a quantum researcher at Georgia Tech Quantum Optics & Quantum Telecom Lab, and the University of Maryland Joint Quantum Institute, I’m now working on superconducting qubit quantum processors at Bleximo. I’ll be speaking during Algorithm Conference in Austin, Texas, July 16 – 18, 2020. Meet me there and let’s chat about progress and hype in quantum computing.