Why did this server run out of disk space?

OSSEC HIDS Linux security

Less than two hours after I logged into the admin end of this website, which is powered by WordPress, the site went offline, with a 502 Bad Gateway error. What the…?

So I logged in via ssh and noticed a serious lag between when I type a letter and when it appears on the screen. I’ve got a problem. And with latency like that, running top will not help. After verifying that all the server applications were running, I checked disk usage using df -h. The output hinted at the problem I was facing.

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda         40G   38G     0 100% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            992M   12K  992M   1% /dev
tmpfs           201M  320K  200M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none           1002M     0 1002M   0% /run/shm
none            100M     0  100M   0% /run/user

The server was out of disk space! But this is a Cloud server with 40 GB of disk space and only serving a couple of websites. What could have consumed all the disk space?

Related Post:  482 of the Top500 supercomputers run Linux, and China's Tianhe-2 is the fastest

Just to recover some space and see if I could get the site back online, I decided to delete a few zip files that had served their purpose. Another df -h after that revealed that I had recovered 500 MB.

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda         40G   37G  500M  99% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            992M   12K  992M   1% /dev
tmpfs           201M  336K  200M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none           1002M     0 1002M   0% /run/shm
none            100M     0  100M   0% /run/user

That brought the site back online and put a smile on my face. And just at that point I recalled having installed a backup plugin. Perhaps the backups were the culprits. An ls -lh in the backup directory confirmed my suspicion.

-rw-r--r-- 1  93M Feb  8 03:00 database-backup-1417314413.sql
-rw-r--r-- 1  14M Jan 25 03:05 backup-1417314413-complete-2015-01-25-03-00-51.zip
-rw-r--r-- 1  11G Feb  1 03:08 backup-1417314413-complete-2015-02-01-03-00-02.zip
-rw-r--r-- 1 3.4G Feb  8 03:00 backup-1417314413-complete-2015-02-08-03-00-00.zip

Two huge backup files were in the directory and the latest had a timestamp that matched when the site went offline. Since I had already downloaded the previous backup, I deleted it from the server. Another df -h broadened my smile.

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda         40G   27G   11G  72% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            992M   12K  992M   1% /dev
tmpfs           201M  336K  200M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none           1002M     0 1002M   0% /run/shm
none            100M     0  100M   0% /run/user

Now the site is running like it’s supposed to. The server running out of disk space was an oversight. The main lesson here is I need a better backup strategy. Probably need to be backing up to a Cloud storage service. The other lesson is I need to have a script that reports disk usage by email daily, because this should never happen again.

Related Post:  TokuMX 2.0 released

Share:

Share on facebook
Facebook
Share on twitter
Twitter
Share on pinterest
Pinterest
Share on linkedin
LinkedIn

Hola! Did you notice that LinuxBSDos.com no longer run network ads?  Yep, no more ads from the usual suspects that track and annoy you across the Internet. But since I still need to pay to keep the site running, feel free to make a small donation by PayPal or your favorite cryptocurrency.

  • Bitcoin
  • Ethereum
  • Xrp
  • Bitcoin cash
  • Bitcoin sv
  • Litecoin
  • Binance coin
  • Cardano
  • Ethereum classic
Scan to Donate Bitcoin to bc1qzvlte2m224zkayhdc7fdfjkp2rsgt0l5a496ua

Donate Bitcoin to this address

Scan the QR code or copy the address below into your wallet to send some Bitcoin

Scan to Donate Ethereum to 0x0F4362DFF77F3Ba0Dc637F5f3Eba35D09a2fA60C

Donate Ethereum to this address

Scan the QR code or copy the address below into your wallet to send some Ethereum

Scan to Donate Xrp to r4ggjvL36njsMCYTkJ3S7cTHscPsMsSGQv

Donate Xrp to this address

Scan the QR code or copy the address below into your wallet to send some Xrp

Scan to Donate Bitcoin cash to qrs0dedzp9t55af3nfwypydghp29r0xguy9s20fz2k

Donate Bitcoin cash to this address

Scan the QR code or copy the address below into your wallet to send some Bitcoin cash

Scan to Donate Bitcoin sv to 15K9TLyVDBtLuG9cYvXCX9SSkq9C9oUKHK

Donate Bitcoin sv to this address

Scan the QR code or copy the address below into your wallet to send some Bitcoin sv

Scan to Donate Litecoin to LetJ9QQMb7u2LMZ9Tu6rtHwcBcQFW98fbG

Donate Litecoin to this address

Scan the QR code or copy the address below into your wallet to send some Litecoin

Scan to Donate Binance coin to bnb1ga8trq08ssqepd90v6225nzfgy448pu5pw8gxp

Donate Binance coin to this address

Scan the QR code or copy the address below into your wallet to send some Binance coin

Scan to Donate Cardano to addr1qx2354yw49etstfljpdhwja3ajjlt487lg95vu9ngy2q6vu4rf2ga2tjhqknlyzmwa9mrm997h20a7stgectxsg5p5esq5l7d9

Donate Cardano to this address

Scan the QR code or copy the address below into your wallet to send some Cardano

Scan to Donate Ethereum classic to 0xcD6CC972a2297FcafACDcfE042C55C69516a9264

Donate Ethereum classic to this address

Scan the QR code or copy the address below into your wallet to send some Ethereum classic

Subscribe for updates. Trust me, no spam!

Sponsored links

1. Attend Algorithm Conference, a top AI and ML event.
2. Reasons to use control panel for your server.
3. DHgate Computers Electronics, Cell Phones & more.

Upcoming events

Leave a Reply

Your email address will not be published. Required fields are marked *

Get the latest

On social media
Via my newsletter

Partner links

1. Attend Algorithm Conference, a top AI and ML event for 2021.
2. Reasons to use control panel for your server.
3. DHgate Computers Electronics, Cell Phones & more.
Hacking, pentesting distributions

Linux Distributions for Hacking

Experts use these Linux distributions for hacking, digital forensics, and pentesting.

Categories
Archives

The authors of these books are confirmed to speak during

Algorithm Conference

T-minus AI

Author was the first chairperson of AI for the U.S. Air Force.

The case for killer robots

Author is the Director of the Center for Natural and Artificial Intelligence.

Why greatness cannot be planned

Author works on AI safety as a Senior Research Scientist at Uber AI Labs.