OSSEC HIDS Linux security

Less than two hours after I logged into the admin end of this website, which is powered by WordPress, the site went offline, with a 502 Bad Gateway error. What the…?

So I logged in via ssh and noticed a serious lag between when I type a letter and when it appears on the screen. I’ve got a problem. And with latency like that, running top will not help. After verifying that all the server applications were running, I checked disk usage using df -h. The output hinted at the problem I was facing.

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda         40G   38G     0 100% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            992M   12K  992M   1% /dev
tmpfs           201M  320K  200M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none           1002M     0 1002M   0% /run/shm
none            100M     0  100M   0% /run/user

The server was out of disk space! But this is a Cloud server with 40 GB of disk space and only serving a couple of websites. What could have consumed all the disk space?

Related Post:  What is Weave Discovery?

Just to recover some space and see if I could get the site back online, I decided to delete a few zip files that had served their purpose. Another df -h after that revealed that I had recovered 500 MB.

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda         40G   37G  500M  99% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            992M   12K  992M   1% /dev
tmpfs           201M  336K  200M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none           1002M     0 1002M   0% /run/shm
none            100M     0  100M   0% /run/user

That brought the site back online and put a smile on my face. And just at that point I recalled having installed a backup plugin. Perhaps the backups were the culprits. An ls -lh in the backup directory confirmed my suspicion.

-rw-r--r-- 1  93M Feb  8 03:00 database-backup-1417314413.sql
-rw-r--r-- 1  14M Jan 25 03:05 backup-1417314413-complete-2015-01-25-03-00-51.zip
-rw-r--r-- 1  11G Feb  1 03:08 backup-1417314413-complete-2015-02-01-03-00-02.zip
-rw-r--r-- 1 3.4G Feb  8 03:00 backup-1417314413-complete-2015-02-08-03-00-00.zip

Two huge backup files were in the directory and the latest had a timestamp that matched when the site went offline. Since I had already downloaded the previous backup, I deleted it from the server. Another df -h broadened my smile.

Filesystem      Size  Used Avail Use% Mounted on
/dev/vda         40G   27G   11G  72% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
udev            992M   12K  992M   1% /dev
tmpfs           201M  336K  200M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none           1002M     0 1002M   0% /run/shm
none            100M     0  100M   0% /run/user

Now the site is running like it’s supposed to. The server running out of disk space was an oversight. The main lesson here is I need a better backup strategy. Probably need to be backing up to a Cloud storage service. The other lesson is I need to have a script that reports disk usage by email daily, because this should never happen again.

Related Post:  Apache OpenOffice: Help pick a new logo

Share:

Share on facebook
Facebook
Share on twitter
Twitter
Share on pinterest
Pinterest
Share on linkedin
LinkedIn

Hola! Did you notice that LinuxBSDos.com no longer runs network ads?  Yep, no more ads from the usual suspects that track you across the Internet.  But since  I still need to pay to keep the site running, feel free to make a small donation by PayPal.

Subscribe for updates. Trust me, no spam!

Mailchimp Signup Form

Sponsored links

1. Attend Algorithm Conference, a top AI and ML event for 2020.
2. Reasons to use control panel for your server.
3. DHgate Computers Electronics, Cell Phones & more.

Leave a Reply

Your email address will not be published. Required fields are marked *

Get the latest

On social media
Via my newsletter
Mailchimp Signup Form

Partner links

1. Attend Algorithm Conference, a top AI and ML event for 2021.
2. Reasons to use control panel for your server.
3. DHgate Computers Electronics, Cell Phones & more.
Hacking, pentesting distributions

Linux Distributions for Hacking

Experts use these Linux distributions for hacking, digital forensics, and pentesting.

Categories
Archives

The authors of these books are confirmed to speak during

Algorithm Conference

T-minus AI

Author was the first chairperson of AI for the U.S. Air Force.

The case for killer robots

Author is the Director of the Center for Natural and Artificial Intelligence.

Why greatness cannot be planned

Author works on AI safety as a Senior Research Scientist at Uber AI Labs.

Anastasia Marchenkova

An invitation from Anastasia Marchenkova

Hya, after stints as a quantum researcher at Georgia Tech Quantum Optics & Quantum Telecom Lab, and the University of Maryland Joint Quantum Institute, I’m now working on superconducting qubit quantum processors at Bleximo. I’ll be speaking during Algorithm Conference in Austin, Texas, July 16 – 18, 2020. Meet me there and let’s chat about progress and hype in quantum computing.