This problem has had me stuck for 3 days and I am making sure to document the adventure here for my future-self and potentially anyone else this may help.
What this is NOT
This is not a writeup on how to free up disk space; this is not a writeup on how to find the largest files on your Ubuntu install. There are 100s of great answers on AskUbuntu and ServerFault that cover this.
What this IS
If your block device (disk drive, USB device, etc.) is “suddenly full” on Ubuntu (well any Linux system really) and you cannot, for the life of you find the file that is eating up all the free disk space, then this is FOR YOU!.
Symptom
$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 549G 549G 0G 99% /You fire off a quick du command at / to find the Top 20 biggest files and see what is taking up so much space, but it tells you nothing helpful!
sudo du / | sort -n -r | head -20
... unhelpful list ...Problem
In *nix systems, a mount point is nothing more than an empty directory representing an inode in the file system that you’d like to (at some point, likely at boot) attach a device’s content root to.
So for example you have /mnt/usb-drive setup to show you the contents of your USB Key when you plug it into your machine.
When nothing is mounted, that mount point is just a plain old empty directory.
So what happens if you put something in the directory BEFORE mounting another block device on top of it?
Ahhhh, so here is the root of the problem… the filesystem of the mounted block device will “cover” and effectively hide the normal contents of directory until it’s unmounted.
So your operating system will correctly report that the host device is FULL, but no matter how hard you try whenever you ls -lFa the contents to see what’s taking up so much space, you’ll be seeing the contents from the mounted block device – hell the mounted device could be EMPTY even, which would be even more confusing.
Solution
What you need to do is “rebind” the host mount to another temporary mount point, to effectively pull it and all it’s contents “out from UNDER” any covering that mounted block devices are doing and THEN analyze it’s files directly.
Good news, this is stupid simple…
- Create a new temporary mount point somewhere, I used:
sudo mkdir /tmp/fake-root - Now bind the root of your device you are trying to investigate to that new mount point:
sudo mount -o bind / /tmp/fake-root/
At this point the contents of /tmp/fake-root have sort of been slid out from under any other mounts attached to the original root device so you can just see the raw contents of this block device; which was /dev/sda2 in my case.
$ sudo du /tmp/fake-root/ | sort -n -r | head -20
536459948 /tmp/fake-root/
531320028 /tmp/fake-root/mnt
212503172 /tmp/fake-root/mnt/4d09d3b4-cf9a-451f-bb36-9b5dee7bf60d
<SNIP, a lot more dirs full>
Well WAIT a second, those /mnt/{UUID} points are suppose to be EMPTY in the raw drive and all mounted to drives in my disk arrays.
I think I found the culprit!
Indeed after some very targeted, manual deletion of these files and then unmounting the /tmp/fake-root dir, the drive is back to 3% space used and 97% free!
Problem Solved!