It was all the way back in April 2018 when I first blogged about setting up my Pi-hole and since then, along with many upgrades and improvements, I've never had a single problem. Until now...
Pi-Hole
If you've not heard of a Pi-hole then you can check out all of my blogs on my Pi-hole setup and the various things I've done with it. In short, it's a DNS server that runs inside your house on a Raspberry Pi. The cool part is that it's super simple to setup and it can block all kinds of nasty malware, tracking and other bad things at DNS, so every device in your house gets protection. It makes things run faster (no bloated ads or other junk), it protects you (blocking nasty stuff) and it improves your privacy (bye bye trackers), and there's basically no downside. Unless it breaks...
No more DNS!
Of course, if your DNS server stops working, you have no DNS resolution in your house. If you have no DNS resolution you have no... well... you have no anything at all 😅
I keep my Pi-hole updated and generally well maintained but there was something that I'd missed that caused me problems. Because it's running on a Raspberry Pi, the OS is running on microSD card and, well, they generally don't have huge amount of storage capacity...
Ooops
Everything simply stopped working in my house. No streaming, browsers wouldn't load pages, even my smart light bulbs all turned off. Actually I made that last bit up, but we're probably not far away from that being true!
I was sat at my PC so I was quickly able to determine that DNS resolution seemed to be dead and I hopped over to the Pi-hole interface where there were a couple of errors highlighted for me in the menu.
I was quite surprised but also relieved that it was something as simple as Disk shortage, so I fired up an SSH connection to see what was going on.
Tidying up!
Once connected to the Pi-hole, it was pretty easy to see why there was a problem.
pi@pihole:~ $ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 15G 15G 0 100% /
devtmpfs 430M 0 430M 0% /dev
tmpfs 462M 3.4M 459M 1% /dev/shm
tmpfs 462M 47M 415M 11% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 462M 0 462M 0% /sys/fs/cgroup
/dev/mmcblk0p1 42M 25M 17M 61% /boot
tmpfs 93M 0 93M 0% /run/user/999
tmpfs 93M 0 93M 0% /run/user/1000Yep, that's definitely running short on disk space! My 16GB microSD card was obviously just not up to the task! There is such a command for these scenarios so I gave it a whirl, but the command to flush the logs takes a backup of the logs, and, with no space to store the backup, it failed...
pi@pihole:~ $ sudo pihole flush
[i] Flushing /var/log/pihole/pihole.log ...
error: error writing to /var/log/pihole/FTL.log.1: No space left on device
error: error copying /var/log/pihole/FTL.log to /var/log/pihole/FTL.log.1: No space left on device
^CI took a look at the /var/log/pihole folder to see what the culprit was and there were a couple of things in there taking up a bunch of space.
pi@pihole:~ $ ls -lah /var/log/pihole/
total 8.2G
drwxr-xr-x 2 pihole pihole 4.0K Aug 15 23:05 .
drwxr-xr-x 7 root root 4.0K Aug 15 23:08 ..
-rw-r--r-- 1 pihole pihole 8.0G Aug 15 23:13 FTL.log
-rw-r--r-- 1 pihole pihole 218M Aug 15 23:11 FTL.log.1
-rw-r--r-- 1 pihole pihole 2.1K Aug 15 00:00 FTL.log.3.gz
-rw-r--r-- 1 pihole pihole 2.4K Aug 14 00:00 FTL.log.4.gz
-rw-r--r-- 1 root pihole 79K Feb 15 2021 pihole_debug.log
-rw-r----- 1 pihole pihole 0 Aug 15 22:49 pihole.log
-rw-r----- 1 pihole pihole 16M Aug 15 22:49 pihole.log.1
-rw-r----- 1 pihole pihole 1.7M Aug 15 00:00 pihole.log.2.gz
-rw-r----- 1 pihole pihole 2.0M Aug 14 00:00 pihole.log.3.gz
-rw-r----- 1 pihole pihole 2.2M Aug 13 00:00 pihole.log.4.gz
-rw-r----- 1 pihole pihole 2.0M Aug 12 00:00 pihole.log.5.gz
-rw-r--r-- 1 root root 23K Aug 14 03:31 pihole_updateGravity.logThat 8GB log file probably isn't helping things so I decided to wipe it along with the smaller pihole.log just to save some space.
pi@pihole:~ $ sudo truncate -s 0 /var/log/pihole/FTL.log
pi@pihole:~ $ sudo truncate -s 0 /var/log/pihole/pihole.logI also decided to save a little more space while I was here and remove the older log files too.
pi@pihole:~ $ sudo rm /var/log/pihole/FTL.log.*
pi@pihole:~ $ sudo rm /var/log/pihole/pihole.log.*I'd already saved enough space at this point but decided to do a little more tidying up in the /var/log/ folder too. You can look for big offender in there with this command.
pi@pihole:~ $ ls -lah /var/log/My auth.log was a particularly hefty file so I trimmed that down and also removed all of the backed up versions and you can use the same pair of commands for any other big files you'd like to clean out, just update the filename.
pi@pihole:~ $ sudo truncate -s 0 /var/log/auth.log
pi@pihole:~ $ sudo rm /var/log/auth.log.*At this point, I've freed up more than enough disk space to get things running again, but I'm in cleaning mode and ran just a few more commands to ensure there wasn't any obviously wasted disk space.
pi@pihole:~ $ sudo apt autoclean
pi@pihole:~ $ sudo apt clean
pi@pihole:~ $ sudo apt autoremoveFinally, you can try the original command that failed and ensure that everything now works.
pi@pihole:~ $ sudo pihole flush
[✓] Restarting DNS server
[✓] Flushed /var/log/pihole/pihole.log
[✓] Deleted 48596 queries from databaseUntil next time!
Of course, I don't want there to be a next time, so I looked at how I can stop this happening again and prevent similar issues by being more cautious with my disk space.
Another file that might be quite sizeable is the pihole-FTL.db file, which you can check like this.
pi@pihole:~ $ ls -lah /etc/pihole/pihole-FTL.db
-rw-rw-r-- 1 pihole pihole 3.0G Aug 17 16:07 /etc/pihole/pihole-FTL.dbIf it's larger than you like, you can modify the retention period for data to be stored in the database from the default of 365 days to something a little less, like mine at 180 days now, by adding/editing the MAXDBDAYS value in pihole-FTL.conf.
pi@pihole:~ $ sudo nano /etc/pihole/pihole-FTL.conf
MAXDBDAYS=180 # Add/modify this line.
pi@pihole:~ $ sudo systemctl restart pihole-FTLThe next thing is to make sure those pesky log files don't get too big again and for that you can tweak the logrotate config located at /etc/pihole/logrotate. If you open that file in an editor, you will see the following. I've added some comments/explanations to each line in the first stanza just for you to see here, but they're not valid syntax, so don't use them.
/var/log/pihole/pihole.log {
su root root # What user/group to do the rotation as.
daily # How often to do the rotation.
copytruncate # How to process the rotation.
rotate 5 # How many old backups to keep.
compress # Whether to compress the old backups.
delaycompress # Keep the most recent backup uncompressed.
notifempty # Do not backup an empty file.
nomail # Do not send any emails.
}
/var/log/pihole/FTL.log {
su root root
weekly
copytruncate
rotate 3
compress
delaycompress
notifempty
nomail
}The FTL.log file is obviously the largest culprit based on the file sizes we saw earlier so I will talk about that one.
In order for logrotate to take a backup of that file, it's using the copytruncate approach. That means it will make a full copy of the file before truncating the original file to zero size, so you have to have at least enough disk space for one full copy of the file. I will change the weekly rotation to daily, so hopefully the file will be smaller each time, and, remove the delaycompress option as I don't want to store any uncompressed files if I don't need to. I also bumped the backup files slightly as it will now be daily instead. Here's the updated version.
/var/log/pihole/pihole.log {
su root root
daily
copytruncate
rotate 5
compress
notifempty
nomail
}
/var/log/pihole/FTL.log {
su root root
daily
copytruncate
rotate 5
compress
notifempty
nomail
}That looks all good and now the disk has much more free space and things are running smoothly again!
pi@pihole:~ $ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 15G 5.8G 8.4G 41% /
devtmpfs 430M 0 430M 0% /dev
tmpfs 462M 1.3M 461M 1% /dev/shm
tmpfs 462M 47M 416M 11% /run
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
tmpfs 462M 0 462M 0% /sys/fs/cgroup
/dev/mmcblk0p1 42M 25M 17M 61% /boot
tmpfs 93M 0 93M 0% /run/user/999
tmpfs 93M 0 93M 0% /run/user/1000
What happened?
In short, I actually don't know what happened and maybe someone can give me some pointers in the comments below. It's obvious that FTL.log just got way too big and if we look at this snippet from further up during the initial investigation, we can see that the current FTL.log is 8GB in size and the previous FTL.log.1 is only 218MB in size. My understanding, given that they rotate each week, is that the previous week created a 218MB log file but the current week managed a whopping 8GB. I've not had any changes in operation and can't explain why this might have happened. I also needed to get my DNS server up and running again quickly so the log files weren't saved.
pi@pihole:~ $ ls -lah /var/log/pihole/
total 8.2G
drwxr-xr-x 2 pihole pihole 4.0K Aug 15 23:05 .
drwxr-xr-x 7 root root 4.0K Aug 15 23:08 ..
-rw-r--r-- 1 pihole pihole 8.0G Aug 15 23:13 FTL.log
-rw-r--r-- 1 pihole pihole 218M Aug 15 23:11 FTL.log.1I'm hoping that the changes above will prevent this from happening again and if you have any tips, please drop them in the comments below. If you're running your own Pi-hole, maybe it's time to do some updates and do some cleaning up!
Update: 18th Aug 2022
Thanks to feedback on Twitter, I have this figured out now. It seems that this is a known issue and there's a new release of FTL to solve it. My UI wasn't showing any updates as being available, but I ran the update command anyway and it picked up the update. Fingers crossed to see how it holds!
pi@pihole:~ $ sudo pihole -up
[✓] Update local cache of available packages
[i] Existing PHP installation detected : PHP version 7.3.31-1~deb10u1
[✓] Checking for git
[✓] Checking for iproute2
[✓] Checking for dialog
[✓] Checking for ca-certificates
[i] Checking for updates...
[i] Pi-hole Core: up to date
[i] Web Interface: up to date
[i] FTL: update available
[i] FTL out of date, it will be updated by the installer.
...