So after my last not-so great experience with prosumer grade stuff, I dwelled more deeper into depth of Router/Firewall OS. After spending some time reading, I decided to give pfSense a try. And it has been working fine for me for a while. There are many things to learn and possibilities are limitless (along with opportunity to learn something new every time)… only downside is (according to my wife ), sometimes I end up spending lot of time to do something so trivial. I experienced something similar few days ago…
At one day for no reason our internet seem to stop working all together. I was able to access local wifi and LAN devices but none of devices were able to reach outside world. My first guess was our IPS connection crapped out because it was windy outside and in past it have happened. I disregarded steady lights in cable modem and power cycled it… no dice. So my next victim was my pfSense router, which I power cycled it as well… that too didn’t help. Things locally were still running just fine but Alexa was still dead . I was able to ping external sites from pfSense router itself… so now I was thoroughly confused and wife was frustrated.
Based on my experience so far, I was able to deduce that something is causing requests to not go past my pfSense (because I was able to ping Google from router itself). I still had no clue what to look for but I connected to router. And on it’s dashboard I saw that all services were running fine except one … “dns resolver (unbound)”. Based on reading pfSense documentation about this service it appear to be critical service which basically does what it’s name suggests. It tries to resolve DNS as one of way to make sure website that you are tring to reach is what it says it is (or at least that is how I understood ). Oddly, I tried to restart that service but it stopped as soon as it came back !! So this kind of pointed me to potential issue because I am able to reach local devices by IP but not anything over internet because DNS can’t resolve names !!
So next, I looked little further… and to my surprise I see that my disk is full !! I have about 25 GB free space which I always thought that more than enough for a router/firewall because most consumer grade devices will probably have storage capacity in meager MBs… because ideally you dont want to keep anything in those devices (and it’s not even their job). Now this gave me some hope that I am on right track…
Then, quick Google Fu came up with search that generally IDS apps like Snort or Suricata gobble up lot of space in log and that is most common issue. So quick search for size in /var/log folder showed me that my snort has over 23GB of logs !!
Finally after spending better part of my evening, I found the culprit. Since I am very new to pfSense, Snort logs didn’t matter to me much… so deleted whole directory. Once I got my space back, dns resolver service stayed running without crapping out.
So lesson learned… never let logs grow without some limit. And always always always check status of services (at least for PfSense) to get idea what the hack is going on.
That’s it for now.