I have a small homelab that’s not nice enough for /r/homelab but is a bit more than just self hosting. Since I’m a decently knowledgeable sysadmin and network engineer, my goal is to build an enterprise-ish environment for myself to tinker around and play inside. This means a lot of my setup is more complicated than it needs to be and I spend a lot of time troubleshooting and debugging my overengineering, so when something breaks my first assumption is that it was something I did. I usually build my stuff to be relatively aelf sufficient when I leave it alone.
But this weekend and today I simply couldn’t find what I broke. I was attempting to move a clunky lets encrypt cert renewal job off of my DNS server to somewhere I could better manage it. Why was it on my DNS server? Because for a while now, dynamic updates only half worked for me. My bind9 server was fully capable and I have a custom nsupdate cronjob to update my DDNS records that I installed on my UDM-Pro. But for whatever reason, as soon as I entered my home network1 it wouldn’t work. Since I thought it better to manage my certs from Proxmox or another internal service, I needed to figure out why this was. I looked high, I looked low, I looked in /etc but there was no configuration error that I could find. I tested the same TSIG key on another machine in my VPC and on my UDM-Pro but there it went without a hitch. The error was weird — NOTIMP — and I couldn’t find anything relevant online. As a last resort I turned to ChatGPT2, but all this confirmed was that there should be no errors with my configuration. It’s conclusion was that it had to be networking.
So i scoured the configuration of my UDM looking for any filtering or traffic rules I had, but nothing was clicking. This wasn’t a connection issue, this is the server telling me that updates were not allowed for this zone. I was clearly hitting the DNS server, right? Well there was nothing in the update logs on the server, so I suspected that for some reason the requests weren’t making it through. So I spun up wireshark on my UDM and on my DNS server, and saw for myself that the dynamic update requests weren’t even reaching the bind server. I would see the update come into the router, and a response from the bind server, so what was responding? This was either some crazy filtering from my ISP — which i knew to be false because updates from the router worked — or my UDM doing something. Finally after some sleep I came back and looked at the UDM cobsole again and it hit me.
Ad block.
I quickly paused it and lo and behold it was blocking my dynamic updates. There was no record of this in the Insights tab; it was just silently absorbing my dynamic updates and masquerading as my name server. I can understand masquerading as name servers due to what its supposed to do, but I have no idea why it would steal my dynamic updates. I wouldn’t think what DNS filtering that enables is fail closed. For being a prosumer company, Ubiquiti’s features always feel halfway implemented to work in most scenarios but never actually developing full support for things. Yes, I brought this onto myself for enabling ad-blocking (it was good while it lasted, I’ll have to reimplement it in a non stupis way) but the fact that it does zero inspection of the DNS opcode before forwarding requests feels dumb.
1I have two “sites”, my homelab and a cloud VPC; critical infra like DNS and mail is hosted in the VPC.
2I minimally use AI for troubleshooting as a last resort to either turn me on a new path to the solution or as a sanity check before I blame a different component.
I turned to ChatGPT2, but all this confirmed was that there should be no errors with my configuration. It’s conclusion was that it had to be networking.
Oh, you’ll smoke a turd in hell for that. /s
I fr hate using AI to troubleshoot because I can feel how it makes me lazy, but sometimes using AI is better than banging my head against a wall for 10 hours. And usually i stop once I find a productive line of research or investigation to follow.
Like google and stack overflow before it AI is a tool. If you use any of these and stop researching after the tool gives you an answer, researching to understand why the answer works in the first place, then any of these tools will make you lazy.
But you are human and it is imposible to understand everything so choose your battles.
Also, dude, your setup sounds sick as hell and this is a fantastic writeup. Thank you.
Thank you, it’s a lot of work and I could get by with a lot less but I’d like to essentially have enterprise level everything for me to just fuck around with and provide to friends as i see fit. It’s a bit if a hodgepodge of well implemented stuff stuck together with duct tape and bubblegum but im refining it slowly all the time.
Meh…it’s a tool that needs some heavy regulation, but a tool nonetheless.
I’ve always been flummoxed by Ubiquity products. I’m no sysadmin but I understand my way around networking and I absolutely agree with your “halfway implemented” critique. I installed Ubiquity at my parents’ house so that I could more easily do remote troubleshooting when something their network goes down. But for myself, I just stick with OpnSense at home. It’s not perfect but it suits my needs.
This was a fun writeup to read. Thanks for taking the time to post it.
Theres so much I end up handling manually with my UDM that at this point i might rather just install open source routing software on it atp. I don’t even use the web UI for wireguard because I can’t even specify the allowed IPs for a connection.
If you’re comfortable with full-fat DNS, Technitium has all the controls of bind9 and can do ad blocking as well, but it isn’t as… esoteric to setup. Easy import/export, decent webui, other quality-of-life features. Highly recommend.
For local DNS i run FreeIPA since everything in my network is domain controlled. I’m gonna look into adding filtering through that, but we’ll have to see how it goes.
dig, learn it, love it- Use a phone or other device outside your network to compare results from #1
I did use dig, but I didn’t do a trace which probably would’ve been helpful. I just didnt anticipate that id be getting MITM by my own infra.
Might be worth looking into a PiHole. One of the nice features is the white lists. So even if a list you are subscribed to is blocking something you need, you can still allow it specifically.
And/or run adblockers on each device individually. I actually do both, as the on-device blockers don’t get things like Windows telemetry. (Thank god the only Windows machine on my network anymore is my work laptop.)
I’m not entirely sure how I want to run my ad blocking yet. I left adblocking on for the wifi subnet because I don’t mind it there, and I have ublock origin on my PC. I might use PiHole but my DNS on my network is actually managed by FreeIPA so making sure everything works properly there is paramount. I’m pretty sure I can do that easily but I need to test it to make sure my forward zones work as expected and nothing breaks.
Is there a way for you to talk to upstream DNS bypassing Ubiquiti’s firewall? Maybe do it on a different port? (idk if the RFC permits this)
I just turned off ad blocking. I can set up network wide filtering without relying on proprietary incompetence.
Sounds like that adblock is implemented as a proxying DNS server? In that case, NOTIMP makes sense, if they haven’t implemented forwarding those type of requests.
Yeah I found some documentation from Ubiquiti afterwards that said all DNS requests would get proxied, although it didn’t mention it wouldn’t forward dynamic updates.
