Introduction

At my previous company, which provides IT solutions to clients in the USA, we specialized in managing and delivering hospitality IT infrastructure. This included tasks like pulling LAN and fiber cables, configuring and installing routers, cameras, GPON systems, Wi-Fi planning, and more.

In 2023, few of our clients faced an issue with remote access to their NVR (Network Video Recorder) using the client application. Upon debugging the issue, we discovered it was related to Dynamic DNS (DyDNS). While installing cameras and NVRs at client properties, we provided remote access using Oracle’s DyDNS service. However, DyDNS failed to resolve client IP addresses properly, even though those IPs were correctly mapped to DyDNS domains.

This problem stemmed from internal issues within Oracle’s infrastructure. Additionally, Oracle announced on their website that they were shutting down their DyDNS service and migrating it to Akamai Link.

During a weekly sprint meeting at the company, I discussed the issue with my reporting manager and highlighted the service disruptions caused by Oracle’s DyDNS. I proposed the idea of creating our own DNS server for our domain as a potential solution. However, we faced another challenge, even if we successfully developed a DyDNS service, how would we integrate it with NVR systems like Dahua, LTS, and others? Nevertheless, our primary goal was to eliminate reliance on Oracle’s DyDNS services.

My reporting manager approved the idea but mentioned that permission was required from the COO, who in turn would need approval from the CEO. In the meantime, he asked me to research how to host a DNS server on our infrastructure. To assist with this, he provided access to the company’s domain name and an AWS VPS.

I began my research by searching online for guidance on hosting a DNS server. However, most of the blogs, articles, and GitHub projects I found focused on hosting private DNS servers on home servers to map LAN IPs to custom names, such as router.local. These setups primarily involved mDNS (Multicast DNS), which was not suitable for our needs.

After conducting extensive research on Google, GitHub, and Reddit, I found that the majority of posts and projects pointed towards using BIND software for setting up a DNS server.

At the same time, I started reading DNS-related RFCs: 1034, 1035, and 1912. For any networking engineer or anyone looking to develop DNS-based applications, I highly recommend reading these RFCs instead of solely relying on online tutorials. Trust me, these documents not only deepen your understanding of the DNS protocol but also provide valuable insights into how other network protocols function.

Even though I configured the company’s domain glue records to point to the public IP address of the VPS, as outlined based on RFC 1912.

  0xmm.in.      in      ns      ns1.0xmm.in.
                in      ns      ns2.0xmm.in.
  ns1.0xmm.in.  in      a       1.2.3.4
  ns2.0xmm.in.  in      a       1.2.3.5

Google was unable to reach the company domain Nameserver. Even after enabling tcpdump and monitoring the public-facing interface on the VPS, no requests were coming from Google or Cloudflare DNS. Through further research, I came across this website which helped me check DNS propagation. Strangely, only this service was able to query the name server. I could see the requests arriving at the DNS server using tcpdump.

I suspected the issue might be related to Google and Cloudflare not recognizing or resolving the name server for the domain. I considered reaching out to Google and Cloudflare to request that they add the domain’s name server to their DNS resolver services because, you know, why not ask the giants to help? But then I realized, maybe they have better things to do than add our inhouse DNS server to their resolver list!

Even after reading all the above-mentioned RFCs, I still wondered if I had missed some configuration. To gain a deeper understanding, I joined an APNIC-sponsored online class titled Open Lab: DNS. This lab focused on hosting your own DNS and reverse DNS servers using BIND9.

Since I was already familiar with BIND9 beacuse Oracle DyDNS using BIND software, I quickly completed the lab exercises. You might wonder, what is APNIC? In simple terms, APNIC (Asia-Pacific Network Information Centre) is the governing body responsible for managing IP addresses in the Asia-Pacific region. For example, if someone wants to establish their own ISP, they need to register with APNIC. Upon registration, they are assigned an ASN (Autonomous System Number) along with public IP addresses. This process is not limited to ISPs but applies to governments and other organizations as well.

For a production environment, I wanted to eliminate the use of BIND9 since I needed a solution that could update or delete DNS entries via a web API. After conducting some research, I came across this website which provided a simple yet perfect code-based example. It served as an excellent starting point for hosting an authoritative DNS server.

At one point, I reached out to an APNIC team member via mail, Awal Haolader, who was responsible for conducting the Open Lab: DNS lectures. I explained the DNS issue I was facing, and out of kindness, he agreed to help. He even sent me a meeting link to discuss the problem in detail.

In this part of the code, I only needed to make a few changes to get a basic authoritative DNS server running on my localhost.

var domainsToAddresses map[string]string = map[string]string{```` “google.com.”: “10.2.3.4”, “0xmm.in.”: “8.8.8.8”, }

Dig response from localhost

root@devtest:~/Desktop/workspace$ dig @127.0.0.1 google.com

; «» DiG 9.18.28-0ubuntu0.22.04.1-Ubuntu «» @127.0.0.1 google.com ; (1 server found) ;; global options: +cmd ;; Got answer: ;; -»HEADER«- opcode: QUERY, status: NOERROR, id: 41757 ;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 ;; WARNING: recursion requested but not available

;; QUESTION SECTION: ;google.com. IN A

;; ANSWER SECTION: google.com. 60 IN A 10.2.3.4

;; Query time: 0 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP) ;; WHEN: Sun Jan 12 15:57:12 IST 2025 ;; MSG SIZE rcvd: 54

root@devtest:~/Desktop/workspace$ dig @127.0.0.1 0xmm.in

; «» DiG 9.18.28-0ubuntu0.22.04.1-Ubuntu «» @127.0.0.1 0xmm.in ; (1 server found) ;; global options: +cmd ;; Got answer: ;; -»HEADER«- opcode: QUERY, status: NOERROR, id: 38607 ;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 ;; WARNING: recursion requested but not available

;; QUESTION SECTION: ;0xmm.in. IN A

;; ANSWER SECTION: 0xmm.in. 60 IN A 8.8.8.8

;; Query time: 0 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) (UDP) ;; WHEN: Sun Jan 12 15:59:37 IST 2025 ;; MSG SIZE rcvd: 48

Since Awal was not familiar with the DNS library, he couldn’t pinpoint the exact cause of the DNS issue. However, I shared all the debugging steps and traces with him during our Zoom meeting. He also checked from his side using an online tool called Intodns, According to this tool, everything appeared to be configured correctly, but DNS resolvers like Google and Cloudflare still couldn’t find the glue record for the nameserver. I didn’t let this go. Instead of relying on AWS and their domain service, I decided to purchase two domains from separate registrars, one from Cloudflare and another from Gandi, along with a public IP from my ISP.

In Cloudflare, I found that it didn’t allow me to point to my nameserver unless I paid extra, which I realized was a business plan. However, Gandi allowed me to configure my own custom nameserver, and it worked. I conveyed this success to him. Now, I could host my own nameserver and no longer depended on the registrar’s nameserver for each entry.

Despite the success, my reporting manager informed me that the CEO was not interested in pursuing this further, as it wasn’t deemed critical and the company could continue paying for Oracle’s DyDNS service. As a result, the project was shut down. However, all the effort wasn’t in vain. I realized that I could implement this for myself as a side project, and so this year, I decided to fully implement it for my personal use.

I’m planning to self-host an authoritative DNS server for my domain at home. Let’s see how this works out.

Conclusion

While the project to create a Authorative DNS server for the company was ultimately halted due to business priorities, the experience proved invaluable. It deepened my understanding of DNS configurations, tools, and the challenges. The hands-on approach and troubleshooting helped me build a stronger foundation in networking and DNS management. Ultimately, I was able to apply the lessons learned to create a personal project, this journey not only expanded my technical skills but also reaffirmed the importance of persistence and continuous learning in the face of challenges.

Tools I used during DNS troubleshooting

In the Dig example above, I want to demonstrate that in a DNS entry, you can use any IP address, including a domain name. However, in a production environment, it’s important to enter your domain or subdomain with an A record or any other relevant record. Feel free to experiment with it, and you’ll gain a better understanding.