magicbeans. nothing if not awkward.

Around midday I received two texts from Chris. One to let me know that he had just received the text I had sent him last night around 8:30. The other to let me know that Cybèle had been trying to get in touch with me because her website was not working. I hadn't heard anything from her. I checked, and yes, her site wasn't working.

I checked frgm.net, also not working. Which implied it was a server-wide problem. No luck connecting through ssh, so it wasn't just Apache. Checked the status of crabapple through Panix's web interface. "vc57 is running," it told me at the top of the page. (vc57 is crabapple's account ID.) I logged in through the console server just to make sure this was in fact the case. It was. Apache was running. BIND was running, but I figured that it must be something to do with DNS.

I hadn't changed anything though, and everything had been running smoothly since I clawed my way out of the hole of a corrupt kernel two and a half months ago. Lookup request worked fine when querying locally, but returned servfail errors when querying outside nameservers. So here's where I stated changing things.

(I was performing all this sysadmining on my iPad, by the way.)

Four hours later, having transferred the zone record that points to my secondary DNS host, nip.ell-eye.com—which is currently also crabapple, but through a different IP, I know I know this does no one any good—to my registrar, queries for my hosted web domains started to resolve and the websites began to load as expected. Though I still had no idea why requests stopped resolving in the first place.

So DNS was working. Except for fruit.org. And, as fruit.org is my networking domain, other services, such as primary DNS (plum) and email (mandarin), were still offline.

Part of the difficulty with troubleshooting and fixing DNS issues is that updated information often needs time to propagate, caches need to be cleared, and so forth. If you're not patient enough, sometimes you'll screw up the thing that would have worked before it actually kicked in. It's certainly possible I did something of the like over the next few hours.

(I hadn't left my bedroom between about 1:30 and 7:30.)

At one juncture, I saw a stray IP address in the DNS section of an whois record. Tracerouting it led to Austin. A reverse lookup indicated it was still assigned to turnip.li.com, which had been the primary DNS host during my tenure as li.com sysadmin—I named all the servers after six-letter root vegetables. It became secondary DNS post-LI as an official entity, when I started using fruit.org for networking and plum for DNS. When Daniel sold the li.com domain name, turnip became melon, though soon changed again to nip when I bought ell-eye.com. That IP address had been offline for eight months, when I moved nip over to crabapple's second IP. And other than some problems over the first few days because of caching, there hadn't been any problems.

I set about updating the DNS info for fruit.org with my registrar. This may have been a mistake. I did receive a few email messages around 5:30, when websites had started working, but then no more after the DNS change. In trying to change it back, the request was listed as "in progress" for hours. The fine print says that it can take up to six, so I decided to wait it out, make some dinner, and re-watch last week's episode of Game of Thrones since I hadn't been to studio to sync anything new with my iPad. Afterward the change request was listed as "failed."

I made another request and spent some more time poking around at other things, but being careful not to change too much. The final outcome would have to wait another day.