A night of WordPress woes, a server fight
Among the many things I do, one is managing a WordPress website I built from scratch. Well, not entirely scratch, but I installed Ubuntu, WordPress, configured Apache and PHP, I host the DNS, and in most respects there’s nothing outside of my control other than network connection.
My site was set up three years ago as an attempt to make a mini-Pocketables so I could determine what Pocketables was doing wrong on the back end – we suffered greatly during Panda, Penguin, and the Mobile Friendly updates, my site didn’t. I win.
This isn’t an invite to hack me as a note, I’m not claiming I’m a security pro, just claiming I built the car and have run it through some crash tests.
Most days my website run fine, last night however things took a turn when I updated the Yoast SEO plugin. I’m going to preface this with none of the events that transpired where their fault.
In WordPress, generally to update something you click update and life goes on approximately 20 seconds later. On a bad night, you click update and things start breaking. Last night was a bad night.
Suddenly the SEO pack failed. It mostly worked, but I wasn’t able to set a keyword. In the process of finding out why it would work on one site (Pocketables,) and not mine, I started disabling plugins, updating the server (figured perhaps my X was out of date,) and by the time I was done 20 minutes later it now took a minute to load a page.
After some trial and error I determined that a Javascript minification plugin called Autoptimize might be the culprit for Yoast’s failing. This was determined during a 20 minute period last night when I had to step away from the keyboard as I had a three month old wanting some loving and feeding. Silly three month olds.
Bam – excluding two scripts from minification and things should be great! Nope. While I’d gotten Yoast to work and restored everything to its original condition, pages were still taking minutes to load. Going through the admin console was painful. It was 1am, I had to give it up for the night. Babies don’t care if you don’t sleep, they just wake up at 4am wanting fed.
Today the problems persisted and I thought about rolling the site back to a snapshot I had (I keep months of snapshots for this eventuality,) but plugged on. The site was at nearly 0% CPU usage, but it was lagging like it was at 100% (adjust for this being Ubuntu and that would be 0.2 and 18).
I jumped into a terminal and just to verify connectivity pinged Google. It took 30 or so seconds for the first ping to go out. maybe six seconds for the second, six more for the third. These should have been coming in about one-second increments.
Ah hah, thought I, I’d updated the kernel recently and I bet that going back to the previous kernel would fix it as this now was most definitely not a WordPress/Apache issue. Nope. They got their kernel right.
So what caused the insane slowdown and my night of tearing apart a WordPress install trying to get thing back to normal? If you guessed DNS malfunction give yourself a prize.
A DNS server run by the bandwidth provider died. It’s unreachable. While this wouldn’t have affected a straight up web page particularly hard, WordPress attempting to look up and push information out the back end through Jetbpack, probably checking in with Analytics or the NSA or something, and generally pulling info in order to create content, it does a lot of DNS lookups per page served and doesn’t cache them evidently.
DNS server #1 would time out after I’m guessing 6 to 8 seconds, and then #2 would be used. While 6-8 seconds doesn’t sound like much, a server looking up a schema, doing name resolution on the current user’s IP, sending a bit of data to a backend, etc really start to add up. In this case it added up to about a minute a page.
Of course, I spent most of the time trying to figure out what I had done wrong, which in this case was attempt a relatively simple repair before going to bed. I’d assume their DNS would have been up by now, but even after reporting it it’s not.
I changed the DNS server order in my resolv.conf, all is right with the world.
Things I learned:
- ping in Ubuntu appears to pull DNS every ping (it would take seconds between ping google.com, but if I pinged the IP address it was less than a second)
- After years of minifying javascript I finally found something it doesn’t play well with
- babies know exactly when you’re about to make a breakthrough and require feeding at that point
- If a DNS server is down, evidently in the default install Ubuntu will keep trying it before switching to secondary lookup. It’ll never just move on to the secondary being the primary.
- Yoast 3.0.1 didn’t play well with minification
That’s about it.
