WE ARE BACK! The Superman Homepage’s Week from Hell!

The Superman Homepage was offline from the morning of Friday, January 3 until the night of Thursday, January 9. Here is what happened…

In the early hours of Friday morning (USA EST) my System Administrator attempted to perform (what should have been) a routine operating system update on our web server. We don’t know why, but the update process crashed the server and it wouldn’t come back online. An additional expert was brought in to help resolve the issue, but every attempt to bring the server back online failed. The hosting company suggested we try reboot the server in Recovery Mode. While that brought the server back online, we were unable to access any of the website’s data.

The hosting company did their own remote investigation and suggested that one or both of the two hard drives within the server had died. They said the hard drives needed to be replaced. Unfortunately, while I have copies of a large amount of the website’s files on my local computer, no proper backup of the entire website was available. The regular backups that were conducted are stored on the same server. Replacing the hard drives would end all hope of getting the data back.

Either way, my guys refused to believe the hard drives were dead, as the server was still online, we just couldn’t access the data. My admins were adamant that it was a misconfiguration and nothing more, because in their opinion it was impossible for the server to still be online if both of the drives were dead. Most the technical jargon goes over my head, and probably doesn’t mean much to you neither. The point is, I was out of my depth and relying on the expert knowledge of the people I trusted.

So, with that theory put forward, IONOS initiated a physical investigation of the actual hard drives. Just to make this matter even more complicate, blizzard conditions at the location of the data center prevented anyone from accessing the building as all the roads were closed, adding a further delay and more stress for me.

When the technicians were finally able to access the data center and investigate my server, they discovered that the hard drives weren’t the problem, it was the server’s motherboard that needed replacing. However, we still weren’t able to access the server, so we went another day before technicians got back to the data center to discover that there was another different problem with the server not receiving reboot signals. Once that was resolved we were able to access the hard drives and determine that all my data was still there! I can’t tell you how relieved I was.

However, (yes, another “however”), the server itself was still broken. The operating software needed to be reinstalled, but before we tackled that, my admins suggested I perform a full and current backup of all my data. 152Gb doesn’t sound like a lot in the grab scheme of things, but downloading that via Secure File Transfer Protocol (SFTP) is an extremely tedious process, especially when you’re transferring all that data from the other side of the world.

While that was happening (and is still happening) another Linux expert was brought in to try to bypass the software issue blocking the website from coming online. Finally the website was back online!

There are still issues that need to be resolved. The broken software remains broken, we need to negotiate with our web hosting company for recompense and get them to move us to a better, more modern and robust server. But the main thing is that the website is back online, no data was lost, and my sanity remains intact (just).

Still, when I look at the news about what’s going on in Los Angeles at the moment, it kind of puts it all in perspective somewhat.

Thank you to everyone who has sent me messages of support throughout this very trying time. I never want to go through this again.

The never-ending battle continues…