Do not Panic! Remote Server (Hetzner) not rebooting any more - A Solution

This blog post was published 10 years ago and may or may not have aged well. While reading please keep in mind that it may no longer be accurate or even relevant.

I went through this experience recently. First of all, don’t panic! I panicked, and because of this, I made a mistake: I didn’t wait long enough for it to come back online. Had I waited up to 60 minutes, it would probably have come online (see reason below). The long story:

I had broken packages on my Ubuntu 10.04 server and decided to fix them by

apt-get update
apt-get upgrade

While updating, I noticed that the package grub-pc also was upgraded, apparently a new bootloader (or bootloader configuration) was installed. This made me feel uncomfortable, since I didn’t know if the server would reboot after this upgrade. So, because of the saying “The devil you know is better than the devil you don’t know” (and a desire to sleep peacefully at night!) I decided to reboot the server and see what would happen. Shockingly for me, it did not come online. SSH connections failed with “Port 22: Connection refused” .

I panicked and asked the Hetzner Support (which is very responsive and supportive!) to install a LARA Remote Console so that I could see the text output of the booting screen. After some regular startup messages, the screen became blank. I panicked even more, declared the server completely dead, and started migrating data. We will see that this was not necessary.

Most regular servers at the Hetzner data center are running Software RAID. It seems that after a reboot (especially if you send a Hardware Reset) the OS needs some time to re-sync or check the file system. I am not sure what caused the delay in my case. Re-syncing the entire RAID array can take up to 1-2 hours, depending on your hardware and disc space.

So, wait at least 1 hour for it to come online, especially after a hardware reset! When you activate Hetzner’s Rescue system (which is very good) it will stay active for a minimum of 1 hour, so your server will be down for 1 hour at least, in any event. So you are not losing much by waiting a bit longer.

Now, in my case, I assumed that Grub2 was broken. So I activated the Hetzner Rescue System, booted into it, and reinstalled Grub2. I found the following method here. First you have to mount the regular RAID filesystem under /mnt:

mount /dev/md2 /mnt
mount /dev/md1 /mnt/boot
mount -t dev -o bind /dev /mnt/dev
mount -t proc -o bind /proc /mnt/proc
mount -t sys -o bind /sys /mnt/sys
chroot /mnt

At this point, you are in your regular root directory. To reinstall Grub2 the Debian Way:

apt-get install --reinstall grub-pc

To make ‘really sure’, I reconfigured the package:

dpkg-reconfigure grub-pc

It will ask you where to install the bootloader. I selected:

\[\*\] /dev/sda
\[\*\] /dev/sdb

No errors were reported. I rebooted again and it did not come online immediately, for the reasons previously mentioned. I waited long enough (in my case, 15 minutes) and it finally did come online. So, rule number 1 is: Don’t panic!

If you found a mistake in this blog post, or would like to suggest an improvement to this blog post, please me an e-mail to michael@franzl.name; as subject please use the prefix "Comment to blog post" and append the post title.