Rescuing a customer on a failed ubuntu upgrade

One of our customers this evening mailed us to report that he’d upgraded Ubuntu on his colocated server and it had gone wrong. The machine refused to boot, and he’d managed to wipe out the serial settings in his grub configuration so he couldn’t alter the boot line in the configuration to add rootdelay=30. Could we help?

With a bit of fiddling, we could. On bootup the machine dropped out into busybox in the initrd.

ls /dev/mapper/system-system

revealed that the device for the lvm root volume was missing.

lvm vgchange -a y system

activated the root partition inside LVM so we could see it in /dev/mapper

fstype /dev/mapper/system-system

revealed the filesystem to be xfs

modprobe xfs
mkdir /mnt
mkdir /mnt/root
cd /mnt
mount /dev/mapper/system-system root

mounted the root filesystem inside of busybox.

modprobe ext3
mount /dev/sda1 root/boot

mounted the /boot partition

cd root
./usr/bin/vim.tiny boot/grub/menu.lst

brought up a minimal vim editing the grub configuration. I could then add the serial console lines,

serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1
terminal --timeout=15 serial console

to the grub config and rootdelay=30 to the kernel line, reboot and the machine came up.

If this is the sort of thing you could have figured out yourself, we’re always happy to accept CVs at our jobs page. If this scares you we’d suggest you’d be interested in our managed hosting where we do these bits for you.