How to recover from a stupid Sysadmin

So there I was, in a hurry to install wowroster's new software right before a raid with my World of Warcraft guild.

And one bit of it required php4-curl.

So I bravely attempted to install it (real quick. hah!).

It required a whole slew of updates. That's what I get for not keeping on track of needed packages.

So i figure, 'what the hell' and start the process.

And the raid draws closer. And someone is sending me screenshots with DKP expenditures in them. And someone wants to know if there's room for them. And someone needs me to go and look at their new outfit.

And the updates are getting there... getting there...

It's 10 minutes until the raid and the updates are almost done. There are some prompts "Are you sure? Are you sure? WARNING!". But I'm in a hurry. So Yes. Yes.

BANG. Broken glibc.


Argh!

So here's what you do in this situation.

Step One: Go to your raid and One-Shot Moroes. Get teh epix for your friend the rogue, and take a strong stab at Romulo and Juilienne.

Step Two: Duck out early (9pm or so).

Step Three: Call the server's caretaker (or get them in a Vent channel) and walk them through these steps.

1. Download and burn a copy of the System Rescue CD. sysresccd.org
2. Boot from it, and at the prompt

   ifconfig eth0 [IP ADDRESS]

3. next,

   ip route add default via [GATEWAY IP]

4. next,

   passwd

   And change the root password to something
5. next

   /etc/init.d/ssh start

And then the idiot sysadmin that broke it in the first place can fix it!

Like so:

1. Log in via ssh w/password set by generous and kind custodian you do not deserve to lick the boots of
2. Mount the root partition (and /usr, /lib, whatever else you need for the system to run) in /mnt/temp1
3. On another, working Debian box, download and use dpkg -x to extract a version of glibc from packages.debian.org. Pick either the version you HAD or the version you WANT. One will work, and one won't, it's sort of up to you which disappointment you want first. ;)
4. scp -r the files into your mounted FS
5. chroot into your mounted file system and begin reinstalling packages using apt-get until things either break or work better.