How to Install/Rescue Arch from Debian.

10/05/2025

Act One

One day I was over-caffeinated and zealously cleaning an Arch server I run. It's a dedicated one that I rent from Hetzner's earlier hardware available for a bargain at auction. It serves as my hub for reliving the unix timesharing golden days with friends and for all public and less-trusted daemons that I need and want, so it contains a lot of messes I wouldn't tolerate on a device that contains more of my personal information: educational NPM environments, a Minecraft server for the pals, or any new, emerging terminal apps that I want to just try, but not necessarily do any auditing for.

In my stupor of purity I ran a fun little command which promptly erased every program on the OS.

I've made a lot of mistakes while running Arch over the past seven years. I don't think any previous mistake has made me laugh quite as hard as this one. It's bizarre and a little creepy to have a shell still running in memory on a powerful computer with zero programs installed. All the commands were not found.

The impulse to simply restore from backup was there for a moment. Then I thought about all our documents and user files, which of course were still there. I realized I'd rather have something of an admin's adventure/punishment/lesson than lazily wait for a lot of files to sync back.

The Fix

I almost always install my unixes via manual bootstrapping (see aforementioned compulsive minimalism) and sometimes break things on existing installations, so I'm very comfortable operating in and rescuing from the Arch Linux installation environment. The problem in the case of rescuing this Arch server was that Hetzner Robot only offers a Debian-based rescue medium.

A few quick web searches later, I had confirmed that all the packages I needed are available to me.

Run on this system, pacstrap would say `error: no usable package repositories configured.` So first, download the mirrorlist and name it /etc/pacman.d/mirrorlist. Create a pacman.conf file and name it /etc/pacman.conf. Then, edit it and replace the definition of core like:

Finally, set up the keyring.

If you don't do this, you get `error: keyring is not writeable` from pacstrap.

At this point, pacstrap, pacman, and arch-chroot work on the Debian Hetzner rescue medium!

Since this server is one of the few unix I installed using an automated script, I needed to check Hetzner's documentation to confirm which partitions were which in the conventional case. The sizes of these partitions made sense. Indeed, md0 was swap, md1 was boot, and md2 was root. You'll see that the server uses RAID1 storage.

I continued:

At this point I could `ls /mnt/home/david` or any other folder and see that my files were all still there and I didn't mess up the disks with my initial mistake or in my rescue up to this point. (Which is obvious. A package manager shouldn't change anything but program files and maybe their associated configurations.)

It was time to put back the files I unwittingly erased: the kernel, bootloader and entire userspace. I guessed since I was already this deep into my endeavor for a cleaner OS, I might as well start small and only add back the minimum at first and see what else I need later.

It was time to save the mountpoints of these partitions in compliance with fstab and start working inside the target environment.

Since they had been removed, the essential systemd services presumably were no longer enabled.

Arch wiki reliably showed me everything I didn't know about installing GRUB on RAID. You have to run grub-install for each boot partition in the array.

I apprehensively thought the system might be ready to boot now. That apprehension lowered my expectations and saved me a lot of frustration because it most certainly did not boot.

The Endless Last Mile

I really could not think of anything that was missing. I spent a few hours in the all-familiar "try shit and reboot" mode. The system sure looked like it was to be booted with MBR, but was I wrong and actually supposed to use EFI? Was something wrong with the RAID array? I could try mdadm reassemble, but would that reformat the partitions which I had taken pains to keep?

This is where I started to use LLMs: Mistral locally and eventually the big gun: Claude Sonnet.

I prompted both with a description of the system, the mistake I had made, then asked how I would make it bootable again. I also had them list potentially faulty assumptions I might be working with. Darn it if I hadn't already done everything the models suggested.

To potentially help narrow the scope of the problem, I had the LLMs provide me with forensically-minded commands and their expected outputs. What was I supposed to see in /boot? What did grub.cfg need to contain? Eventually I got so paranoid that before each reboot attempt I would run `sync` as was apparently common practice in unix days of yore when it couldn't be relied upon that the memory buffer was actually written to the medium.

Before too long, all there was left to check was the configuration that I had the least hands-on experience with and knew was my biggest blindspot: RAID.

First of all, I had neglected to reinstall the relevant program. This being my only system with RAID and my only system installed with an automated script, I didn't know better.

To install it without leaving the chroot:

I had Claude help me figure out if my intact mdadm.conf was the problem. Sure enough, the UUIDs pointing to each partition were not correct. The UUIDs in /etc/fstab were correct since I had already written to it from genfstab. I had Claude show me how to make the correct mdadm.conf and any other necessary configuration. Of course I only want Claude to narrow my search, not be my source of information, so I verified and obtained extra context by reading arch wiki and man mdadm.

Edit /etc/mkinitcpio.conf and insert "mdadm" in the correct sequence in the list of declared hooks.

If you use systemd and not udev in your init environment, use "mdadm" not "mdadm_udev". Depending on other programs you need in the init environment, you might need to place it in a different order than me, so consult the wiki.

I felt closer than ever but I still couldn't log into my server. The client ssh error message was no longer "Unreachable" but "Access Denied: publickey". I could almost taste success. I knew I didn't change my ssh keys, but after a brief session of head-scratching and chin-rubbing I realized the nature of unix had one last lesson to hand this single-minded minimalist.

I obviously don't allow root login via ssh, and my user account's shell is zsh, yet another program I mistakenly thought I could just install later.

After one final rescue boot where I `chroot`-ed and did `pacman -S zsh`, the party server was back.

TLDR

These steps make pacman, pacstrap, and arch-chroot work on Debian.

Download the mirrorlist and name it /etc/pacman.d/mirrorlist.

Create a pacman.conf file and name it /etc/pacman.conf. Then, edit it and replace the definition of core like: