nixos-config

NixOS for four machines. ARCHITECTURE.md explains the layout and its rules. This file covers what Nix can't do: installing, bringing state back, and the steps outside the config.

Host	What it is
summit	leasewi.com: the forge and binary cache, Home Assistant, and the host of the anvil and blighttown guests
akulilo	flaked.ru: mail, Matrix, inference, the devboxes and the VPN
anvil	donut's desktop, a guest on summit
scout	donut's laptop, with a TPM-unlocked root and Secure Boot

Install summit first, because the others install from its forge and cache.

Before reinstalling

The install formats each system disk and never touches the bulk arrays (/mnt/bulk). State comes back as exports where a service has one, so the new versions start from data instead of migrating an old store. On each server, with its services running, carry the state to the bulk array:

sudo install -d -o $USER /mnt/bulk/carry && cd /mnt/bulk/carry
for db in $(sudo -u postgres psql -Atc 'select datname from pg_database where not datistemplate'); do
  sudo -u postgres pg_dump -Fc "$db" > "$db.dump"
done
sudo cp -a --parents /etc/ssh /home <the host's directories under "Bring back state"> .

On akulilo, also export the mail, account by account, as messages, folders and filters, and keep each account's record for its password hash (API and AUTH as in modules/mail/README.md):

nix shell nixpkgs#jq nixpkgs#stalwart-cli
mkdir mail
for account in $(curl -su "$AUTH" "$API/api/principal?types=individual" | jq -r '.data.items[].name'); do
  curl -su "$AUTH" "$API/api/principal/$account" | jq .data > "mail/$account.json"
  stalwart-cli -u "$API" -c "$AUTH" export account "$account" mail
done

anvil and scout carry /home and /etc/ssh (scout also /var/lib/sbctl) to summit's /mnt/bulk/carry/<host> the same way, over ssh. They also need a hardware report, which the servers already have: run nix run nixpkgs#nixos-facter -- -o hosts/<host>/facter.json on the old system, then in the host file swap ./hardware-configuration.nix for hardware.facter.reportPath = ./facter.json;, delete the old file, and push.

Install

1. Boot the NixOS 26.05 minimal ISO with its latest kernel entry; the default entry has no bcachefs. anvil boots it as a CD attached to its domain. Set a root password with sudo passwd root, then continue over ssh as root.

2. Fetch the flake with the host's own substituters. Write nix-cache-netrc's content to /root/netrc first, because akulilo can't reach cache.nixos.org and relies on the fleet cache. summit has no forge yet, so copy your working copy over instead of cloning:

   export NIX_CONFIG='experimental-features = nix-command flakes'
   git clone https://git.leasewi.com/donut/nixos-config && cd nixos-config   # summit: your copied working copy
   setting() { nix eval --raw ".#nixosConfigurations.<host>.config.nix.settings.$1" --apply toString; }
   opts=(--option netrc-file /root/netrc --option substituters "$(setting substituters)"
         --option trusted-public-keys "$(setting trusted-public-keys)")
   

3. Format and mount the system disk. Check first that lsblk and the host's disko.devices.disk.main.device name the same disk, because everything on it goes. On scout, write the root passphrase to /tmp/root.key first (printf %s '<passphrase>' > /tmp/root.key).

   nix build "${opts[@]}" .#nixosConfigurations.<host>.config.system.build.diskoScript && ./result
   

4. Bring back the keys. /etc/ssh/agenix_ed25519 opens the host's secrets, and the host keys keep clients' known_hosts valid. On scout, /var/lib/sbctl keeps the Secure Boot keys, so PCR 7 and the TPM seal stay valid. On a server, mount the bulk array (its device is in the host file) and copy from its carry:

   mkdir /carry && mount -t bcachefs <bulk device> /carry
   mkdir -p /mnt/etc /mnt/var/lib && cp -a /carry/carry/etc/ssh /mnt/etc/
   

   anvil and scout copy theirs from summit's /mnt/bulk/carry/<host> with scp. A new machine gets a new identity: run ssh-keygen -t ed25519 -N '' -f /mnt/etc/ssh/agenix_ed25519, put its public half in fleet.hosts, then run agenix -r in secrets/ and push. A scout without its old /var/lib/sbctl needs the firmware in Secure Boot setup mode before the install: Limine then makes and enrolls new keys, and the TPM seal needs renewing (hosts/scout/README.md).

5. Install:

   nixos-install --root /mnt --no-root-passwd --flake .#<host> "${opts[@]}"
   for admin in <admins>; do nixos-enter --root /mnt -c "passwd $admin"; done
   

   SSH takes keys only, so the password is for sudo and the console.

6. Reboot. systemctl --failed should be empty.

The bootloader goes to the removable path, so no machine's NVRAM changes.

Bring back state

Copy each directory from the carry with its unit stopped, chown -R it to the unit's user, then start the unit:

Host	State
every host	/home
summit	/var/lib/forgejo, /var/lib/hass, /var/lib/libvirt/images/win.qcow2
akulilo	/var/lib/hass (it holds the Zigbee network key), /var/lib/matrix-synapse, /var/lib/vaultwarden, /var/lib/private/open-webui, /srv/minecraft, and each box's /var/lib/nixos-containers/<name>

Restore each database into the one the config created, with its service stopped. Under the convention, the role that owns a database has its name:

sudo -u postgres pg_restore --clean --if-exists --no-owner --role=<db> -d <db> /mnt/bulk/carry/<db>.dump

A database that no module creates needs sudo -u postgres createdb -O <role> <db> first.

Mail: add each domain and account as in modules/mail/README.md, passing the secrets from the account's saved record, then run stalwart-cli -u "$API" -c "$AUTH" import account <account> /mnt/bulk/carry/mail/<account>. New DKIM keys mean new DNS records.

Delete /mnt/bulk/carry once everything works.

Outside the config

• DNS. Give each vhost its own A record, and don't use a wildcard: a name that holds any other record, such as a TXT, stops matching the wildcard (RFC 4592). A name without a record gets no certificate. To list the names:

  nix eval --json .#nixosConfigurations.<host>.config.services.nginx.virtualHosts --apply builtins.attrNames
  

• Router forwards. Forward the ports in networking.firewall, the devbox ports and blighttown's port:

  Host	TCP	UDP
  summit	22 (sshd and git), 24 (gpu box), 27 (blighttown), 80, 443, 5432
  akulilo	22, 24 and 26 (boxes), 25, 465, 587, 993 (mail), 80, 443, 1337 (Minecraft), 5432, 7881 (LiveKit)	7882 (LiveKit), 51867 (VPN)

  A box's port and blighttown's are DNAT rules, so nft list chain inet nixos-fw input-allow doesn't show them.

• summit's gateway must proxy cache.nixos-cuda.org, as it does cache.nixos.org. Direct, most of its lookups stall, and Nix then compiles CUDA instead (10-04: 125 of 150 narinfo requests stalled from summit, none from akulilo).

• First accounts. Open WebUI (ai.flaked.ru), Immich (photos.), Jellyfin (media.) and Home Assistant (home.) make the first visitor their admin. On a fresh instance, claim each of them before forwarding 80 and 443. Open WebUI's admin can run Python on the server.

• Tokens only the owner can mint. Each goes into its file with agenix -e <file>, run in secrets/:

  • the forge's runner registration token, as TOKEN=<token> in summit/forgejo-runner.age (Site administration → Actions → Runners);
  • an alert bot invited to fleet.alerts.room (#alerts:flaked.ru: aliases end in the server name, not matrix.), with its access token alone (syt_…, nothing around it) in shared/matrix-alerts.age and in the repository's Actions secret MATRIX_ALERTS. journalctl -u 'alert@*' shows the homeserver's reply when a post fails;
  • each Hermes bot, as described in modules/devbox/README.md, and each instance's secret as its host's hermes.nix describes it.

• Mail. Domains, accounts and DKIM keys live in Stalwart's store: modules/mail/README.md.

• The gpu box. gpu-box trusts the box's host key on first use. Compare ssh-keygen -lF '[<summit>]:24' in a box with ssh-keygen -lf /var/lib/nixos-containers/gpu/etc/ssh/ssh_host_ed25519_key.pub on summit.

Upgrades and backups

The servers upgrade at 05:00 from the forge's main and never reboot on their own. To test, run sudo systemctl start nixos-upgrade. At 03:00 each server backs up its small state to the other's restic server, at backup.<domain>. To restore, run this from secrets/:

export $(agenix -d <host>/restic-rest.age) RESTIC_PASSWORD=$(agenix -d <host>/restic-password.age)
restic -r rest:https://backup.<peer domain>/<host> restore latest --target /tmp/restore

The service databases are in /var/backup/postgresql/<db>.sql.gz; people's and applications' own are not backed up.

WinApps

blighttown runs Windows on summit. Its OpenSSH server is reached on summit's port 27 and forwards RDP only. To add a client:

1. On the client, run ssh-keygen -t ed25519 -N '' -f ~/.ssh/winapps_ed25519.
2. In Windows, add the public half to C:\ProgramData\ssh\administrators_authorized_keys for an administrator, or to %USERPROFILE%\.ssh\authorized_keys otherwise, behind restrict,port-forwarding,permitopen="127.0.0.1:3389".
3. Put that person's account in users.<name>.windows in fleet.nix, and their password in ~/.config/winapps/winapps.local.conf (RDP_USER="…", RDP_PASS="…", mode 0600). Then run winapps-setup --user.