server-config/AGENTS.md

AGENTS.md - server-config (NixOS server "muffin")

Overview

NixOS flake-based server configuration for host muffin (deployed to root@server-public). Uses deploy-rs for remote deployment, disko for disk management, impermanence (tmpfs root), agenix for secrets, lanzaboote for secure boot, and ZFS for data storage.

Target Hardware

• CPU: AMD Ryzen 5 5600X (6C/12T, Zen 3 / znver3)
• RAM: 64 GB DDR4, no swap
• Motherboard: ASRock B550M Pro4
• Boot drive: WD_BLACK SN770 1TB NVMe (f2fs: 20G /persistent, 911G /nix; root is tmpfs)
• SSD pool tank: 4x 2TB SATA SSDs (raidz2) -- services, backups, music, misc
• HDD pool hdds: 4x 18TB Seagate Exos X18 (raidz1)-- torrents
  • Connected via esata to external enclosure
• USB: 8GB VFAT drive mounted at /mnt/usb-secrets (agenix identity key)
• GPU: Intel (integrated, xe driver) -- used for Jellyfin hardware transcoding
• NIC: enp4s0 (static 192.168.1.50/24)

Build / Deploy / Test Commands

# Format code (nixfmt-tree)
nix fmt

# Build the system configuration (check for eval errors)
nix build .#nixosConfigurations.muffin.config.system.build.toplevel -L

# Deploy to server
nix run .#deploy -- .#muffin

# Run ALL tests (NixOS VM tests, takes a long time)
nix build .#packages.x86_64-linux.tests -L

# Run a SINGLE test by name (preferred during development)
nix build .#test-zfsTest -L
nix build .#test-testTest -L
nix build .#test-fail2banSshTest -L
nix build .#test-ntfyAlertsTest -L
nix build .#test-filePermsTest -L
# Pattern: nix build .#test-<testName> -L
# Test names are defined in tests/tests.nix (keys of the returned attrset)

# Check flake outputs (list what's available)
nix flake show

# Evaluate without building (fast syntax/eval check)
nix eval .#nixosConfigurations.muffin.config.system.build.toplevel --no-build 2>&1 | head -5

Code Style

Nix Formatting

• Formatter: nixfmt-tree (declared in flake.nix). Always run nix fmt before committing.
• Indentation: 2 spaces (enforced by nixfmt-tree).

Module Pattern

Every .nix file is a function taking an attrset with named args and ...:

{
  config,
  lib,
  pkgs,
  service_configs,
  ...
}:
{
  # module body
}

• Function args on separate lines, one per line, with trailing comma.
• Opening brace on its own line for multi-line arg lists.
• Use service_configs (from service-configs.nix) for all ports, paths, domains -- never hardcode.

Service File Convention

Each service file in services/ follows this structure:

1. imports block with lib.serviceMountWithZpool and optionally lib.serviceFilePerms
2. Service configuration (services.<name> = { ... })
3. Caddy reverse proxy vhost (services.caddy.virtualHosts."subdomain.${service_configs.https.domain}")
4. Firewall rules if needed (networking.firewall.allowed{TCP,UDP}Ports)
5. fail2ban jail if the service has authentication (services.fail2ban.jails.<name>)

Custom Lib Functions (modules/lib.nix)

• lib.serviceMountWithZpool serviceName zpoolName [dirs] -- ensures ZFS datasets are mounted before service starts, validates pool membership
• lib.serviceFilePerms serviceName [tmpfilesRules] -- sets file permissions via systemd-tmpfiles before service starts
• lib.optimizePackage pkg -- applies -O3 -march=znver3 -mtune=znver3 compiler flags
• lib.vpnNamespaceOpenPort port serviceName -- confines service to WireGuard VPN namespace

Naming Conventions

• Files: lowercase with hyphens (jellyfin-qbittorrent-monitor.nix)
• Test names: camelCase with Test suffix in tests/tests.nix (fail2banSshTest, zfsTest)
• Ports: all declared in service-configs.nix under ports.*, referenced as service_configs.ports.<name>
• ZFS datasets: tank/services/<name> for SSD-backed, hdds/services/<name> for HDD-backed
• Commit messages: terse, lowercase; prefix with service/module name when scoped (caddy: add redirect, zfs: remove unneeded options). Generic changes use update or short description.

Secrets

• git-crypt: secrets/ directory and usb-secrets/usb-secrets-key* are encrypted (see .gitattributes)
• agenix: secrets declared in modules/age-secrets.nix, decrypted at runtime to /run/agenix/
• Identity: USB drive at /mnt/usb-secrets/usb-secrets-key
• Encrypting new secrets: The agenix identity is an SSH private key at usb-secrets/usb-secrets-key (git-crypt encrypted). To encrypt a new secret, use the SSH public key directly with age -R:

  age -R <(ssh-keygen -y -f usb-secrets/usb-secrets-key) -o secrets/<name>.age /path/to/plaintext
  

• DO NOT use ssh-to-age. Using ssh-to-age to derive a native age public key and then encrypting with age -r age1... produces X25519 recipient stanzas. The SSH private key identity on the server can only decrypt ssh-ed25519 stanzas. This mismatch causes age: error: no identity matched any of the recipients at deploy time. Always use age -R with the SSH public key directly.
• Never read or commit plaintext secrets. Never log secret values.

Important Patterns

• Impermanence: Root / is tmpfs. Only /persistent, /nix, and ZFS mounts survive reboots. Any new persistent state must be declared in modules/impermanence.nix.
• Port uniqueness: flake.nix has an assertion that all ports in service_configs.ports are unique. Always add new ports there. Make sure to put them in the specific "Public" and "Private" sections that are seperated by comments.
• Hugepages: Services needing large pages declare their budget in service-configs.nix under hugepages_2m.services. The kernel sysctl is set automatically from the total.
• Domain: Primary domain is sigkill.computer. Old domain gardling.com redirects automatically.
• Hardened kernel: Uses _hardened kernel. Security-sensitive defaults apply.
• PostgreSQL as central database: All services that support PostgreSQL MUST use it instead of embedded databases (H2, SQLite, etc.). Connect via Unix socket with peer auth when possible (JDBC services can use junixsocket). The PostgreSQL instance is declared in services/postgresql.nix with ZFS-backed storage. Use ensureDatabases/ensureUsers to auto-create databases and roles.

Test Pattern

Tests use pkgs.testers.runNixOSTest (NixOS VM tests):

{ config, lib, pkgs, ... }:
pkgs.testers.runNixOSTest {
  name = "descriptive-test-name";
  nodes.machine = { pkgs, ... }: {
    imports = [ /* modules under test */ ];
    # VM config
  };
  testScript = ''
    start_all()
    machine.wait_for_unit("multi-user.target")
    # Python test script using machine.succeed/machine.fail
  '';
}

• Register new tests in tests/tests.nix with handleTest ./filename.nix
• Tests needing the overlay should use pkgs.appendOverlays [ (import ../modules/overlays.nix) ]
• Test scripts are Python; use machine.succeed(...), machine.fail(...), assert, subtest

SSH Access

ssh root@server-public    # deploy user
ssh primary@server-public # normal user (doas instead of sudo)