Building a Redundant Active Directory: A Second DC, Then a Clean Rebuild on Debian 13

This is a write-up of rebuilding my home lab's Active Directory to be actually redundant: adding a second domain controller, then rebuilding the original DC from scratch on Debian 13 rather than doing an in-place upgrade. What follows is what worked, and more usefully, the gotchas that ate most of the time.

Why bother

The real trigger was unrelated to AD: one of my Proxmox hosts needed to move to PVE9, but it was the sole host running my domain controller VM. Rebooting it for the upgrade would have taken down DNS/AD for the entire network, not just its own services. Rather than just live-migrating the DC somewhere else for the reboot, I decided it was time to actually have two DCs like a normal person.

Step 1: Bring up a second DC (dc2)

Cloned a Debian 13 cloud template, joined it to the domain as a full DC, copied sysvol over with tar --xattrs --acls, and ran ntacl sysvolreset. Pointed the router's secondary DNS entry at it. Two things broke immediately:

  • Samba's internal DNS server was IPv4-silent. It was binding only [::]:53 and then failing to bind 0.0.0.0:53 with NT_STATUS_ADDRESS_ALREADY_ASSOCIATED — a bind-order difference between the Samba version on the old DC and this new one. Fixed with explicit binding in smb.conf instead of relying on wildcard binds: 
    interfaces = lo eth0
bind interfaces only = yes
    I also disabled IPv6 entirely on this box rather than deal with dual-stack for now.
  • cloud-init's default /etc/hosts broke GSS-TSIG self-canonicalization. The stock 127.0.1.1 dc2 ... entry confuses Samba's DNS update auth. More on the actual fix for this below — the first thing I tried didn't survive a reboot.

Step 2: Rebuild the original DC as dc1

With a second healthy DC in place, I could safely rebuild the original DC (Debian 12, an older Samba version) instead of doing a risky in-place OS + 5-minor-version Samba upgrade. dc2 had already shaken out the whole recipe, so this rebuild carried a lot less risk than the alternative.

Built a new VM on a different host, Debian 13 cloud template, given a temporary IP so it wouldn't collide with the still-running original DC. Same package recipe as dc2, but with every fix already applied up front — no debugging needed this time. Joined it as a third DC, copied sysvol, ran ntacl sysvolreset.

Then transferred every FSMO role from the old DC (which held all seven, having been the only DC until dc2 joined) over to the new one, one at a time: 

samba-tool fsmo transfer --role=SchemaMaster -U administrator
samba-tool fsmo transfer --role=NamingMaster -U administrator
samba-tool fsmo transfer --role=InfrastructureMaster -U administrator
samba-tool fsmo transfer --role=RidAllocationMaster -U administrator
samba-tool fsmo transfer --role=PdcEmulationMaster -U administrator
samba-tool fsmo transfer --role=DomainDnsZonesMaster -U administrator
samba-tool fsmo transfer --role=ForestDnsZonesMaster -U administrator

Once that was confirmed, the old DC was cleanly demoted, not just deleted: 

samba-tool domain demote -U administrator

This matters — just deleting the VM leaves orphaned NTDS Settings / connection-object metadata behind in AD, the same kind of stale crumb I'd hit earlier from an aborted join attempt. After the demote, I confirmed it was actually clean: the old DC's name no longer resolved at all, no leftover deletion artifacts showed up in samba-tool drs showrepl on either remaining DC, and dbcheck --cross-ncs came back with 0 errors on both.

The old DC's VM was powered off (not deleted, yet) once the new one was verified.

Step 3: Move the new DC onto the old DC's IP

Rather than pick a new address for the rebuilt DC and have to update every host's resolver and the router's DNS config, I moved it onto the original DC's old IP. This is where most of the actual pain was.

  • cloud-init only applies netplan config once per instance-id. Changing the VM's IP config at the hypervisor level and rebooting does nothing — the guest just keeps its old address, since cloud-init already "did its job" for that instance. Had to edit /etc/netplan/50-cloud-init.yaml directly and run netplan apply (no reboot needed), then drop network: {config: disabled} into /etc/cloud/cloud.cfg.d/ so cloud-init wouldn't fight it on the next boot.
  • netplan's default link-local addressing quietly re-enables IPv6 per-interface, overriding the global net.ipv6.conf.all.disable_ipv6=1 sysctl. The per-interface sysctl gets reset independently. Needed link-local: [] in the netplan ethernets stanza — the global sysctl alone isn't enough.
  • Samba doesn't tolerate its bound IP disappearing out from under it. After the address change it failed with NT_STATUS_ADDRESS_NOT_ASSOCIATED and needed a manual systemctl restart samba-ad-dc.
  • A DC's own DNS record doesn't update itself when its IP changes — same broken self-registration path as the PTR issue below. Needed a manual delete/re-add of the A record, plus a fresh PTR record in the reverse zone.
  • Even after the DNS record was correct, samba-tool drs replicate kept failing with WERR_HOST_UNREACHABLE. Network and ports were fine. Turned out systemd-resolved's own cache on the peer DC — separate entirely from Samba's own DNS zone data — still had the old address cached. resolvectl flush-caches on the peer fixed it instantly. Any client with a cached lookup for a DC that changes IP needs the same treatment (one of my workstations had a stale entry too); otherwise it just self-expires on the zone's TTL eventually.
  • samba-tool dns add/delete over Kerberos can hang for a surprisingly long time (one case took over a minute) when a DC's own SRV record briefly points at an unreachable candidate first during a transition. Forcing NTLM and skipping KDC discovery avoids it: 
    samba-tool dns add ... -U administrator -k no

Small bonus: getting both DCs onto matching Samba versions mostly fixed a separate samba_dnsupdate TSIG verification issue I'd been living with since dc2 first joined. Before, every dynamic DNS update failed with a TSIG verify failure. After, only one record still does (a low-impact SRV locator record) — added manually the same way, and not worth chasing further given how narrow it's become.

Step 4: Actually verify both DCs survive a reboot

Two fixes made earlier for dc2 looked right but turned out not to be durable. Only caught this by actually rebooting both DCs for real, not just re-applying config live.

  • manage_etc_hosts: false as a cloud-init drop-in never actually worked. The Debian 13 cloud template ships its own per-instance userdata that sets manage_etc_hosts: true, and that always wins over a system-level config drop-in in cloud-init's merge order. A reboot silently reverted /etc/hosts back to the default 127.0.1.1 entry every time. This broke the DC's own hostname resolution badly enough that DNS RPC calls against itself failed instantly with a transport-connection-refused error — which looks exactly like an auth problem, not a hosts-file problem, so it took a while to trace.

    The real fix was editing /etc/cloud/templates/hosts.debian.tmpl directly — this is the actual template that gets rendered fresh on every boot, not something userdata can override. Swapped the default 127.0.1.1 line for each DC's real static IP. Verified surviving an actual reboot on both boxes. Deliberately did not bake this into the base cloud template itself, since that would just hide the regression for the next person cloning it — it's a fast two-line fix to apply per-VM after cloning.

  • The IPv6 link-local fix from Step 3 also didn't originally exist on dc2 — it predated that discovery, and IPv6 quietly came back after a reboot test. Applied and reboot-verified there too.

End state

Two Debian 13 DCs, matching Samba versions, replication and dbcheck clean on both. The new DC holds all seven FSMO roles. The original DC's VM is powered off but not yet deleted, kept around until I'm confident nothing else is still expecting it to exist.

Notes / things I'd tell past-me

  • If you're about to do a risky in-place upgrade on your only DC, stop and build a second DC first. It turns "one scary in-place upgrade" into "one already-debugged rebuild with a safety net."
  • Demote, don't delete. A raw VM delete leaves orphaned AD metadata behind that's annoying to clean up later.
  • cloud-init's per-instance netplan/hosts state is stickier than it looks — both the network config and the hosts file need per-VM overrides that survive a real reboot, and the only way to be sure is to actually reboot and check, not just re-apply config live.
  • DNS has more caches than you think: the AD-integrated zone itself, plus systemd-resolved's cache on every peer, plus every client's own resolver cache. An IP move isn't done until all of them agree.
by on

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)