> ## Content Index > Fetch the complete content index at: https://blog.coopnet.us/llms.txt > Use this file to discover other available public pages before exploring further. # Getting a single node Proxmox cluster ready for a second site - URL: https://blog.coopnet.us/proxmox-cluster-across-sites/ - Published: 2026-10-28T23:30:45.000Z - Updated: 2026-10-31T16:52:56.000Z - Description: My corosync edit kept reverting because I changed the local copy. Edit /etc/pve/corosync.conf with copy, edit, move and bump config_version. A single node cluster opens no UDP on 5405, so the overlay link can't be tested until node two, and quorum needs a plan first. - Author: Renstur - Tags: Proxmox & storage, proxmox, clustering, corosync The first edit I made to corosync.conf on my Proxmox node undid itself. I changed one line, restarted corosync to test it, and the line was back the way it was. I did it again. Same result. The config wasn't wrong. I was editing the wrong file, and the restart I used to check my work was the thing putting it back. The goal was getting a single node cluster ready for a second site, which meant moving corosync onto an overlay network and confirming it took. What I didn't expect was that none of it can really be proven until another node exists. ## What the node is running This was on Proxmox VE 9 in October 2026. My node is a cluster of one. `pvecm status` reports one node, one expected vote, knet transport, and secure auth on: ```sh pvecm status ``` That might sound pointless. But once you create a cluster, corosync and the Proxmox cluster filesystem (pmxcfs) are in charge of `/etc/pve`, and they behave the same with one member or five. The plan is to add a node at a second site eventually. The cluster link has to cross sites, and I didn't want it on the node's default address. I wanted it on my NetBird overlay. Changing the ring address now, while nothing depends on it, seemed like the easy time to do it. It was easy. It just took me a few tries to see that. ## Two copies of corosync.conf There are two corosync.conf files on a Proxmox node: - `/etc/corosync/corosync.conf`, the one corosync reads locally - `/etc/pve/corosync.conf`, which lives on pmxcfs and is shared across the cluster The second one is the real one. Changes made there propagate to the local copy, but not the other way around. So if you edit `/etc/corosync/corosync.conf` and restart to test, the cluster copy wins and your change gets overwritten. On my node it happened on that exact restart. That's the loop I was stuck in. Two more things make a correct edit look like a failure. First, `config_version` has to go up. If you change the config and leave the version alone, corosync treats it as the same config and ignores you. Bump it by one every time. Second, don't edit the file in place under `/etc/pve`. pmxcfs isn't a normal filesystem. Something like `sed -i`, or an editor that writes partial files, can leave a half written config that gets pushed out. The Proxmox docs describe a copy, edit, move pattern instead: ```sh cd /etc/pve cp corosync.conf corosync.conf.new cp corosync.conf corosync.conf.bak nano corosync.conf.new mv corosync.conf.new corosync.conf ``` Make your change and bump `config_version` in the `.new` file. The `mv` swaps it in as one step. Keep the `.bak` somewhere you can reach if pmxcfs stops working. More on that below. Before touching anything, back up both corosync.conf copies and the pmxcfs database at `/var/lib/pve-cluster/config.db`. Copy them off the node, not into another folder on it. ## Moving the ring address to the overlay The change itself is one key. In the nodelist, `ring0_addr` for my node went from its original address to its overlay address, something like `100.64.x.x`. `config_version` went from 1 to 2: ```text nodelist { node { name: pve-01 nodeid: 1 quorum_votes: 1 ring0_addr: 100.64.x.x } } ``` I applied it with the `mv` above and no restart. Then I restarted corosync and `pve-cluster` on purpose to make sure it stuck this time. It did. The node stayed quorate, `/etc/pve` stayed writable, and the VMs didn't notice. ## Confirming what corosync actually loaded After the reverting edit, I stopped trusting the file on disk. What matters is what corosync has in memory. Two commands show it: ```sh corosync-cfgtool -s corosync-cmapctl | grep -E 'ring0_addr|config_version' ``` `corosync-cfgtool -s` shows the link and the address corosync is using. `corosync-cmapctl` dumps the runtime keys. The two to look for are `nodelist.node.0.ring0_addr` and `totem.config_version`. If those show the overlay address and version 2, the change is live. If they show the old values, the file doesn't matter. ## Boot order, and the IPv6 gap I left open The next worry was boot order. The overlay interface comes up late, after `network-online.target`. If corosync starts first and tries to bind an address that doesn't exist yet, the bind fails. With no corosync, pmxcfs can drop `/etc/pve` to read only. That's a bad way to come back from a reboot. It turned out not to be a problem on my node. `net.ipv4.ip_nonlocal_bind = 1` was already set in a file under `/etc/sysctl.d/`. That setting lets a process bind an IPv4 address the machine doesn't have yet, so corosync can bind the overlay address before the overlay is up. There's a gap, though. The IPv6 version, `net.ipv6.ip_nonlocal_bind`, is still 0, and totem is set to `ip_version: ipv4-6`. With an IPv4 ring address that doesn't matter. If I ever move the ring to an IPv6 overlay address, the same boot race comes back with nothing guarding it. I've left it for now and written it down. If you're doing this over IPv6, set both. ## No UDP listener on 5405 The Proxmox docs say corosync uses UDP port 5405\. With knet, each link uses 5405 plus its link number, and the firewall range Proxmox asks you to open is 5405 to 5412\. So after moving the ring address, I looked for corosync listening on the overlay address: ```sh ss -ulpn | grep 5405 ``` Nothing. Corosync wasn't listening on 5405 at all, on any address. Every socket it had open was a Unix socket for local IPC. No UDP. My best guess is simple. With no peer, knet has no one to talk to, so it never opens the UDP socket. On my node, at least, a single node cluster sends no corosync traffic over the network. That changes what the ring address move proved. It proved I can edit the config the right way and that corosync loads the new address. It didn't prove the overlay can carry cluster traffic, because there isn't any yet. That can't be tested until a second node joins. ## Planning quorum before node two So the real prep work isn't the ring address. It's deciding how the cluster keeps quorum once it spans two sites. Two things matter here. The first is latency. On my node the totem token timeout is 3000 ms and the knet ping timeout is 1500 ms. Proxmox's guidance is that corosync wants latency close to a LAN, and the docs warn against stretching a cluster over links with high latency. An encrypted overlay between two sites is exactly that kind of link. It might be fine on a good day. What I care about is a bad one, and that isn't testable yet either. The second is what happens when the link drops. With two nodes at one vote each, losing the link means neither side has a majority. Both lose quorum and `/etc/pve` goes read only on both. With HA on, it's worse. A node that loses quorum fences itself through the watchdog, which means it reboots to protect the data. A flaky overlay plus HA is a good way to watch two nodes reboot each other's workloads away. The options I see: - A QDevice. This is a small external service (`corosync-qnetd`) on a third machine, ideally somewhere neither site depends on. Each node runs `corosync-qdevice`, and you set it up with `pvecm qdevice setup `. It gives the cluster a tie breaker vote. Proxmox recommends it for an even number of nodes and advises against it for odd counts. - A real third node. More hardware, but it's a full member with its own vote. - Two nodes, no tie breaker, HA off. A link outage makes `/etc/pve` read only on both sides until it comes back. VMs keep running, but config changes wait. For a homelab that might be acceptable. Know it going in. I haven't picked one yet. Writing it down now is the point. This is a much easier call before node two exists than after the first time the link flaps. ## Getting out of a read only pmxcfs If an edit goes wrong and `/etc/pve` drops to read only, you can't fix the config the normal way. The file you'd fix is on the filesystem that's locked. Proxmox has a local mode for this: ```sh systemctl stop pve-cluster systemctl stop corosync pmxcfs -l cp /root/corosync.conf.bak /etc/pve/corosync.conf killall pmxcfs systemctl start pve-cluster systemctl start corosync ``` `pmxcfs -l` mounts `/etc/pve` writable in local mode and ignores quorum. Put the backed up config back, stop the local instance, then start the services normally. This is why the backups come first. Read the official docs before running this on a cluster that still has other members. Local mode on a node with live peers is a different situation. ## VM start order on the same node This one's related to boot order, and I hit it on the same node. A VM with no startup order set starts after every VM that has one. If your router, DHCP or auth VMs have no order and some other VM does, the infrastructure comes up last. Give them an explicit order: ```sh qm set --startup order=1,up=60 ``` `up=60` waits 60 seconds before starting the next VM in the sequence. ## What I'll test when node two exists My list for the day a second node joins: - Confirm corosync opens UDP on 5405 on the overlay address on both nodes. - Watch `corosync-cfgtool -s` for link status and check it stays connected for a few days. - Measure real latency over the overlay at a busy time, not just a quiet one. - Pull the link on purpose with HA off and see what read only on both sides actually looks like. - Pick a quorum plan before turning HA on. Until then, the config is ready and the edit sticks. The link itself is untested.