> ## Content Index
> Fetch the complete content index at: https://blog.coopnet.us/llms.txt
> Use this file to discover other available public pages before exploring further.

# Stop Ansible hanging forever on a host that is "up"
- URL: https://blog.coopnet.us/stop-ansible-hanging-on-a-reachable-host/
- Published: 2026-07-17T06:00:00.000Z
- Updated: 2026-10-18T02:30:47.000Z
- Description: A host answers pings and a bare ssh, but Ansible freezes on it indefinitely and takes the whole run down. Here is why ConnectTimeout and wait_for_connection do not save you — and how to hard-bound every SSH.
- Author: Renstur

One host in a fleet run wedged the entire play for nine minutes before I gave up and killed it. The host wasn't down — it answered pings, and `ssh host-b true` returned instantly. But the moment Ansible tried to do real work on it, everything froze. And with `serial: 1`, one stuck host means the whole run is stuck.

Here is what is actually happening, and how to make it impossible.

## The failure mode

The host was in a half-up state: reachable at the TCP layer, and the SSH *handshake* completed — you even get the friendly `Warning: Permanently added 'host-b' to the list of known hosts` — but the actual command and module execution hung. This shows up when an SSH daemon accepts connections but can't spawn a session, or, in my case, when the control connection runs *over a mesh VPN tunnel* that dropped mid-session: the socket is `ESTABLISHED` but silently black-holing packets.

Your instinct is "surely there is a timeout for this." There is. It just does not cover this case:

- **`ansible_timeout` / `ConnectTimeout`** bounds the *TCP connect* only. The connect already succeeded, so this never fires.
- **`ServerAliveInterval` / `ServerAliveCountMax`** only begin counting *after the session is established*. A hang during key exchange or command exec sits before or around that window, so they do not reliably kick in.
- **`wait_for_connection`'s `timeout`** feels like the answer, but it is checked *between* connection attempts. If one attempt hangs, the loop is stuck inside that attempt and never gets to evaluate the timeout.

So nothing at the Ansible layer bounds a hung `ssh` invocation. The bound has to come from *outside* the ssh process.

## The fix: wrap ssh in `timeout`

Point Ansible at a wrapper that runs `ssh` under coreutils `timeout`, so no single ssh call — connect, handshake, auth, or module transfer — can outlive it:

```yaml
- name: Fleet run
  hosts: all
  become: true
  serial: 1
  gather_facts: false
  ignore_unreachable: true
  vars:
    ansible_timeout: 30
    ssh_timeout_wrapper: /tmp/ssh-timeout.sh
    ansible_ssh_executable: "{{ ssh_timeout_wrapper }}"
    # ControlMaster=no keeps each ssh a standalone process the timeout can cleanly kill
    ansible_ssh_common_args: >-
      -o ServerAliveInterval=10 -o ServerAliveCountMax=3
      -o ControlMaster=no -o ControlPath=none
  pre_tasks:
    - name: Install the timeout-wrapped ssh on the controller
      delegate_to: localhost
      become: false
      ansible.builtin.copy:
        dest: "{{ ssh_timeout_wrapper }}"
        mode: "0755"
        content: |
          #!/bin/sh
          exec timeout -k 5 60 ssh "$@"

```

That is the whole trick. `ansible_ssh_executable` tells Ansible to invoke `/tmp/ssh-timeout.sh` instead of `ssh`; the wrapper just re-execs `ssh` under `timeout -k 5 60` — SIGTERM at 60 seconds, SIGKILL five seconds after that if it is really stuck. A wedged host now fails its task within about a minute instead of hanging forever. Pair it with `ignore_unreachable: true` and a `block`/`rescue` around your tasks, and the run logs the bad host and moves on instead of dying.

## Details that bite

- **Set the timeout above your slowest *legitimate* synchronous task.** An `apt update` against a slow mirror can hold one SSH session for 30–40 seconds. 60 is a safe default; 30 will start killing real work.
- **Disable `ControlMaster`.** Ansible's default SSH multiplexing keeps a persistent master connection alive. If `timeout` kills that master mid-life, other multiplexed sessions break in confusing ways. `ControlMaster=no` makes each ssh a clean, independently killable process. You give up a little speed and get determinism.
- **Pipelining helps here.** With `pipelining = True` in `ansible.cfg`, the module is piped over that single wrapped ssh, so the module transfer is bounded too — no separate `sftp`/`scp` connection sneaking past the timeout.
- **The wrapper lives on the *controller*.** `delegate_to: localhost` writes it wherever Ansible runs — your laptop, a CI runner, a Semaphore box — never on the targets.

## But shouldn't you just fix the host?

Yes — a host that hangs SSH is broken and deserves attention. But fleet automation should never be one bad node away from a nine-minute hang, let alone an indefinite one. This wrapper converts "the whole run is stuck" into "one host got skipped, and here is which one" — which is exactly the failure mode you want at 3 a.m.