The Checkmk REST API turned off TLS checks on my alert webhook

I keep a Checkmk notification rule that posts alerts to a Discord webhook. I wanted that webhook call to verify TLS, so I set it through the REST API instead of clicking through the UI. The API took the change. Then the webhook ran with verification switched off.

checkmk: 2.5.0p17 Community, in a container
api: REST API v1
notifications: built-in slack plugin, Discord webhook

What the PUT actually did

The request was a PUT on the rule, with the TLS option set the way the schema describes:

curl -X PUT \
  -H "Authorization: Bearer <automation-user> <secret>" \
  -H "Content-Type: application/json" \
  -H "If-Match: <etag>" \
  "https://mon-01.example.lab/<site>/check_mk/api/v1/objects/notification_rule/<id>" \
  -d @rule.json   # ... "disable_ssl_cert_verification": {"state": "disabled"} ...

It didn't update the parameter set the rule was using. It made a new one, named "Auto-generated during rule creation via the REST API", put 'ignore_ssl': True in it, and pointed the rule at the copy. The original set was still there, untouched. The UI showed a working rule. Every alert after that went out unverified.

Nothing failed, so nothing told me.

Why "disabled" still means ignore

Two pieces of the notification code line up badly. The plugin helper in notification_plugins/utils.py turns verification off when the key exists in the context:

if "PARAMETER_IGNORE_SSL" in context:

It never looks at the value. And add_to_event_context writes even a None value into the context, as an empty string. So once the parameter set has the key at all, the plugin sees it and skips the check.

Discord doesn't get a pass here. The slack plugin only builds a Slack-shaped message when the webhook host ends in slack.com. Everything else gets the Mattermost-shaped payload, which is what Discord's /slack endpoint accepts. Both go out through the same helper, so the same check applies.

Measured: the duplicate parameter set, the ignore_ssl entry in it, and the rule pointing at it. Read from the source: the key-presence check. I didn't trace every code path between them.

Fixing it in the files

I stopped using the API for notification rules. I edit the two files directly, with a backup, then compile:

cd /omd/sites/<site>
cp etc/check_mk/conf.d/wato/notifications.mk notifications.mk.bak-$(date +%F)
cp etc/check_mk/conf.d/wato/notification_parameter.mk notification_parameter.mk.bak-$(date +%F)
cmk -O

Then the check I run after any notification change:

grep -rn ignore_ssl etc/check_mk/conf.d/wato/

No output is the answer I want.

Hand editing has its own trap on 2.5. My rule held its plugin parameters as an inline list, and cmk-validate-config rejected it with "Input should be a valid string". 2.5 wants the rule to name a parameter set by its ID, with the parameters living in notification_parameter.mk. The list still worked at runtime through a fallback, which is why alerts never broke. The next upgrade might not be so forgiving, so I converted it and re-ran the validator until it passed.

One more thing I missed: 2.5 has no cmk --notify analyse. There's replay, but it really sends the notification. Only test with it if a real message is fine.

A 404 that looked like zero pending changes

When I scripted host removal, the API fooled me a second way. I'd guessed the activation endpoint as /domain-types/activate_changes/.... That path doesn't exist and returns 404. My script read the pending list with len(d.get("value", [])). A 404 body has no value key, so the script printed "0 pending changes" and looked like it had worked.

It hadn't. The real paths are under activation_run:

GET  /domain-types/activation_run/collections/pending_changes
POST /domain-types/activation_run/actions/activate-changes/invoke
GET  /objects/activation_run/<id>/actions/wait-for-completion/invoke

Activation needs an If-Match header. The docs pass the ETag that comes back with the pending changes list. If-Match: * also worked on my site, and leaving the header out got me 428 Precondition Required. The sequence that works for me:

# 1. delete the host (etag from a GET on the host first), expect 204
curl -X DELETE -H "If-Match: <etag>" ... /objects/host_config/<host>

# 2. activate, expect a run id back
curl -X POST -H "If-Match: *" -H "Content-Type: application/json" ... \
  /domain-types/activation_run/actions/activate-changes/invoke \
  -d '{"redirect": false, "sites": ["<site>"], "force_foreign_changes": true}'

# 3. wait, expect 204
curl ... /objects/activation_run/<id>/actions/wait-for-completion/invoke

A note on force_foreign_changes. By default the API only activates changes made through the API by the same user. Setting it to true activates everyone's pending changes, including half-finished work someone left in the UI. I'm the only one on this site, so true is fine for me. On a shared site I'd leave it false.

My script checks the status code on every call now, and fails on anything it didn't expect. I get endpoint paths from the site's own /openapi-doc.json instead of guessing them.

What I'd tell myself

The API saying yes doesn't mean the config says what I sent. After a scripted change, read the generated files and the status code. If it touched notifications, grep for ignore_ssl. It takes a second.