Set up webhook notifications

Send CosmicAC job and model health events to an HTTPS endpoint, including a Slack webhook.

Send CosmicAC events to an HTTPS endpoint that you control. CosmicAC posts a payload when a job fails, degrades, or recovers, and when a served model's health changes. One webhook covers the whole deployment.

Prerequisites

Before you start, make sure that you have the following.

A running CosmicAC deployment. See Set up CosmicAC.
The platform administrator role. See Teams and roles.
An HTTPS endpoint that accepts POST requests. For Slack, create an incoming webhook, and then copy its URL.

Steps

Open the Notifications page

In the left navigation, click Settings, and then under Instance, click Notifications.

Enter the webhook URL

In Webhook URL, enter the HTTPS URL that receives the events. Don't click Save until you select the payload format.

Select the payload format

Under Format, select one of the following formats.

Generic JSON: a flat event object carrying id, type, schema, timestamp, instance, and the fields for that event type.
Slack-compatible: a Slack incoming webhook body carrying text and a color-coded attachments block. Red marks a failure, green a recovery, and orange a degraded state.

Payload Preview shows a sample payload in the selected format.

Select Slack-compatible when the URL points to Slack. CosmicAC posts the body straight to the incoming webhook URL, so it needs no Slack workspace credentials, bot token, or OAuth 2.0 authorization.

Save the webhook

Click Save.

When you save a webhook for the first time, CosmicAC does the following.

Turns on every event under Events.
Generates the signing secret and shows it on the page.

Copy the signing secret

Copy the signing secret, and then store it in your receiver's configuration. CosmicAC shows the secret only once, and your receiver needs it to verify signatures.

Turn off the events that you don't need

Under Events, turn off each event that you don't want CosmicAC to send, and then click Save. CosmicAC sends only the events that are on.

The four job lifecycle events also need the matching preference on the job that raises them. For more information, see What controls delivery.

Event	Fires when
job.failed	A job transitions to Failed, and the event carries the failure reason.
job.degraded	Healthy replicas drop below the count you set, and the endpoint stays live.
job.recovered	A job returns to Active from Degraded or Failed.
job.restart_storm	Any replica restarts three times within 10 minutes.
model.health.down	A served model's health enters Down.
model.health.recovered	A served model's health leaves Down.

Send a test event

Click Send test event. CosmicAC posts a test.ping payload to the saved URL.

Confirm the delivery

Check Recent Deliveries. Each row lists the event type, the target URL, the response status, and how long ago CosmicAC sent it. A successful delivery shows the response code, such as 200 OK.

A delivery counts as failed when the receiver answers with a status outside the 2xx range, or doesn't answer within 10 seconds. CosmicAC retries a delivery that fails with a 5xx status, a 429 status, a network error, or a timeout. It retries up to three times, after 1, 2, and 4 seconds. It doesn't retry any other status, and it discards the event after the last attempt.

Verify the signature on your receiver

Verify signatures when your receiver is a service you run. A Slack incoming webhook can't run verification code, so it ignores the signature header.

Your receiver needs the signing secret to confirm that a payload came from CosmicAC. CosmicAC shows the secret once, when you first save the webhook URL. If you didn't copy it then, click Regenerate to issue a new one.

CosmicAC sends the following headers.

X-Cosmic-Event: the event type, such as job.failed.
X-Cosmic-Delivery: the unique event identifier. Use it to discard duplicates.
X-Cosmic-Timestamp: the send time in milliseconds.
X-Cosmic-Attempt: which attempt this delivery is, starting at 1.
X-Cosmic-Signature: sha256= followed by the HMAC-SHA256 digest of the raw request body, keyed with the signing secret. CosmicAC sends this header, and the identical X-Cosmic-Signature-256, only when a signing secret exists.

Recompute the digest over the raw body and compare it before you trust a payload.

const crypto = require('crypto')

const expected =
  'sha256=' +
  crypto
    .createHmac('sha256', process.env.COSMICAC_WEBHOOK_SECRET)
    .update(rawBody)
    .digest('hex')

const signature = req.headers['x-cosmic-signature']

const valid =
  typeof signature === 'string' &&
  expected.length === signature.length &&
  crypto.timingSafeEqual(
    Buffer.from(expected),
    Buffer.from(signature)
  )

What controls delivery

Every job has four notification preferences, one for each lifecycle event. CosmicAC turns on all four when you create the job. To set them, see Create a GPU Container Job, Create a vLLM Managed Inference Job with the CLI, and Create a Parakeet Managed Inference Job with the CLI.

An event reaches your webhook only if both of the following settings are on.

The job's notification preference, which controls whether CosmicAC creates the event.
The event under Events on the Settings > Notifications page, which controls whether CosmicAC posts it.

If either one is off, the event doesn't reach your webhook. If you turn it on later, CosmicAC doesn't deliver the events that it already skipped.

CosmicAC delivers each event that it creates to two destinations.

Destination	Requirements
The notification bell in the web interface	None. CosmicAC shows every event it creates.
Your webhook	A saved webhook URL, and the event turned on under Events on the Settings > Notifications page.

The Events selection applies only to the webhook. If you turn off an event under Events, it still appears in the notification bell.

The CLI and the web interface use different names for the same event. Pass the underscore form, such as job_failed, to cosmicac jobs create --notify. The web interface shows job.failed.

The model.health.down and model.health.recovered events have no per-job preference, because no job raises them. The Events selection is the only setting that controls them.

Rotate the signing secret

Click Regenerate to replace the signing secret. Copy the new secret before you leave the page, because CosmicAC shows it only once. The previous secret stops working immediately, so update your receiver in the same maintenance window.