Datadog Checks

Integrating with Datadog

This guide will walk you through integrating Datadog with OpsLevel Maturity Checks so that you can build them into your Service Maturity Rubric. You can also integrate OpsLevel and Datadog using OpsLevel Alert Sources to show the current monitor status in OpsLevel and receive OpsLevel notifications. To configure Datadog as an alert source, check out the Datadog Integration guide.

Add a Datadog Check Integration

1. In the OpsLevel app, Click Integrations in the left sidebar.
2. Click on the + New Integration button.
3. Click the Datadog Check tile to add the integration.

Create a Check

1. Navigate to the Rubrics sub menu under the Service Maturity menu in OpsLevel
2. Hover over the cell that corresponds to the level and category you want your check to live in and click the + Add Check button.

1. Select the Datadog check to start creating the check (the Datadog Check integration will be automatically selected in the Integration dropdown)
2. Choose one of the Check Templates from the dropdown.

1. Here is what each template is for:
2. Check For SLO Error Budget Alert: The check will fail if the alert is triggered.
3. We will populate the Service Specifier field, which we use to determine which service to run the check for, and the Success Condition field, which we use to determine if the check should pass or fail. We also provide a sample payload to test the check.

Send payload to OpsLevel

For any Datadog metric alert, you can create a Webhook using the URL from the integration you created above. See here for more details.

If the payload you will be sending in the webhook does not have a service identifier present, we recommend appending a query parameter to your URL like the following: ?serviceAlias=my-service-alias

The minimum set of payload variables we recommend are:

{
  "id": "$ID",    
  "body": "$EVENT_MSG",
  "alertId": "$ALERT_ID",
  "eventMessage": "$TEXT_ONLY_MSG",
  "title": "$EVENT_TITLE",
  "url": "$LINK",
  "alertTransition": "$ALERT_TRANSITION",
  "hostname": "$HOSTNAME",
  "orgName":"$ORG_NAME",
  "priority":"$PRIORITY",
  "snapshot": "$SNAPSHOT",
  "alertQuery": "$ALERT_QUERY",
  "alertScope": "$ALERT_SCOPE",
  "alertStatus": "$ALERT_STATUS",
  "eventType": "$EVENT_TYPE",
  "lastUpdated":"$LAST_UPDATED"
}

To use your webhook, add @webhook-<WEBHOOK_NAME> in the Notify your team section when creating or editing a monitor alert.