Custom Event Checks

OpsLevel helps keep your services consistent and healthy through the use of various pre-built checks. However, not all checks fall in line with one of the pre-built checks provided by OpsLevel.

With custom event checks, you can easily integrate with any third party platform to create custom, data driven checks. The result message has the full context which can be templated to provide service owners with an informative message.

Custom event checks allow you to evaluate various conditions by sending any JSON payload to OpsLevel. The following are examples of conditions you can evaluate using these checks:

• Using your vulnerability detection tool to ensure that no Tier-1 services have open vulnerabilities.
• Using your vulnerability detection tool to ensure no Ruby services have high criticality vulnerabilities.
• Ensuring your test coverage stays above a certain threshold.

🚧

Size Limit

Payloads have a size limit of 4MiB, payloads above this size will received a HTTP 413 response

Getting Started with Custom Event Checks

Step 1: Create a Custom Event Integration

To set up a custom event check endpoint, visit the Integrations tab and select “New Integration”, followed by Custom Event.

Click \+ Add Integration on the Custom Event Integration card, enter the name of the system you are integrating with (e.g. PagerDuty) and click “Create”.

After creating a Custom Event Integration, you’ll be redirected to a page that looks like:

Step 2: Create a Custom Event Check

To create a custom event check, navigate to the Rubrics tab

Either in the top right corner, or inside your rubric, press the + Add Check button.

From the Create Check modal, select Custom Event Check from the Type dropdown:

Select the Integration you previously created, it’s URL will be displayed below the dropdown. This is where you should be sending your JSON payloads for the check.

To identify your service, or services, in the sent JSON payload, you can specify a Service Identifier jq expression that will find your service aliases in the payload or query params. Refer to the service identifier section for more information and examples.

To evaluate the success of your check, specify a Success Condition jq expression that will be run against your payload to extract a boolean value to determine whether your check is passing or failing. Refer to the success condition section for more information and examples.

You can enter a Result Message to provide feedback to users regarding the success or failure of your check. The message template supports Markdown and Liquid templating. For more information and examples, refer to the Liquid Templating section.

In the Test Check section, you can test your check before creating it. Enter a sample JSON payload that resembles what you’ll be sending to the custom check endpoint and your inputs will be used to evaluate your expression, display the found services, and render your Result Message.

Note: Only the check for the first found service in your payload will be evaluated for the test.

After successfully creating a check, you will see it in your rubric.

Step 3: Sending a JSON Payload to OpsLevel

Open one of your Custom Event Checks in your rubric, and copy the URL present in the modal. This is the endpoint you’ll be sending your JSON payloads to.

Using that endpoint, make an HTTP POST request like the example below, substituting your custom check endpoint URL in place of xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxx:

curl -X POST https://app.opslevel.com/integrations/custom_event/xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxx \
-H 'content-type: application/json' \
-d '
{
  "messages": [
    {
      "event": "incident.trigger",
      "incident": {
        "incident_key": "12",
        "status": "resolved",
        "service": {
          "id": "service1",
          "name": "Production XDB Cluster",
          "description": "This service was created during onboarding on July 5, 2017.",
          "auto_resolve_timeout": 14400
        }
      },
      "id": "69ad-11e7-a799-22000a15839c7ced0-a2c",
      "created_on": "2017-09-26T15:14:36Z"
    },
    {
      "event": "incident.trigger",
      "incident": {
        "incident_number": 33,
        "status": "resolved",
        "service": {
          "id": "service2",
          "name": "Production XDB Cluster",
          "description": "This service was created during onboarding on July 5, 2017.",
          "auto_resolve_timeout": 14400,
          "acknowledgement_timeout": 1800,
          "created_at": "2017-07-05T17:33:09Z",
          "status": "critical",
          "last_incident_timestamp": "2017-09-26T15:14:36Z"
        },
        "id": "69a7ced0-99-22000a15839ca2cd-11e7-a7",
        "created_on": "2017-09-26T15:14:36Z"
      }
    }
  ]
}

If your request was successful, you will receive the following response:

{ "result": "ok" }

Examples

Single Service Payload: Check for Vulnerabilites in yarn.lock using Snyk

We can use Snyk to find security vulnerabilities in our projects. Specifically we’ll assess our yarn.lock file and send the results to OpsLevel.

snyk test --prune-repeated-subdependencies --json --file=yarn.lock | curl -X POST http://app.opslevel.com/integrations/custom_event/xxxxxxxx-xxxx-xxxx-xxxxxxxxxxxx -H 'content-type: application/json' --data-binary @-

We use the Snyk CLI to assess our yarn.lock file and pipe the JSON output to a curl command which POSTs to our custom event endpoint.

Note: We use the \--prune-repeated-subdependencies flag to compress our Snyk payload due to our 4MiB payload limit.

The payload sent by Snyk will look something like this:

{
  "vulnerabilities": [
      {
        "severity": "medium",
        "severityWithCritical": "medium",
        "title": "Regular Expression Denial of Service (ReDoS)",
        "from": [
          "app"
        ],
        "upgradePath": [],
        "isUpgradable": false,
        "isPatchable": false,
        "isPinnable": false,
        "name": "lorem ipsum",
        "version": "4.5.394"
      },
      {
        "severity": "high",
        "severityWithCritical": "high",
        "title": "Regular Expression Denial of Service (ReDoS)",
        "from": [
          "app"
        ],
        "upgradePath": [
          false
        ],
        "isUpgradable": true,
        "isPatchable": false,
        "isPinnable": false,
        "name": "lorem ipsum",
        "version": "6.0.1"
      }
  ],
  "filesystemPolicy": false,
  "filtered": {
    "ignore": [],
    "patch": []
  },
  "uniqueCount": 2,
  "projectName": "app_name",
  "foundProjectCount": 1,
  "displayTargetFile": "yarn.lock",
  "path": "/path/to/project"
}

For our check, we’ll write a Service Identifier to parse the service alias from the JSON payload.

.projectName

We’ve identified our service, now we can write our Success Condition for the payload.

([.vulnerabilities[] | select(.severity == "high")] | length == 0) and ([.vulnerabilities[] | select(.severity == "medium")] | length <= 3) < code>

Here we’re searching inside the vulnerabilites array for items with high and medium severities, and checking if there are three or less.

Finally, we’ll craft our result message using a combination of Markdown and Liquid templating.

{% if check.passed %}
  ### Check passed
{% else %}
  ### Check failed

  service **{{ data.projectName }}** has **{{ data.vulnerabilities | where: "severity", "high" | size }}** high vulnerabilities and **{{ data.vulnerabilities | where: "severity", "medium" | size }}** medium vulnerabilities.

It has **{{ data.uniqueCount }}** vulnerabilities in total.
{% endif %}

Multi-Service Payload: Check for PagerDuty Triggered Incidents

In PagerDuty, you can specify a webhook endpoint to send events that happen in your account, this would be the URL to your custom check integration.

We’ll write a Service Identifier to parse your service aliases from the JSON payload.

.messages[] | .incident.service.id

Now that we have our services identified we can work on writing a Success Condition for the payload, scoped down to the section relevant to each found service.\

.messages[] | select(.incident.service.id == $ctx.alias) | .incident.status == "resolved"

Here we are finding each service in the list of messages, and passing the check if the incident has been resolved. The other possible values, acknowledged or triggered, will fail the check.

Finally, we’ll craft our result message using a combination of Markdown and Liquid templating.

{% if check.passed %}
  ### Check passed
{% else %}
  ### Check failed
  **{{ data.messages[ctx.index].incident.service.id }}** has an unresolved incident.
{% endif %}

Using URL Query Params: Check for Number of Issues in Sentry

In Sentry you are able to specify a webhook to receive event payloads. To identify the service, we will specify it using query params.

In Sentry, navigate to your project and select Settings > Integrations, then search for “Webhook”. Select it and add it to the project of interest.

In your Sentry project you can configure your Sentry project to send webhooks. In this example we will configure it to send payloads to the custom event integration.

Once this is configured Sentry will begin sending webhooks to OpsLevel. For our check we will use the $params variable in our service specifier.

$params.alias

A completed check will look like:

Custom Check Fields

Service Identifier

When crafting service identifier expression, OpsLevel provides access to your URL query params to help route your check in the scenario where your payload does not include a service alias.

Success Condition

When crafting your success condition, OpsLevel provides access to variables to help specify which part of the payload is relevant to each found service.

Another way of looking at it is that $ctx is an object:

{
  "alias": "billing",
  "index": 2,
  "count": 5
}

Liquid Templating

When creating custom result messages with Liquid, OpsLevel provides access to variables to help in crafting helpful messages.

Note that the variables in liquid templating do not include a dollar sign ($) at the front.

Liquid Filters

Filters are simple expressions you can use to transform text and variables. For example, to generate the string filtrd txt, you can write:

{{ "Filtered TEXT" | downcase | remove: "e" }}

In addition to the default Liquid filters you can use these filters:

Example: Display the Number of High Vulnerabilities on your Service’s Repositories

In this example, a user wants to evaluate if there are any high criticality vulnerabilities on their service’s repo. They can send a simple JSON payload to OpsLevel:

{
  "vulns": {
    "high": 0
  }
}

If there are any high criticality vulnerabilities, we want to show the number of such vulns. Otherwise, if there are zero vulns, we want to show a “Congratulations” message.

The result message template looks like this:

{% if check.passed %}
## Congratulations!
{% else %}
## Critical Vulnerabilities Detected
You have {{ data.vulns.high }} high vulnerabilities.  
{% endif %}

Pass Result Message

Fail Result Message

Example: Display Detailed Vulnerability Information about your Service’s Repositories with Snyk

In this example, a user again wants to evaluate if there are any high criticality vulnerabilities on their service’s repo. However, in this case, the JSON payload contains additional detail about each vulnerability that can be shown on the check result message.

Using the Snyk API, a user can get detailed information about each vulnerability and send it to OpsLevel:

{
  "results": [
    {
      "issue": {
        "title": "Denial of Service (DoS)",
        "url": "https://snyk.io/vuln/SNYK-RUBY-RAILS-1071903",
        "severity": "high"
      }
    },
    {
      "issue": {
        "title": "Cross-site Scripting (XSS)",
        "url": "https://snyk.io/vuln/SNYK-RUBY-RAILS-536099",
        "severity": "medium"
      }
    },
    {
      "issue": {
        "title": "SQL Injection",
        "url": "https://snyk.io/vuln/SNYK-RUBY-RAILS-472693",
        "severity": "medium"
      }
    }
  ]
}

The user can add the following jq expression to check if there are any high vulnerabilities:

[ .results | .[] | select(.issue.severity == "high") ] | length == 0

Then the user can use the following Liquid template to show a result message with information about how to resolve each vulnerability:

{%- if check.status == 'passed' %}  
  **Congrats**. No high criticality vulns found.
  {%- assign medium_vulns = data.results | map: "issue" | where: "severity", "medium" %}
  {%- if medium_vulns.size > 0 %}
    Want to be awesome? Fix these medium ones:
    {%- for vuln in medium_vulns %}
- [{{vuln.title}}]({{vuln.url}})
    {%- endfor %}
  {%- endif %}
{%- else %}
  ### Action Required: You have critical security vulns.
  Please fix the following immediately:
  {%- assign high_vulns = data.results | map: "issue" | where: "severity", "high" %}
  {%- for vuln in high_vulns %}
- [{{vuln.title}}]({{vuln.url}})
  {%- endfor %}
{% endif %}

Pass Result Message

Fail Result Message

Liquid Error Messages

When authoring templates in Liquid, you may experience one of the following Liquid related error messages. OpsLevel uses strict variables and strict filters in Liquid template rendering.

Importing and Exporting Custom Event Checks

Another way that custom event checks can be created is through our Import Check feature.

Exporting

Custom event checks can be defined using a YAML file. You can export a YAML file directly from any of your custom event checks by clicking on Download Check Config from the dropdown menu of the check card.

From here, you will have the option to either download or copy the YAML file that defines that specific check.

Importing

To create a custom event check from a YAML file, click on the Import Check button from the Rubric page.

Here you can import or directly paste a YAML file into the text box. After importing, the create check modal will be opened and populated with the check details specified in the YAML. You will need to make sure to specify the integration the check belongs to as well as the Level and Category the check should belong to before you can create the check.

Example of a Custom Event Check Config

version: 1
kind: generic
spec:
  name: Custom Event Check
  serviceSelector: .messages[] | .incident.service.id
  successCondition: .messages[] |   select(.incident.service.id == $ctx.alias) | .incident.status == "resolved"
  message: |
    {% if check.passed %}
      ### Check passed
    {% else %}
      ### Check failed
    {% endif %}
  samplePayload: '{
  "messages": [
    {
      "incident": {
        "status": "triggered",
        "service": {
          "id": "service1"
        }
      }
    },
    {
      "incident": {
        "status": "resolved",
        "service": {
          "id": "service2"
        }
      }
    }
  ]
}
'
  sampleQueryParams: alias[]=service1&alias[]=service2
  

The following table defines the properties supported in a custom event check YAML: