Setting Up a Pull Configuration for Your Customizable Data Mapping
This document walks through a sample extraction definition, with the intent of highlighting some of the additional available features.
The extraction definition contains a number of features that empower our users to better control how their data enters and persists within OpsLevel. The features that will be covered within the context of this example are intended to highlight and clarify the behaviour of the following extraction definition keys:
http_polling
iterator
exclude
expires_after_days
In this example, we will be configuring an extraction definition to poll the Jira API for Jira issues. This definition will also assist us with managing which issues persist within OpsLevel.
Prerequisites
As this is a more detailed look into the extraction definition, the expectation is that a reader of this guide has perused its parent document Mapping Integration Data to Custom Properties. The following items should already be configured if our reader wishes to roughly follow along:
- A simple transform mapping should be in place for this example to run on its own. The following is what will be used in this example. This transform describes a Jira issue, identified within OpsLevel by its key. This component will contain a property "status" that contains the status of any imported issue data.
---
transforms:
- opslevel_identifier: ".id"
external_kind: jira_issue
opslevel_kind: jira_issue
on_component_not_found: create
properties:
status: '.fields.status.name'
- A pre-existing component type that matches the
opslevel_kind
key. I have configured a Jira Issue, to match the transform above.

- An existing endpoint for OpsLevel to poll. This will take the form of the fictional company Saurons Kitchen's Jira API.
The Extraction Definition
The following extraction definition will be used to extract Jira issues into OpsLevel.
extractors:
# Extract Jira issues for tracking technical debt and incidents
- external_kind: jira_issue
http_polling:
url: "https://sauronskitchen.atlassian.net/rest/api/3/search?jql=project=TECH AND labels=technical-debt&startAt={{ cursor | default: '0' }}&maxResults=50"
method: GET
headers:
- name: Authorization
value: "Bearer {{ 'jira_api_token' | secret }}"
- name: Accept
value: "application/json"
- name: Content-Type
value: "application/json"
errors:
# Jira returns 400 for invalid JQL queries
- status_code: 400
matches: '.errorMessages[] | contains("Invalid JQL")'
handler: no_data
# Handle rate limiting from Atlassian
- status_code: 429
handler: rate_limit
next_cursor:
from: payload
value: ".startAt + .maxResults | if . >= .total then null else tostring end"
iterator: ".issues"
external_id: ".key"
exclude: '.fields.status.name == "Done" or .fields.status.name == "Closed"'
expires_after_days: 30
This extraction definition is far more complex than the base example abes it implements the following parameters, increasing the complexity of the integration & the control a user has over their data:
- The
http_polling
key tells OpsLevel that it will need to poll for integration data instead of waiting for it to be pushed - The
iterator
key tells OpsLevel that the objects we want to extract that are of theexternal_kind
jira_issue
will be in the form of an array, and can be found under theissues
key within all received payloads. - The
exclude
key allows a user to define what data will persist within OpsLevel. This will act as a form of deletion, items that are explicitly excluded will be deleted from OpsLevel. - The
expires_after_days
key specifies how long data within OpsLevel will be permitted to persist without an update.
Let's take a closer look at these parameters & explore the impact they will have on the integration.
http_polling
http_polling
Enabling http_polling
allows OpsLevel to go get your data instead of you sending it to us. This is ideal for customers who don't want to set up some sort of cronjob on their side to push data or for an integration with a third party that does not send a webhook when data changes. HTTP polling comes with a number of features, almost all of which have been used in the example extraction definition.
url
url
This is the endpoint that OpsLevel will hit. This required parameter is a string that supports liquid templating . Alongside having access to OpsLevel Secrets, this url
also has access to cursor
, a special parameter made available through the use of the next_cursor
parameter which will be highlighted below. This will allow OpsLevel to make use of your API's pagination solution, if one exists.
method
method
The HTTP method to be used with the url
. This required parameter is an enum, and can be either GET
or POST
.
http_polling:
url: "https://sauronskitchen.atlassian.net/rest/api/3/search?jql=project=TECH AND labels=technical-debt&startAt={{ cursor | default: '0' }}&maxResults=50"
method: GET
In the example, OpsLevel will be making a paginated GET request to the Jira issues endpoint.
headers
headers
The headers that are to be included in each request. This optional parameter is a list of objects, with values that support liquid templating. As seen in the example, we have the Accept
and Content-Type
headers, alongside an Authorization
header. This header is retrieving an API key from OpsLevel Secrets to authenticate the requests OpsLevel will make. In our example we can see that we are sending a Bearer authentication header.
http_polling:
headers:
- name: Authorization
value: "Bearer {{ 'jira_api_token' | secret }}"
- name: Accept
value: "application/json"
- name: Content-Type
value: "application/json"
errors
errors
This optional parameter is used to define explicit error handling behaviour that OpsLevel will take when an error is encountered while polling your endpoint. The errors parameter accepts a list of objects containing the following parameters:
- The
status_code
required parameter acts as a filter, & will catch all error responses with a matching code - The
matches
optional parameter acts as a filter as well & will only be checked against if thestatus_code
is a match. This parameter accepts a JQ string & offers a more granular approach to error handling. - The
handler
required parameter is an enum that defines the action OpsLevel will take. Options are eitherno_data
orrate_limit
.no_data
resolves in a quiet end to the request, andrate_limit
results in a rate limit error being surfaced within OpsLevel.
Requests that result in errors that are not explicitly handled will be surfaced on the integration. From the example, the configuration is terminating requests that result in 400 errors with "Invalid JQ" in the error message. 429 errors will use the rate_limit handler which will retry the requests with backoff. All other errors will be surfaced on the integration.
http_polling:
errors:
# Jira returns 400 for invalid JQL queries
- status_code: 400
matches: '.errorMessages[] | contains("Invalid JQL")'
handler: no_data
# Handle rate limiting from Atlassian
- status_code: 429
handler: rate_limit
next_cursor
next_cursor
The optional parameter next_cursor
is used to indicate that OpsLevel will be polling a paginated endpoint. Use of this parameter enables the url
to have access to the cursor
variable. This parameter contains the following:
- The
from
required parameter is an enum that accepts the valuespayload
orheader
. This indicates the origin of where the next cursor information can be found - The
value
required parameter accepts a JQ string that can be used to evaluate the object selected by thefrom
parameter to identify what the next cursor is.
In our example, the next cursor value can be calculated from the payload, and is found by summing the startAt & maxResults field. If that sum is less than the total, then that value is converted to a string and provided as the value of the cursor
variable.
http_polling:
next_cursor:
from: payload
value: ".startAt + .maxResults | if . >= .total then null else tostring end"
As a result of only this configuration, OpsLevel will now poll the provided endpoint daily with behaviours as defined above.
iterator
iterator
Enabling the optional iterator
parameter allows OpsLevel to extract multiple data objects from a single chunk of data. The iterator
parameter accepts a JQ string and is used to indicate to OpsLevel that the data we have received has to be decomposed into the data we wish to represent within OpsLevel. This JQ string upon evaluation, needs to result in an array. In our example, we define the iterator for the external_kind
jira_issue
as follows:
http_polling:
iterator: ".issues"
This indicates that our data can be found by iterating over the issues key in our response data. A sample of response data that we would expect to parse with the above iterator would be this:
{
"isLast": true,
"issues": [
{
"key": "c53bd80c-a56d-46c8-8835-26dc9db6be5d",
"id": "APB-1001",
"fields": {
"status": {
"name": "Open"
}
}
},
{
"key": "d60a2bcf-7f9b-464b-a90e-89fc9f582e3a",
"id": "APB-1002",
"fields": {
"status": {
"name": "In Progress"
}
}
}
]
}
The result of which would be two Jira issues with unique IDs, all originating from the issues array in the payload.
exclude
exclude
The exclude
optional parameter accepts a JQ string and acts as a control for data persistence. Entities that are caught in the filter defined by the expression are deleted from OpsLevel automatically. A simple use case can be seen in our sample exclusion parameter.
http_polling:
exclude: '.fields.status.name == "Done" or .fields.status.name == "Closed"'
This exclusion rule will catch any issues that have a "Closed" or "Done" status. That means that any incoming Jira issue that matches the filter defined by exclude will be omitted. That also means that any existing Jira issue within OpsLevel that has an external_id
matching an incoming external_id
(indicating that they are the same issue) will be deleted.
expires_after_days
expires_after_days
The optional expires_after_days
parameter accepts an integer, and acts as a catch-all for stale data. Users may use this parameter to clean up their catalog data. All objects within OpsLevel of the external_kind
type that have gone expires_after_days
many days without being synced will be destroyed.
http_polling:
external_kind: jira_issue
expires_after_days: 30
In this example, all jira_issue
objects that have not been synced within the last 30 days will be deleted.
Updated about 18 hours ago