Docs

Getting Started with Self Hosted OpsLevel

Suggest Edits

This guide will walk you through how to setup your self hosted OpsLevel application and the prerequisites required to start the application.

Prerequisites

  • A license file provided by OpsLevel
  • A place to host the application (VM or Kubernetes cluster). The OpsLevel Application requires the following minimum specifications:
    • 16 CPU
    • 16 GB of Memory
    • Capacity for at least 20 pods
    • Disk Storage: minimum of 24 Gi (for pilots) and 120 Gi (for production)
  • Admin access in various third party software to setup integrations

The Self Hosted OpsLevel application is mostly configured using Kubernetes. We recommend ensuring that you have an in-house expert on your internal Kubernetes capabilities when setting up your self hosted OpsLevel.

Guide to setting up your OpsLevel Application

Installation

Your OpsLevel account manager will provide your license and instructions for how to perform the installation depending on if you are using a Kubernetes Cluster (preferred) or VM to host the application.

To learn more about the installation methods that Replicated supports, check out their documentation here. During and after installation, you'll have access to the self hosted OpsLevel Admin Console and Configuration page to manage the lifecycle of the self hosted OpsLevel application. This console is powered by replicated and you can see more details by reviewing their documentation here.

Upload your license file

Once you have performed the installation and you have logged into the customer admin console, you will be asked to provide your license file before you can begin any of the configuration.

Customer Admin Console

If you have provided a valid license you will then be prompted with a configuration screen to supply any additional configuration for connecting to an external DB or integrating with third party software like GitHub, Jira, PagerDuty, etc.

📘

Pilot or Production?

In order to make it easy to get started with Pilots and Proof of Concepts, OpsLevel has provided limited default infrastructure for MySQL, Postgres, Redis and Elasticsearch. For these use cases, you can complete your installation after following the Account Configuration, SMTP Configuration and Runner Configuration sections.

Before your production rollout, we recommend configuring your own infrastructure components as well.

All self hosted OpsLevel applications will follow these steps to get started:

  1. Create an initial user and configure the domain that users will use to access the OpsLevel Application.
  2. (Optional, but recommended) Enable email by setting up the SMTP Configuration.
  3. Deploy your configuration changes and login to the OpsLevel Application.
  4. (Optional, but recommended) Setup your OpsLevel Runner configuration and redeploy your configuration.
  5. Setup your initial set of users via SSO, SCIM or manual invitations.
  6. Start integrating to build out your catalog.

Add Basic Settings via Account Configuration

The Account Configuration section allows you to configure an initial Admin user and define how your users will navigate to OpsLevel while on your network.

Configuring the initial user

The initial OpsLevel User should be considered a throwaway user, so feel free to use the default configuration we provide. You may modify the User's name, email and password if you prefer to override these settings.

Once the application is running, you can login with the initial user credentials and proceed with adding additional users to OpsLevel. You can do this by:

Before removing the initial user, ensure that you've promoted at least one of your real users to the Admin role. Once the initial user is removed, it cannot be restored via the KOTS Admin panel.

🚧

Before you invite users via the UI or API, you'll need enable your OpsLevel application to send emails by using the SMTP Configuration. Once this is configured, users will be able to receive invites and welcome emails.

See the Configuring Email via SMTP section below for more details.

OpsLevel Domain configuration and ingress details

The Account Configuration section provides Subdomain and Domain fields which are used by the application to generate URLs that enable users to navigate around the application.

By default, the OpsLevel configuration does not include creation of any Kubernetes ingress and the service resource is not exposed. So it is up to you use expose the application in whatever way your company see fit with whatever ingress methods you use.

🚧

When you expose the application via ingress we recommend that you make sure you set the proxy body and header size limits to at least 4 MB for payload and large webhook ingestion.

Configuring Email via SMTP

The SMTP Configurationsection allows you to configure SMTP settings so that the application may send emails. It is recommended that you give the application access to send email so you may receive the following types of emails:

  • User Invite
  • Password Reset
  • Weekly Report Emails for Service Owners
  • Campaign Notifications

Enabling Historical Reporting

Historical reporting for OpsLevel allows you to see changes in certain metrics and service levels over a long period of time. By default, historical reporting is not enabled in self hosted.

To enable historical reporting with self hosted OpsLevel, go to the KOTS Admin site.

In the Self Hosted OpsLevel Configuration page

Navigate to the Historical Reporting section.

Toggle on the Enable Historical Reporting Service? option.

📘

Required infrastructure for historical reporting

Historical reporting in OpsLevel requires a working Postgres and Redis to work. Self hosted OpsLevel includes an out-of-the-box container for Redis and Postgres, but they are not recommended for production and should be used by pilots only.

Configuring OpsLevel Runner

After configuring the basics, you'll want to configure an OpsLevel Runner. OpsLevel Runners are used for a variety of asynchronous jobs in OpsLevel including running Repo Grep Checks and Service Detection from Git Integrations.

To configure OpsLevel Runners with self hosted OpsLevel, first you'll need to get your OpsLevel application running,

In OpsLevel

📘

Ensure you are logged in as an Admin user

This is required to create an API token with Read and Write permissions.

  1. Navigate to Integrations > API Tokens
  2. Click Create API Token
  3. In the dialog, add a Token Description (For example, "OpsLevel Runner Token") and ensure that Read-only is unchecked. Note: if you cannot uncheck the Read-only toggle, you may not be an Admin user.
  4. Copy the Token value and store the value somewhere safe for use later. Note: the token will only be visible at this time. If you lose the value, you will need to create a new API token.

In the Self Hosted OpsLevel Configuration page

  1. Return to the KOTS Admin site and navigate to the OpsLevel Application's Config page.
  2. Navigate to the Runner Configuration section.
  3. Toggle on the Enable OpsLevel Runner? option.
  4. Enter the Token value from step 4 above into the OpsLevel Runner API Token field.
  5. (Optional) Override the OpsLevel Runner Max Replicas setting to handle more or less OpsLevel Runner Jobs. Note: You may want to increase the number of replicas if you have many repositories, large monorepos attached to many services or many Repo Grep Checks that you want to run at once.
  6. Once you are finished with the settings, click Save Config
  7. Deploy the OpsLevel Application in order to have your settings take effect. During your next deployment, additional pods will be deployed with your OpsLevel Runner configuration.

Connect to your own MySQL (Production Requirement)

📘

Note on MySQL included in installation

By default, the installation creates a MySQL container alongside the application for getting started quickly.

We do not recommend using the included MySQL container in production deployments of OpsLevel! This section provides you with the detailed configuration needed to allow the application to connect to an “external” MySQL database that has been hardened for production and is monitored by you.

The MySQL database is where all of your application data is stored. If this database is destroyed, all of the data within OpsLevel is also destroyed. To prevent data loss, we recommend you setup regular backups.

There are a handful of MySQL parameters that need to be set that we have enumerated here, but do also note that we have configured preflight checks to ensure the database OpsLevel connects to will work properly.

    [mysqld]
    init_connect='SET collation_connection = utf8mb4_unicode_ci,NAMES utf8mb4'
    character_set_server = 'utf8mb4'
    collation_server = 'utf8mb4_unicode_ci'

    [client]
    default_character_set = 'utf8mb4'

Subject to change: Our application is configured to perform any pending migrations on this database at startup. Please be aware that this is not an “online” process, and as such may lock the database for a period of time while the migrations complete, thereby preventing use of the application.

Connect to your own Postgres (Production Requirement)

📘

Note on Postgres included in installation

If historical reporting is enabled, the installation creates a Postgres container alongside the application for getting started quickly.

We do not recommend using the included Postgres container in production deployments of OpsLevel! This section provides you with the detailed configuration needed to allow the application to connect to an “external” Postgres database that has been hardened for production and is monitored by you.

The Postgres database is where all of your historical reporting data is stored. If this database is destroyed, all of the historical reporting data within OpsLevel is also destroyed. To prevent data loss, we recommend you setup regular backups.

Subject to change: as of right now, there are no specific Postgres parameters that need to be set to ensure the database OpsLevel connects to will work properly.

Connect to your own Redis (Production Requirement)

📘

Note on Redis included in installation

By default, the installation creates a Redis container alongside the application for the purpose of getting started quickly.

We do not recommend using the included Redis container in production deployments of OpsLevel!

This section provides you with the configuration needed to allow the application to connect to an “external” Redis instance that has been hardened for production and is monitored by you.

The Redis database is used to store pending asynchronous jobs waiting to be processed. If this database is destroyed all that is lost is pending jobs. For the most part many asynchronous tasks will be re-enqueued at a later time, but there are a few cases where losing the job means that specific data that was ingested will never be processed.

Connect to your own Elasticsearch Cluster (Production Requirement)

📘

Note on Elasticsearch included in installation

By default, the installation creates an Elasticsearch container alongside the application for the purpose of getting started quickly.

We do not recommend using the included Elasticsearch container in production deployments of OpsLevel!

This section provides you with the configuration needed to allow the application to connect to an “external” Elasticsearch instance that has been hardened for production and is monitored by you.

This section allows you to connect an Elasticsearch cluster to OpsLevel to power features related to improved Search and Tech Docs.

Option 1: Recommended Configuration - Bring your own Elasticsearch

OpsLevel expects that your Elasticsearch cluster version is 8.4.2 or higher. If you don't have an Elasticsearch cluster available, we recommend using a deployment provided by Elastic.co directly.

Configuration:

  1. Protocol - WARNING: always select https!
  2. Username - usually elastic
  3. Password
  4. Endpoint Hostname - something like your-endpoint-name.es.us-east-1.aws.elastic-cloud.com
  5. Port Number - usually 443 OR 9200

Option 2: Testing Only Configuration - Use the bundled Elasticsearch

OpsLevel bundles a single-node Elasticsearch cluster on version 8.8.2. As mentioned above, this is only intended to be used for testing and is not suitable for production!

Configuration:

  1. Protocol - select http (The elasticsearch-0 Pod lives on the same network as the OpsLevel web Pods)
  2. Username - elastic
  3. Password
  4. Endpoint Hostname - elasticsearch
  5. Port Number - 9200
  6. Kubernetes Storage Class - gp2 suggested
  7. Elasticsearch Storage Size (Gi) - 10 suggested

Advanced Configuration and Integrations

You'll notice that the OpsLevel configuration has many more sections than those that we've walked through above. Most of the additional settings are related to specific integrations. Many of the integrations can be configured from within the OpsLevel application using Personal Access Tokens or other authentication tokens, however some integrations require that additional applications be configured within the 3rd Party Service.

Integrations that require 3rd party applications and updates to your OpsLevel Application configuration include:

For more information about how to perform the advanced integration configurations, see our list of guides here.

Preflight Checks

Once you have completed all the needed configurations you can click Next and the preflight checks will begin to run. The purpose of these checks is to ensure the provided configuration and application hosting location have the best chance of starting up correctly. If any of the preflight checks fail you will need to fix them before completing the installation.

Updated about 1 month ago

What’s Next

Learn more about how to connect to your self hosted OpsLevel application through our tooling or integrations