Docs

Getting Started with Self Hosted OpsLevel

Suggest Edits

This guide will walk you through how to setup your self hosted OpsLevel application and the prerequisites required to start the application.

Prerequisites

  • The helm CLI installed
  • Credentials for the OpsLevel Helm registry (provided to you by OpsLevel account manager)
  • A Kubernetes cluster to install the Helm chart to. The OpsLevel Application requires the following minimum specifications:
    • 16 CPU
    • 20 GB of Memory
    • Capacity for at least 20 pods
    • Disk Storage: minimum of 120 Gi (less for when production database are hosted elsewhere)
  • Admin access in various third party software to setup integrations

The Self Hosted OpsLevel is a cloud native Kubernetes based application so we recommend ensuring that you have an in-house expert on your internal Kubernetes cluster when setting up your self hosted OpsLevel.

Guide to setting up your OpsLevel Application

Installation

Your OpsLevel account manager will provide you with the credentials needed to access the Helm registry. To login to the registry use the following:

helm registry login registry.replicated.com --username XXXXXXX --password XXXXXXX

After you have logged into the registry you can perform the install using:

helm install --namespace opslevel opslevel oci://registry.replicated.com/opslevel/helm/opslevel

The helm chart is setup with sane defaults that should work for every Kubernetes cluster but if you find a problem please report it to your OpsLevel account manager so we can get a fix put in place.

📘

Pilot or Production?

In order to make it easy to get started, OpsLevel has provided in cluster pods for MySQL, Postgres, Redis, Elasticsearch and Object Storage.

Before your production rollout, we recommend configuring your own infrastructure components and setting the values for using those external resource that are hardened for production usecases. See below additional documentation about bringing your own datastores.

If you want to view the default values.yaml to see whats configurable you can see the chart here.

If you would like to upgrade your installation to the latest version you can simply use (if you configured your release with a values.yaml and needed to make changes please add the -f values.yaml argument to this command):

helm upgrade --reuse-values opslevel oci://registry.replicated.com/opslevel/helm/opslevel

Preflight Checks

There is a way to perform preflight checks before you helm install or helm upgrade the release. We leverage the utility https://troubleshoot.sh/ to perform these checks and they are optionally triggered depending on configuration settings in your values.yaml file. For example if you are bringing an external MySQL instance we will run preflight checks against it to make sure you have the correct minimum version, have set the proper collation and typeset, etc etc. You will need to install this utility here is one way but checkout the utilities documentation for other ways.

curl https://krew.sh/preflight | bash

To run the preflight checks you do (if you configure your release with a values.yaml please add the -f values.yaml argument to this command)

helm template opslevel oci://registry.replicated.com/opslevel/helm/opslevel | kubectl preflight -

This will pull down the latest chart, render out the templates and pass it to the preflight command. The preflight command will find all of the preflight checks and run them for you. You will be dropped in an interactive text gui where you will see which preflight checks have run and can inspect any of the failing tests.

Ingress & Domain Configuration

By default, the configuration does not include creation of any Kubernetes ingress and the service resource is not exposed as a node port. So it is up to you use expose the application in whatever way your company's ingress methods are.

To get access initially you can use kubectl to port forward the web pods until you have and an alternative ingress method setup.

kubectl port-forward deployment/opslevel-web 8080:http

Initial User Account

Once the application is running, you can login with the initial user credentials and proceed with adding additional users to OpsLevel. You can do this by:

Before removing the initial user, ensure that you've promoted at least one of your real users to the Admin role. Once the initial user is removed, it cannot be restored.

🚧

Before you invite users via the UI or API, you'll need enable your OpsLevel application to send emails by using the SMTP Configuration. Once this is configured, users will be able to receive invites and welcome emails.

See the Configuring Email via SMTP section below for more details.

Configuring Email via SMTP

The SMTP Configurationsection allows you to configure SMTP settings so that the application may send emails. It is recommended that you give the application access to send email so you may receive the following types of emails:

  • User Invite
  • Password Reset
  • Weekly Report Emails for Service Owners
  • Campaign Notifications

The example section of the values.yaml file for smtp creation is as follows:

smtp:
  enabled: true
  secret:
    emailDisplayName: "OpsLevel Self Hosted"
    emailDomain: "example.com"
    emailUsername: "info"
    host: "smtp.sendgrid.net"
    password: "XXXXXXXXXXXXXXX"
    port: "587"
    username: "apikey"

Bring Your Own Secrets

The helm chart is setup with functionality to allow you to bring you own secret resources instead of having to specify them in the values.yaml file provided at install time. The shape of every secret section follow the same general conventions - here is an example:

opssight:
  secret:
    create: true
    name: "opssight-secrets"
    username: "root"
    password: "admin123!"

By default this will create a secret in Kubernetes with the name opssight-secrets and fill the data section with the values provide from the values.yaml. But in the case where you want to bring you own secrets from an external secrets management tool you can turn off create and specify the name of the secret resource that will exist in the Kubernetes cluster that it should use.

opssight:
  secret:
    create: false
    name: "my-external-secrets"

This will direct the helm chart to look for the Kubernetes secret resource with the name my-external-secrets and you will need to ensure that exists in the cluster and namespace you install the helm chart to.

🚧

Currently all secrets are loaded as environment variables so if you bring your own secrets you will need to inspect the helm chart to see all the places where a secret value might be injected. An example is the redis configuration secrets get handed to a number of different deployments in the application because they all connect to the same redis instance. This will mean if you decide to bring your own secrets for redis you'll need to ensure you also bring your own secrets for the other applications as well.

In the future we hope to rectify this with more advanced helm templating.

Configuring External Datastores

By default the helm chart stands up statefulsets for the various datastores needed. Each datastore can be configured to come from an external source. Here is an example of configuring an external redis via the values.yaml.

redis:
  external: true
  secret:
    host: "my-redis-6.df59a8.ng.0001.use1.cache.amazonaws.com"

The key external gets set to true and that means a statefulset will not be created. Under the secret section you can configure things like host port username password etc. These values will be propagated into all the different deployments that need to access a redis cluster.

Configuring OpsLevel Runner

After configuring the basics, you'll want to configure the OpsLevel Runner. OpsLevel Runners are used for a variety of asynchronous jobs in OpsLevel including running Repo Grep Checks, Service Detection from Git Integrations, SBOM generation, etc.

To configure OpsLevel Runners with self hosted OpsLevel, first you'll need to get your OpsLevel application running,

In OpsLevel

📘

Ensure you are logged in as an Admin user

This is required to create an API token with Read and Write permissions.

  1. Navigate to Integrations > API Tokens
  2. Click Create API Token
  3. In the dialog, add a Token Description (For example, "OpsLevel Runner Token") and ensure that Read-only is unchecked. Note: if you cannot uncheck the Read-only toggle, you may not be an Admin user.
  4. Copy the Token value and store the value somewhere safe for use later. Note: the token will only be visible at this time. If you lose the value, you will need to create a new API token.

In the Helm values.yaml configuration

runner:
  enabled: true
  secret:
    token: "XXXXXXX"

Then redeploy the helm chart and you should see an additional deployment of opslevel-runner that will connect up to the application and perform the jobs types mentioned above.

Connect to your own MySQL (Production Requirement)

📘

Note on MySQL included in installation

By default, the installation creates a MySQL container alongside the application for getting started quickly.

We do not recommend using the included MySQL container in production deployments of OpsLevel! This section provides you with the detailed configuration needed to allow the application to connect to an “external” MySQL database that has been hardened for production and is monitored by you.

The MySQL database is where all of your application data is stored. If this database is destroyed, all of the data within OpsLevel is also destroyed. To prevent data loss, we recommend you setup regular backups.

There are a handful of MySQL parameters that need to be set that we have enumerated here, but do also note that we have configured preflight checks to ensure the database OpsLevel connects to will work properly.

    [mysqld]
    init_connect='SET collation_connection = utf8mb4_unicode_ci,NAMES utf8mb4'
    character_set_server = 'utf8mb4'
    collation_server = 'utf8mb4_unicode_ci'

    [client]
    default_character_set = 'utf8mb4'

Subject to change: Our application is configured to perform any pending migrations on this database at startup. Please be aware that this is not an “online” process, and as such may lock the database for a period of time while the migrations complete, thereby preventing use of the application.

In the Helm values.yaml configuration

mysql:
  external: true
  secret:
    host: "mysql.ahe5pdz9efix.us-east-1.rds.amazonaws.com"
    username: "my-admin"
    password: "XXXXXXXXXXXXX"

Connect to your own Postgres (Production Requirement)

📘

Note on Postgres included in installation

If historical reporting is enabled, the installation creates a Postgres container alongside the application for getting started quickly.

We do not recommend using the included Postgres container in production deployments of OpsLevel! This section provides you with the detailed configuration needed to allow the application to connect to an “external” Postgres database that has been hardened for production and is monitored by you.

The Postgres database is where all of your historical reporting data is stored. If this database is destroyed, all of the historical reporting data within OpsLevel is also destroyed. To prevent data loss, we recommend you setup regular backups.

Subject to change: as of right now, there are no specific Postgres parameters that need to be set to ensure the database OpsLevel connects to will work properly.

In the Helm values.yaml configuration

postgres:
  external: true
  secret:
    host: "postgres.ahe5pdz9efix.us-east-1.rds.amazonaws.com"
    username: "my-admin"
    password: "XXXXXXXXXXXXX"

Connect to your own Redis (Production Requirement)

📘

Note on Redis included in installation

By default, the installation creates a Redis container alongside the application for the purpose of getting started quickly.

We do not recommend using the included Redis container in production deployments of OpsLevel!

This section provides you with the configuration needed to allow the application to connect to an “external” Redis instance that has been hardened for production and is monitored by you.

The Redis database is used to store pending asynchronous jobs waiting to be processed. If this database is destroyed all that is lost is pending jobs. For the most part many asynchronous tasks will be re-enqueued at a later time, but there are a few cases where losing the job means that specific data that was ingested will never be processed.

In the Helm values.yaml configuration

redis:
  external: true
  secret:
    host: "reds.3d46o8.ng.0001.use1.cache.amazonaws.com"
    port: "6379"
    tls: true

Connect to your own Elasticsearch Cluster (Production Requirement)

📘

Note on Elasticsearch included in installation

By default, the installation creates an Elasticsearch container alongside the application for the purpose of getting started quickly.

We do not recommend using the included Elasticsearch container in production deployments of OpsLevel!

This section provides you with the configuration needed to allow the application to connect to an “external” Elasticsearch instance that has been hardened for production and is monitored by you.

An Elasticsearch cluster helps OpsLevel to power features related to improved Search and Tech Docs.

OpsLevel expects that your Elasticsearch cluster version is 8.4.2 or higher. If you don't have an Elasticsearch cluster available, we recommend using a deployment provided by Elastic.co directly.

In the Helm values.yaml configuration

elasticsearch:
  external: true
  secret:
    host: "my-endpoint.es.us-east-1.aws.elastic-cloud.com"
    port: "443"
    username: "elastic"
    password: "XXXXXXXXXXX"

Advanced Configuration and Integrations

You'll notice that the OpsLevel configuration has many more sections than those that we've walked through above. Most of the additional settings are related to specific 3rd party tool integrations. Many of the integrations can be configured from within the OpsLevel application using authentication tokens, however some integrations require additional configuration of applications within the 3rd Party Service.

Integrations that require additional configure before the Opslevel application starts up include:

Updated about 1 month ago

What’s Next

Learn more about how to connect to your self hosted OpsLevel application through our tooling or integrations