​Database

How to configure Kestra to use a database for its Queue and Repository.

First, you need to configure Kestra to use a database for its Queue and Repository. Here is an example for PostgreSQL:

kestra:
  queue:
    type: postgres
  repository:
    type: postgres

Currently, Kestra supports Postgres, H2, MySQL, and SQL Server (available in a preview in the Enterprise Edition since Kestra 0.18.0).

• H2 can be convenient for local development.
• For production, we recommend PostgreSQL. If PostgreSQL is not an option for you, MySQL and SQL Server can be used as well.

Check the Software Requirements section for the minimum version of each database.

Minimal configuration

In order to connect to a database, you need to configure a datasource. Kestra uses The HikariCP connection pool under the hood, and if needed, you can configure multiple options from the HikariCP documentation directly in your datasource configuration.

PostgreSQL

Here is a minimal configuration for PostgreSQL:

kestra:
  queue:
    type: postgres
  repository:
    type: postgres

datasources:
  postgres:
    url: jdbc:postgresql://localhost:5432/kestra
    driverClassName: org.postgresql.Driver
    username: kestra
    password: k3str4

MySQL

Here is a minimal configuration for MySQL:

kestra:
  queue:
    type: mysql
  repository:
    type: mysql

datasources:
  mysql:
    url: jdbc:mysql://localhost:3306/kestra
    driverClassName: com.mysql.cj.jdbc.Driver
    username: kestra
    password: k3str4
    dialect: MYSQL

SQL Server

Here is a minimal configuration for a SQL Server database, currently available in preview in the Enterprise Edition since Kestra 0.18.0:

kestra:
  repository:
    type: sqlserver
  queue:
    type: sqlserver

datasources:
  sqlserver:
    url: jdbc:sqlserver://localhost:1433
    driverClassName: com.microsoft.sqlserver.jdbc.SQLServerDriver
    username: kestra
    password: k3str4
    dialect: SQL_SERVER
# Uncomment only if your database is not encrypted
#    data-source-properties:
#      encrypt: false

H2

Here is a minimal configuration for H2:

kestra:
  queue:
    type: h2
  repository:
    type: h2

datasources:
  h2:
    url: jdbc:h2:mem:public;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE
    username: sa
    password: ""
    driverClassName: org.h2.Driver

Connection pool size

The total number of connections opened to the database will depend on your chosen architecture. Each Kestra instance will open a pool of up to the maximum-pool-size (10 by default), with a minimum size of the minimum-idle (also set to 10 by default).

• If you deploy Kestra as a standalone server, it will open 10 connections to the database.
• If you deploy each Kestra component separately, it will open 40 connections to the database (10 per component).
• If you deploy each Kestra component separately with 3 replicas, it will open 120 connections to the database.

Usually, the default connection pool sizing is enough, as HikariCP is optimized to use a low number of connections.

Datasource properties

Here are the datasource configuration properties. For more details, check the HikariCP configuration documentation:

Here is the default HikariCP configuration:

transaction-isolation: default # driver default
pool-name: HikariPool-<Generated>
connection-init-sql: null
connection-test-query: null
connection-timeout: 30000 # 30 seconds
idle-timeout: 600000 # 10 minutes
minimum-idle: 10 # same as maximum-pool-size
initialization-fail-timeout: 1
leak-detection-threshold: 0
maximum-pool-size: 10
max-lifetime: 1800000 # 30 minutes
validation-timeout: 5000

Queues configuration

kestra.jdbc.queues

Kestra database queues simulate queuing doing long polling. They query a queues table to detect new messages.

You can change these parameters to reduce the polling latency, but be aware it will increase the load on the database:

• kestra.jdbc.queues.poll-size: the maximum number of queues items fetched by each poll.
• kestra.jdbc.queues.min-poll-interval: the minimum duration between 2 polls.
• kestra.jdbc.queues.max-poll-interval: the maximum duration between 2 polls.
• kestra.jdbc.queues.poll-switch-interval: the delay for switching from min-poll-interval to max-poll-interval when no message is received. (ex: when one message is received, the min-poll-interval is used, if no new message arrived within poll-switch-interval, we switch to max-poll-interval).

Here is the default configuration:

kestra:
  jdbc:
    queues:
      poll-size: 100
      min-poll-interval: 25ms
      max-poll-interval: 1000ms
      poll-switch-interval: 5s

kestra.jdbc.cleaner

Kestra cleans the queues table periodically to optimize storage and performance. You can control how often you want this cleaning to happen, and how long messages should be kept in the queues table .

Here is the default configuration:

kestra:
  jdbc:
    cleaner:
      initial-delay: 1h
      fixed-delay: 1h
      retention: 7d