​Key ​Value (​K​V) ​Store

Key Value (KV) Store allows you to have key-value pairs at the namespace level.

The Key Value Store, generally called as KV Store, is an implementation on top of Kestra's internal storage. As the name suggests, it is a store for key value pairs.

The motivation behind introducing KV store is:

Privacy: We never want Kestra to store user private data. This means that all values will be stored in the user’s private cloud storage bucket, and the Kestra's database only contains metadata about it, such as the key, file URI, any attached metadata about the object like TTL, creation date, last updated timestamp, etc.

Ease of implementation/migration: Users can easily switch from open-source to cloud/EE because the implementation and data storage will be the same regardless of whether Kestra runs on top of Kafka or JDBC.

Keys and Values

Keys are arbitrary strings. Keys can contain:

• characters in uppercase and or lowercase
• standard ASCII characters

Values are stored as ION files in Kestra's internal storage. Values are strongly typed, and can be of one of the following types:

• string
• number
• boolean
• datetime
• date
• duration
• JSON

You can associate TTL with the KV pair. The KV pair can only be used until it is active as per the TTL.

Namespace binding

Key value pairs are tied to a namespace.

Users should be able to create and read KV pairs across namespaces as long as those namespaces are allowed.

Namespace Management in OSS

Starting in 0.18.0, Kestra has opened up a lightweight version of Namespace Management in OSS that will help you manage the KV store. You will now have access to the Namespaces tab from the left navigation menu on the Kestra's UI. You will be able to see all the available namespaces on this tab, as well as create one.

For any namespace, you will be able to see the namespace's overview, its corresponding editor, flows, dependencies and the KV store.

Managing KV Store

You can manage the KV store via:

• the UI
• the tasks
• the API
• the terraform resource

Managing KV Store via UI

In the Kestra UI,

1. Navigate to the Namespaces tab from the left navigation menu.
2. Select the namespace where you want to create the KV pair.
3. Navigate to the KV Store tab. This is where you can see all the KV pairs associated with this namespace.
4. Click on New Key-Value button on the top right to create a new KV pair.
5. Put an appropriate name for the Key.
6. Choose an appropriate Type for the value.
7. Put an appropriate value in the Value textbox.
8. You can choose an expiration time (TTL) for the KV pair from the dropdown. Note that the dropdown contains some standard durations. In case you want some custom duration, you can enter it in the Custom duration textbox.
9. Click on Save button at the bottom to save this KV pair.

With this, a new KV pair is created.

You can edit / delete the KV pair by clicking on the appropriate button towards the right of the corresponding KV pair.

Managing KV Store via tasks

You can manage the KV store via tasks in the following fashion:

Set the KV pair

tasks:
  - id: query
    type: io.kestra.plugin.core.http.Download
    uri: https://huggingface.co/datasets/kestra/datasets/raw/main/csv/orders.csv

  - id: set_kv
    type: io.kestra.plugin.core.kv.Set
    key: my_key
    value: "{{ outputs.query.uri }}"
    namespace: company.team # the current namespace of the flow can be used by default
    overwrite: true # whether to overwrite or fail if a value for that key already exists; default true
    ttl: P30D # optional TTL

You can use set to create or edit the KV pair. Ensure overwrite is set to true while editing the KV pair.

Get the KV pair

The easiest way to retrieve a value by key is to use the Pebble function following this syntax:

{{ kv('KEY_NAME', namespace_name, errorOnMissing_boolean) }}

For example,

tasks:
  - id: log_key
    type: io.kestra.plugin.core.log.Log
    message: "{{ kv('my_key') }}" # assuming you retrieve it in a flow in the same namespace as the one for which key was created

If you prefer, you can also retrieve the value using a task:

tasks:
  - id: get_value_by_key
    type: io.kestra.plugin.core.kv.Get
    key: my_key
    namespace: company.team # the current namespace of the flow can be used by default
    errorOnMissing: false # bool

  - id: log_key_get
    type: io.kestra.plugin.core.log.Log
    message: "{{ outputs.get_value_by_key.value }}"

And if you want to check if some values already exist for a given key, you can search keys by prefix:

tasks:
  - id: get_keys_by_prefix
    type: io.kestra.plugin.core.kv.GetKeys
    prefix: "test_"
    namespace: company.team # the current namespace of the flow can be used by default

  - id: log_key_prefix
    type: io.kestra.plugin.core.log.Log
    message: "{{ outputs.get_keys_by_prefix.keys }}"

The output will be a list of keys - if no keys were found, an empty list will be returned.

Delete a KV pair

The io.kestra.plugin.core.kv.Delete task will produce the boolean output deleted to confirm it was deleted or not.

tasks:
  - id: delete_kv_pair
    type: io.kestra.plugin.core.kv.Delete
    key: my_key
    namespace: company.team # the current namespace of the flow can be used by default
    errorOnMissing: false

  - id: log_kv_deleted
    type: io.kestra.plugin.core.log.Log
    message: "{{ outputs.delete_kv_pair.deleted }}"

Managing KV Store via API

You can manage the KV store via APIs with the following URLs:

Set the KV pair

The API call to set the KV pair follows the structure:

curl -X PUT -H "Content-Type: application/json" http://localhost:8080/api/v1/namespaces/{namespace}/kv/{key} -d '<value>'

For example:

curl -X PUT -H "Content-Type: application/json" http://localhost:8080/api/v1/namespaces/company.team/kv/my_key -d '"Hello World"'

The above curl call will create the KV pair with key my_key with value Hello World string in the company.team namespace. The API does not return any response as such.

Get the KV pair

You can get any particular KV pair using:

curl -X GET -H "Content-Type: application/json" http://localhost:8080/api/v1/namespaces/{namespace}/kv/{key}

For example:

curl -X GET -H "Content-Type: application/json" http://localhost:8080/api/v1/namespaces/company.team/kv/my_key

This would retrieve a KV pair with the key my_key in the company.team namespace. The output of the API will contain the data type of the value and the retrived value of the KV pair. The output will look like:

{"type": "STRING", "value": "Hello World"}

You can also get all the keys in the namespace using the curl call:

curl -X GET -H "Content-Type: application/json" http://localhost:8080/api/v1/namespaces/{namespace}/kv

For example, the following curl call will return all the keys in the company.team namespace:

curl -X GET -H "Content-Type: application/json" http://localhost:8080/api/v1/namespaces/company.team/kv

The output will be as follows:

{"key":"my_key","creationDate":"2024-07-15T06:10:33.422Z","updateDate":"2024-07-15T06:11:08.911Z"},{"key":"test_key","creationDate":"2024-07-15T04:37:18.196Z","updateDate":"2024-07-15T04:37:18.196Z"}]

Delete the KV pair

You can delete the KV paior using the following API:

curl -X DELETE -H "Content-Type: application/json" http://localhost:8080/api/v1/namespaces/{namespace}/kv/{key}

This call returns the boolean indicating whether the key existed for deletion or not.

For example, the following curl call will delete the key my_key in the company.team namespace:

curl -X DELETE -H "Content-Type: application/json" http://localhost:8080/api/v1/namespaces/company.team/kv/my_key

Managing KV Store via Terraform

You can manage the KV store via Terraform in the following fashion:

Create the KV pair

resource "kestra_kv" "new" {
  namespace = "company.team"
  key       = "my_key"
  value     = "Hello Woprld"
  type      = "STRING"
}

Get the KV pair

data "kestra_kv" "new" {
  namespace = "company.team"
  key       = "my_key"
}

Accessing JSON value

For accessing JSON value, you need to use the json() function on the retrieved value. This is demonstrated in the following flow:

id: kv_json_flow
namespace: company.team

tasks:
  - id: set_json_kv
    type: io.kestra.plugin.core.kv.Set
    key: "my_json_key"
    value: | 
      {
        "name": "John Doe",
        "age": 32,
        "address": {
          "city": "Paris",
          "country": "France"
        }
      }
    overwrite: true

  - id: get_json_kv_pebble
    type: io.kestra.plugin.core.log.Log
    message:
      - "Name: {{ json(kv('my_json_key')).name }}"
      - "Age: {{ json(kv('my_json_key')).age }}"
      - "City: {{ json(kv('my_json_key')).address.city }}"

  - id: get_json_kv_task
    type: io.kestra.plugin.core.kv.Get
    key: my_json_key

  - id: get_json_kv_nested_value
    type: io.kestra.plugin.core.log.Log
    message: "Country: {{ json(outputs.get_json_kv_task.value).address.country }}"