Connect a Warehouse

Beta

Experimentation is in beta.

Experiment data (exposures and conversion events) is stored and queried in a data warehouse. Each environment has one warehouse connection, and it must be in place before you can create experiments.

Self-hosted

The managed Flagsmith warehouse is only available on Flagsmith Cloud. When self-hosting on PostgreSQL, you can use the Flagsmith database instead. See Self-Hosting and Open Source for what each option needs deployed.

Plans

On Flagsmith Cloud, connecting your own ClickHouse, Databricks or PostgreSQL requires a Scale-Up or Enterprise subscription. Self-hosted Flagsmith has no plan restriction.

Connect the Flagsmith warehouse

Flagsmith can store experiment events for you, with nothing to provision. On Flagsmith Cloud, this is the managed Flagsmith warehouse. When self-hosting on PostgreSQL, it is the Flagsmith database, the database your installation already runs on.

Flagsmith Cloud

1. Go to Environment Settings › Warehouse.
2. Select Flagsmith and click Enable, then confirm.

The connection is created immediately. To verify that events can flow, click Send your first event and Flagsmith sends a test event on your behalf. The connection shows Pending Connection while it waits for the first event, and switches to Connected as soon as the environment's first event arrives; an event sent from your own application counts too, not just the test event.

note

Processing the first event can take up to a few hours. The status refreshes about once a minute while you keep the Warehouse tab open.

Once connected, the warehouse card shows the total number of events received and the number of unique event names, which is useful to verify that instrumentation data is arriving.

Self-hosted: use the Flagsmith database

This needs EXPERIMENTATION_ENABLED=true on the API and a PostgreSQL database; it fails on MySQL or Oracle. See Self-Hosting and Open Source.

1. Go to Environment Settings › Warehouse.
2. Select Flagsmith database and click Enable, then confirm.

The connection shows Connected straight away. If Flagsmith can't read the events table, it shows Errored with the reason; fix it and click Test connection on the card.

Then set each SDK's events URL to your Flagsmith API URL ending in /api/, for example https://flagsmith.example.com/api/. Don't add v1/events; the SDKs append it. The card shows this URL with a copy button. For each SDK, see Initialise. To verify events arrive, click Send your first event on the card.

The connection has no settings to edit. To stop storing events, delete the connection; events already stored stay in the table.

Connection statuses

The current status is shown on the warehouse card in Environment Settings › Warehouse.

• Created: the connection exists but no events have been received yet.
• Pending Connection: waiting for the first event to arrive.
• Connected: events have been received; you are ready to run experiments.
• Errored: the card shows the reason. See Troubleshooting for Databricks or PostgreSQL; otherwise contact support.

Bring your own ClickHouse

Instead of the managed Flagsmith warehouse, you can store experiment events in your own ClickHouse instance. It must be reachable from Flagsmith: on Flagsmith Cloud, over the internet; when self-hosting, from your Flagsmith API and warehouse delivery.

1. Configure your instance by running the following against it, using an administrative user (the statements require CREATE USER, GRANT, CREATE DATABASE and CREATE TABLE privileges). Replace <USER> and <YOUR_PASSWORD> with your own values. This is a one-off setup step; the same SQL is available to copy from the Flagsmith dashboard.

   -- ClickHouse schema for experimentation events.
   CREATE DATABASE IF NOT EXISTS flagsmith_exp;
   
   CREATE TABLE IF NOT EXISTS flagsmith_exp.events
   (
       environment_key      LowCardinality(String),
       event                LowCardinality(String),
       feature_name         LowCardinality(String),
       timestamp            DateTime64(3),
       collected_at         DateTime64(3),
       identifier           String,
       value                String                          CODEC(ZSTD(3)),
       traits               String                          CODEC(ZSTD(3)),
       metadata             String                          CODEC(ZSTD(3)),
       sdk_language         LowCardinality(String),
       sdk_version          LowCardinality(String),
   
       INDEX idx_identity identifier TYPE bloom_filter GRANULARITY 4,
   
       CONSTRAINT environment_key_not_empty CHECK environment_key != '',
       CONSTRAINT event_not_empty           CHECK event != '',
       CONSTRAINT timestamp_sane            CHECK timestamp > toDateTime64('2020-01-01 00:00:00', 3)
   )
   ENGINE = MergeTree
   PARTITION BY toYYYYMMDD(timestamp)
   ORDER BY (environment_key, event, feature_name, timestamp, identifier);

   -- Create a dedicated user for Flagsmith
   CREATE USER IF NOT EXISTS <USER>
       IDENTIFIED WITH sha256_password BY '<YOUR_PASSWORD>';
   
   -- Allow it to write and read experiment events
   GRANT SELECT, INSERT ON flagsmith_exp.events TO <USER>;
   
   -- Allow it to check the events table exists
   GRANT SHOW TABLES, SHOW COLUMNS ON flagsmith_exp.* TO <USER>;

2. Go to Environment Settings › Warehouse, select ClickHouse and enter your connection details: host, HTTPS port (8443 by default), database (flagsmith_exp by default), and the username and password of the user you created in step 1.

3. Click Test connection. Once the test succeeds, save the connection.

4. On Flagsmith Cloud, click Send your first event on the connection card to verify events are arriving in your warehouse; the button disappears once the first event is received.

Connections use the ClickHouse HTTP(S) interface, with TLS enabled by default.

Experiment results and exposures are computed from your ClickHouse instance.

Bring your own Databricks

You can also store experiment events in your own Databricks workspace, and Flagsmith computes results there. See Connect Databricks for setup and troubleshooting.

Bring your own PostgreSQL

You can store experiment events in your own PostgreSQL database, or a PostgreSQL-compatible service, and Flagsmith computes results there. It suits low to moderate event volumes. See Connect PostgreSQL for setup and troubleshooting.

Coming soon

Bring your own warehouse: connections for Snowflake and BigQuery.