Destinations: export synced records to your lake or warehouse
A destination copies the records OpenSteward has synced into its data store to storage you own, on a schedule:
- Parquet files in an object store: Amazon S3 and S3-compatible stores (MinIO, Cloudflare R2, Backblaze B2, …), Google Cloud Storage, or Azure Blob Storage. Point DuckDB, Spark, Athena, BigQuery external tables, Snowflake stages or Databricks at the files.
- Rows in Postgres: one table per resource type, kept up to date with upserts and deletes.
- Tables in a warehouse: Snowflake, BigQuery or Databricks (Delta). The
loader stages Parquet files in an object store the warehouse can read, loads
them, and
MERGEs them into one current-state table per resource type, applying deletes too. See warehouse loaders. - Apache Iceberg (experimental): tables managed in an Iceberg REST catalog, with Iceberg-ready Parquet files you register with your engine. See Iceberg.
Destinations work on SQLite and Postgres deployments alike. They need the data
store, so they are unavailable with DATA_STORE_MODE=passthrough (the API
answers 409 DATA_STORE_DISABLED).
How an export runs
Section titled “How an export runs”The export worker runs inside integration-server. Every 30 seconds it looks for
destinations that are due: enabled, with their interval (intervalSec,
default 3600, minimum 60) elapsed since the last run. For each one it:
- Takes the destination’s lease. A conditional
UPDATEmeans exactly one replica runs a given destination at a time. The lease lasts 10 minutes and is renewed at every file written. - Exports changed rows. For each (vertical, resource type) in the destination’s scope, it reads rows whose data changed since that pair’s cursor, masks them, and writes them in files of at most 50,000 rows or about 128 MB. The cursor moves only after the file (or Postgres or warehouse batch) has been written.
- Exports deletions, from the tombstone feed, after the rows.
- Writes a manifest listing every file the run wrote (object stores only).
- Records the run in the destination’s run history.
A row only counts as changed when its data changes. A full re-sync that re-writes identical data doesn’t re-export it. Rows and deletions from the last 120 seconds are held back until the next run, because a write stamped at time T can commit after a reader has already moved past T.
Delivery is at least once
Section titled “Delivery is at least once”If a run fails after writing a file but before saving its cursor, the next run
writes those rows again, in a new file. Consumers should deduplicate on
_pk, keeping the row with the greatest changed_at. The Postgres sink and
the warehouse loaders do exactly that.
A failed run leaves every cursor at the last successful write. The next run (at the next interval, or when you trigger one) continues from there.
File layout (object stores)
Section titled “File layout (object stores)”All paths are under the destination’s prefix:
<prefix>vertical=<vertical>/resource_type=<type>/dt=<YYYY-MM-DD>/<runId>-<part>.parquet<prefix>_deletes/dt=<YYYY-MM-DD>/<runId>-<part>.parquet<prefix>_manifests/<runId>.json<prefix>_opensteward/connection-test.json (written by "test connection")- The paths are Hive-style partitions, so most engines pick up
vertical,resource_typeanddtas columns. dtis the UTC date the run started, not the date the record changed.<runId>is the run’s id (a UUID, also in the run history).<part>is a zero-padded counter that is unique within the run.- Files are never overwritten or deleted. A run’s files are complete once its manifest exists, so readers that need consistency should list manifests rather than data files.
The manifest looks like this:
{ "version": 1, "format": "parquet", "runId": "5b1c…", "destinationId": "9f0e…", "trigger": "schedule", "startedAt": "2026-09-29T12:00:00.000Z", "finishedAt": "2026-09-29T12:00:04.210Z", "includePii": false, "rowsWritten": 1250, "deletesWritten": 3, "files": [ { "path": "vertical=crm/resource_type=contacts/dt=2026-09-29/5b1c…-00000.parquet", "kind": "rows", "vertical": "crm", "resourceType": "contacts", "rows": 1250 }, { "path": "_deletes/dt=2026-09-29/5b1c…-00001.parquet", "kind": "deletes", "rows": 3 } ]}Parquet files use ZSTD compression by default. You can set compression to
snappy or none instead.
Columns
Section titled “Columns”Every row file has these fixed columns, in this order:
| Column | Type | Meaning |
|---|---|---|
_pk |
string | Stable row key: vertical:platform:connection_id:resource_type:resource_id. Deduplicate on this. |
connection_id |
string | The OpenSteward connection the record came from. |
owner_tenant_id |
string, nullable | The deployment tenant that owns the connection. |
platform |
string | Upstream platform slug (hubspot, shopify, …). |
vertical |
string | accounting, crm, ecommerce, … |
resource_type |
string | contacts, invoices, orders, … |
resource_id |
string | The platform’s id for the record. |
changed_at |
timestamp (ms, UTC) | When the record’s data last changed in the store. Use it to order versions. |
norm_version |
string, nullable | Which normalizer produced the record. |
_deleted |
boolean | Always false in row files. |
data |
string (JSON) | The whole record, after masking, in OpenSteward’s canonical schema. |
After the fixed columns come the record’s top-level scalar fields (strings, numbers and booleans), flattened for convenience:
- Names are converted to
[a-z0-9_], with camelCase split into snake_case (firstNamebecomesfirst_name). A name that clashes with a fixed column gets afield_prefix (platformbecomesfield_platform). - Types are inferred per file. A column is
booleanif every value is a boolean,int64if every value is an integer,doubleif every value is a number, andstringotherwise. A mix of types becomes a string. - Nested objects and arrays are only in
data.datais authoritative. The flattened columns are there so you can filter and join without parsing JSON.
Delete files (_deletes/) have these columns: tombstone_id, scope, _pk,
vertical, platform, connection_id, owner_tenant_id, resource_type,
resource_id, deleted_at, and _deleted (always true).
Postgres tables
Section titled “Postgres tables”Each (vertical, resource type) gets its own table,
<schema>.os_<vertical>_<resource_type>, with names sanitised to
[a-z0-9_]. For example, opensteward.os_crm_contacts or
opensteward.os_customer_support_tickets. Each table has:
- the fixed columns above, with
changed_atastimestamptzanddataasjsonb; - an
_exported_atcolumn; - one column per flattened field, added as new fields appear (up to 400 per
table). A value that doesn’t fit its column’s type is written as
NULLthere, but is still indata.
The worker creates the schema (default opensteward) and the tables if they
don’t exist. The Postgres user needs CONNECT on the database and CREATE on
the schema, or on the database if the schema doesn’t exist yet.
Rows are upserted on _pk and only move forward: an older version replayed
by an at-least-once retry never overwrites a newer one.
Warehouse loaders
Section titled “Warehouse loaders”The snowflake, bigquery and databricks types keep one current-state
table per (vertical, resource type) in your warehouse, like the Postgres
sink, but load it the way warehouses prefer: in bulk, from staged files.
How a batch is loaded
Section titled “How a batch is loaded”For every batch the export worker hands it (at most 50,000 rows), a loader:
- Dedupes the rows on
_pk, keeping the greatestchanged_at. AMERGErefuses two source rows for one target row. - Creates the table if it doesn’t exist, and adds a column for every new flattened field (up to 400 per table). A column’s type is fixed by the first batch that sees it.
- Writes one Parquet file whose columns are exactly the table’s: every
flattened column is present, and a value that doesn’t fit its column’s
type is written as
NULL(it is still indata). - Uploads it to the stage:
<stage>/_opensteward_stage/os_<vertical>_<type>/<id>-rows.parquet. - Loads it into a staging table (Snowflake
COPY INTOa temporary table, a BigQuery load job, DatabricksCOPY INTOa Delta table) andMERGEs the staging table into the target on_pk. - Drops the staging table and deletes the staged file.
The tables are named like the Postgres ones: os_crm_contacts,
os_customer_support_tickets. Snowflake folds them to upper case
(OS_CRM_CONTACTS, column FIRST_NAME), so they resolve the same way as
unquoted names and you can query them without quotes.
MERGE semantics
Section titled “MERGE semantics”MERGE INTO target t USING staging s ON t._pk = s._pkWHEN MATCHED AND t.changed_at <= s.changed_at THEN UPDATE SET …, _exported_at = nowWHEN NOT MATCHED THEN INSERT (…)- Forward only. A row only changes when the incoming version is at least as new. An older version replayed by an at-least-once retry is ignored, so a retried batch converges on the same table.
- Every column is written. A flattened field that disappeared from a record
becomes
NULLin its column. - Types. The fixed columns match the Columns table.
dataisVARIANTin Snowflake,JSONin BigQuery andSTRINGin Databricks (usedata:fieldorfrom_json). Flattened columns areVARCHAR/STRING,NUMBER(38,0)/INT64/BIGINT,FLOAT/FLOAT64/DOUBLEorBOOLEAN._exported_atis when the row was last written. - No multi-statement transaction. Each statement is atomic on its own, and
each is idempotent (the
MERGEonly moves rows forward and a delete only removes rows no newer than its tombstone), so a failure part-way through a batch is repaired by the retry.
Deletes
Section titled “Deletes”Loaders apply tombstones with the same rule as the
Postgres sink: delete the rows they cover whose changed_at is not after
the tombstone’s deleted_at.
| Scope | Snowflake | BigQuery | Databricks |
|---|---|---|---|
record |
Staged Parquet file, COPY INTO a temporary table, DELETE … USING |
DELETE … WHERE EXISTS (SELECT … FROM UNNEST(@deletes)), 10,000 per statement |
Staged file, COPY INTO a staging table, MERGE … WHEN MATCHED THEN DELETE |
connection |
DELETE … WHERE platform = ? AND EQUAL_NULL(connection_id, ?) |
DELETE … WHERE platform = @platform AND connection_id = @connection_id |
DELETE … WHERE platform = ? AND connection_id <=> ? |
vertical, all |
DELETE … WHERE changed_at <= ? |
same | same |
Wide scopes (connection, vertical, all) apply to every table the
destination has written in that vertical. Every value, tombstone or row, is a
bound parameter or travels in a staged file. None is ever spliced into SQL.
The stage
Section titled “The stage”Loaders stage files in an object store: S3, GCS or Azure Blob, whichever your
warehouse can read. The loader writes with the stage* credentials. The
warehouse reads with its own access: a storage integration, a Unity Catalog
external location, or the service account.
- Staged files live under
_opensteward_stage/for a few seconds each. A failed batch can leave one behind, so add a lifecycle rule that expires_opensteward_stage/after a day or two. They hold export rows, masked like every other export. - The upload credentials need write and delete on the stage prefix. Without delete, loads still work, but files accumulate until the lifecycle rule removes them.
- “Test connection” writes
_opensteward_stage/_connection_test/probe.parquet, has the warehouse load it into a throwaway table, then removes both. That proves the upload credentials, the warehouse’s read access to the stage andCREATE TABLEin the target schema in one go.
Stage URLs may be spelled s3://bucket/prefix/, gcs:// or gs://,
azure://<account>.blob.core.windows.net/<container>/prefix/, or
abfss://<container>@<account>.dfs.core.windows.net/prefix/. Each loader
renders them the way its warehouse expects. Path segments may contain only
letters, digits, ., _, - and =.
Snowflake
Section titled “Snowflake”| Config | |
|---|---|
account |
Account identifier: myorg-myaccount, or a locator such as xy12345.eu-west-1. |
host |
Optional. Overrides <account>.snowflakecomputing.com, for PrivateLink or .snowflakecomputing.cn. |
warehouse, database, schema, role |
Where to load. The schema must exist. role defaults to the user’s default role. |
stage |
{ type: "s3" | "gcs" | "azure", url, region?, storageIntegration? } |
Secrets: username, plus either password or privateKey (PEM, with
privateKeyPassphrase if it’s encrypted). Key-pair authentication is
recommended. Also the stage credentials: stageAccessKeyId +
stageSecretAccessKey (+ stageSessionToken) for S3, stageCredentialsJson
for GCS, or stageAccountKey/stageSasToken for Azure.
Setup. Create a storage integration for the stage location, and a least-privilege role for the loader:
-- As ACCOUNTADMIN (or a role with CREATE INTEGRATION).CREATE STORAGE INTEGRATION opensteward_stage TYPE = EXTERNAL_STAGE STORAGE_PROVIDER = 'S3' ENABLED = TRUE STORAGE_AWS_ROLE_ARN = 'arn:aws:iam::123456789012:role/snowflake-opensteward' STORAGE_ALLOWED_LOCATIONS = ('s3://lake/stage/');DESC INTEGRATION opensteward_stage; -- trust-policy values for the IAM role
CREATE ROLE opensteward_loader;GRANT USAGE ON INTEGRATION opensteward_stage TO ROLE opensteward_loader;GRANT USAGE ON WAREHOUSE load_wh TO ROLE opensteward_loader;GRANT USAGE ON DATABASE analytics TO ROLE opensteward_loader;GRANT USAGE, CREATE TABLE ON SCHEMA analytics.opensteward TO ROLE opensteward_loader;
CREATE USER opensteward_loader DEFAULT_ROLE = opensteward_loader DEFAULT_WAREHOUSE = load_wh RSA_PUBLIC_KEY = 'MIIBIjANBgkqh…';GRANT ROLE opensteward_loader TO USER opensteward_loader;- The IAM role behind the integration needs
s3:GetObject,s3:GetObjectVersionands3:ListBucketon the stage prefix. For GCS or Azure integrations, follow Snowflake’s instructions for granting the integration’s service principal read access. - The loader owns the tables it creates. If you create them yourself, grant the
role
SELECT, INSERT, UPDATE, DELETEon them. CREATE TABLEon the schema also covers the temporary staging tables. They are session-scoped, so an interrupted load never leaves one behind.- Without a storage integration (S3 and Azure only), the loader passes the
stage credentials in the
COPYstatement’sCREDENTIALSclause, so they are part of statement text Snowflake receives. Prefer an integration. GCS always needs one.
Costs. Warehouse time for COPY and MERGE on every batch. Size the
warehouse small (X-Small handles tens of thousands of rows per batch) and let it
auto-suspend. A destination’s intervalSec controls how often it wakes.
BigQuery
Section titled “BigQuery”| Config | |
|---|---|
projectId |
Project that owns the dataset and runs the jobs. |
dataset |
Dataset for the tables. It must exist. |
location |
Dataset location. Required unless it’s US or EU. |
gcsBucket, gcsPrefix |
Where files are staged. The bucket must be in a location the dataset can load from. |
Secret: serviceAccountJson, used for BigQuery and the bucket. Leave it out to
use Application Default Credentials (workload identity, for example).
Setup. Grant the service account:
roles/bigquery.jobUseron the project, to run load and query jobs;roles/bigquery.dataEditoron the dataset, to create tables, run DML and readINFORMATION_SCHEMA.COLUMNS;roles/storage.objectUseron the staging bucket (or a prefix-scoped IAM condition), to write, read and delete staged files.
Each batch loads into its own staging table, os_stage_<table>_<id> (Parquet,
WRITE_TRUNCATE), in the same dataset, then merges it and drops it. Target
tables are CLUSTER BY _pk.
Costs. Load jobs are free. The MERGE and DELETE statements are billed
as queries: on-demand pricing bills the bytes they read from the target and
staging tables, and clustering on _pk keeps that down. Mutating DML on one
table is serialized by BigQuery, which is fine at the export’s pace. A longer
intervalSec means fewer, larger merges.
Databricks
Section titled “Databricks”| Config | |
|---|---|
host |
Workspace host name without https://, e.g. dbc-1234.cloud.databricks.com. |
httpPath |
The SQL warehouse’s HTTP path, e.g. /sql/1.0/warehouses/abc123. |
catalog, schema |
Where the Delta tables go. The schema must exist. Omit catalog for the workspace default. |
stage |
{ type, url, region?, credentialName? }. credentialName is a Unity Catalog storage credential used as COPY INTO … WITH (CREDENTIAL …). |
Secrets: token (a personal access token or a service principal’s token), plus
the stage* upload credentials.
Setup (Unity Catalog, as a metastore or catalog admin):
GRANT USE CATALOG ON CATALOG main TO `opensteward-loader`;GRANT USE SCHEMA, CREATE TABLE ON SCHEMA main.opensteward TO `opensteward-loader`;-- Let the warehouse read the stage:GRANT READ FILES ON EXTERNAL LOCATION opensteward_stage TO `opensteward-loader`;- Grant
CAN USEon the SQL warehouse, too. - Either an external location covers the stage URL (above), or set
stage.credentialNameto a storage credential the principal may use. - The loader owns the tables it creates. If you create them yourself, grant
SELECTandMODIFYon them. - Each batch creates a schemaless staging table,
os_stage_<table>_<id>, fills it withCOPY INTO … FILEFORMAT = PARQUET(mergeSchema), merges it and drops it. An interrupted batch can leave one behind, and it’s safe to drop.
Costs. SQL warehouse time while batches run. A serverless warehouse with a short auto-stop suits the export’s bursty pattern.
Iceberg (experimental)
Section titled “Iceberg (experimental)”The iceberg type is experimental. The catalog lists it with
experimental: true, and its layout may still change.
What it does, using only the JSON parts of the Iceberg REST catalog spec:
- reads the catalog’s path prefix from
GET /v1/config?warehouse=…; - creates the namespace if it’s missing;
- creates one table per (vertical, resource type),
os_<vertical>_<type>, with format version 2, a schema derived from the export columns, and aschema.name-mapping.defaultproperty so Parquet files without Iceberg field ids map onto the schema by column name; - evolves that schema as new flattened fields appear (
add-schema+set-current-schema, guarded byassert-current-schema-id; it retries on a concurrent change); - writes each batch’s rows as an Iceberg-ready Parquet file, typed to the
table’s schema, with
changed_atas a microsecondtimestamptz.
What it does not do: commit data snapshots. An append commit needs Avro manifest and manifest-list files, and a partial implementation could corrupt your table. Rather than risk that, OpenSteward leaves the commit to your engine, and every batch is recorded as pending:
<storage>/<namespace…>/<table>/data/batch_<id>/00000.parquet rows<storage>/<namespace…>/<table>/_deletes/batch_<id>.parquet tombstones (same columns as _deletes/ above)<storage>/<namespace…>/<table>/_opensteward/pending/<id>.json what to registerA pending manifest looks like this:
{ "version": 1, "format": "iceberg-ready", "experimental": true, "kind": "rows", "table": { "namespace": ["lake", "opensteward"], "name": "os_crm_contacts", "location": "s3://lake/iceberg/lake/opensteward/os_crm_contacts" }, "schemaId": 2, "directory": "s3://lake/iceberg/lake/opensteward/os_crm_contacts/data/batch_5b1c…/", "files": [{ "path": "…/00000.parquet", "format": "PARQUET", "recordCount": 1250, "fileSizeInBytes": 88213 }], "columns": [{ "name": "email", "type": "string" }], "createdAt": "2026-09-29T12:00:04.210Z"}Registering the files. Each batch has its own directory, so it can be added exactly once. With Spark:
CALL lake.system.add_files( table => 'lake.opensteward.os_crm_contacts', source_table => '`parquet`.`s3://lake/iceberg/lake/opensteward/os_crm_contacts/data/batch_5b1c…/`');Then delete, or move, the pending manifest. Once the files are committed,
anything that reads the catalog can query the table: Spark, Trino, Flink,
Snowflake through a catalog integration, and so on. Before that, you can read
the files directly, for example with DuckDB’s
read_parquet('s3://…/data/*/*.parquet').
The table is an append log, not current state:
- Deduplicate on
_pk, keeping the greatestchanged_at. - Apply
_deletes/with the usual rule (delete rows whosechanged_atis not afterdeleted_at). - A periodic
MERGEinto a compacted table works well for this.
Erasure in Iceberg tables is your responsibility, as with Parquet files.
| Config | |
|---|---|
catalogUri |
REST catalog base URL: the part before /v1/…, such as https://polaris.example.com/api/catalog. |
warehouse |
Passed to GET /v1/config?warehouse=. The catalog decides what it means (a name or a location). |
namespace |
Dot-separated, default opensteward. Created if missing. |
storage |
{ type, url, region?, endpoint?, forcePathStyle? }, where the files and the table locations go. The catalog must allow table locations under it. |
Secrets: token, a bearer token for the catalog, plus the stage* upload
credentials for storage.
Limits.
- Authentication is a static bearer token. OAuth2 client credentials, SigV4 (the AWS Glue REST endpoint) and credential vending aren’t supported.
- The table location is set to
<storage url>/<namespace path>/<table>. Catalogs that assign locations themselves may refuse it. - Tested against the REST spec’s request and response shapes, not against every catalog implementation. Treat it as a preview.
Deletions (tombstones)
Section titled “Deletions (tombstones)”When a record leaves the data store, OpenSteward records a tombstone, and the next export forwards it. A record leaves the store when it is deleted through the API, when a full sync finds it gone upstream, when its connection is deleted, or when an operator clears the cache. A tombstone has one of four scopes:
scope |
Set columns | Means |
|---|---|---|
record |
all, including _pk |
One record was deleted. |
connection |
vertical, platform, connection_id |
Every record of that connection on that platform. |
vertical |
vertical |
Every record in the vertical (an operator cleared it). |
all |
none | Every record. |
To apply a tombstone, delete the rows it covers whose changed_at is not
after its deleted_at. A record that was deleted and later re-created keeps
its newer version. The Postgres sink and the warehouse loaders apply deletes this
way, and apply wider scopes to every table the destination has written (see
warehouse deletes). Iceberg logs them to _deletes/.
A new destination starts reading the delete feed from the moment it is created. Older deletions removed records it could never have exported.
Scope and tenants
Section titled “Scope and tenants”A destination’s scope decides what it exports:
{ "tenants": ["tenant-a"], "verticals": { "crm": "*", "ecommerce": ["orders"] }, "maskFinancial": false}tenants: the deployment tenants whose connections are exported (matched on the connection’s owner tenant), or"*"for every tenant. Only a"*"destination exports connections that have no owner tenant.verticals: which verticals, and within each which resource types ("*"means all of them). Leave it out to export everything.maskFinancial: also mask financial fields (amounts, balances).
Through admin-api, a user can only create or edit a destination whose scope fits inside their own access. The same applies to their tenants, verticals and financial masking. A destination outside a user’s access is invisible to them.
Widening a destination’s scope doesn’t re-export rows that were skipped before. Run a backfill after widening it.
Records are masked by default, using the same field classification the admin data viewer uses (see data classification):
- PII fields (names, emails, phone numbers, addresses, the raw upstream
metadata) are masked. - Secrets are always removed.
- Financial fields are masked only with
maskFinancial.
includePii: true exports PII unmasked. Only a user holding
data:read_pii can turn it on, and they are recorded as its approver
(piiApprovedBy). Changes that could redirect an approved PII export also need
that permission: its config, scope or credentials. Turning PII or the
destination off doesn’t.
Erasure
Section titled “Erasure”Parquet files in your bucket are immutable. OpenSteward never rewrites or
deletes them. When a data-subject erasure removes records from OpenSteward, the
tombstones go to _deletes/, but erasing the data from files already
exported is your responsibility. For example, compact or rewrite partitions
with your lake’s tooling, or apply lifecycle rules. The same goes for Iceberg
tables. Postgres, Snowflake, BigQuery and Databricks destinations do apply the
deletes (warehouse time travel and fail-safe keep old versions for a while, per
your warehouse’s retention settings). See also
data erasure.
Configuration per type
Section titled “Configuration per type”Secrets are write-only. The API stores them encrypted and returns only
secretFieldsSet, the names of the fields that hold a value.
| Type | Config | Secrets |
|---|---|---|
s3 |
bucket, region (default us-east-1), prefix, endpoint (MinIO/R2/…), forcePathStyle (MinIO: true), compression |
accessKeyId + secretAccessKey (+ sessionToken). Omit both to use the default AWS credential chain (instance role, IRSA, env). |
gcs |
bucket, prefix, projectId, apiEndpoint, compression |
credentialsJson (a service-account key). Omit it to use Application Default Credentials. |
azure_blob |
accountName, container, prefix, endpoint (Azurite), compression |
accountKey or sasToken (container SAS with create + write) |
postgres |
host, port (5432), database, user, schema (opensteward), ssl (disable / require / verify-full, default require) |
password |
snowflake |
account, host, warehouse, database, schema, role, stage (details) |
username + password or privateKey (+ privateKeyPassphrase), stage* |
bigquery |
projectId, dataset, location, gcsBucket, gcsPrefix (details) |
serviceAccountJson (or Application Default Credentials) |
databricks |
host, httpPath, catalog, schema, stage (details) |
token, stage* |
iceberg (experimental) |
catalogUri, warehouse, namespace, storage (details) |
token, stage* |
stage* are the credentials a loader uploads staged files with:
stageAccessKeyId + stageSecretAccessKey (+ stageSessionToken) for S3
(omit both for the default AWS chain), stageCredentialsJson for GCS (omit for
Application Default Credentials), and stageAccountKey or stageSasToken for
Azure.
Every name that reaches warehouse SQL (databases, schemas, datasets, catalogs,
roles, storage integrations and credentials) must match a strict pattern
(letters, digits and _, plus $ in Snowflake names and - in BigQuery
project ids). It is quoted even then.
GET /admin/destinations/types returns every type’s config and secret fields
as JSON Schema, with descriptions and defaults.
Private hosts (SSRF guard)
Section titled “Private hosts (SSRF guard)”The worker connects to whatever host a destination names, from inside your
network. Every host a destination names must resolve to public addresses:
a custom S3 endpoint, a GCS or Azure endpoint, a Postgres host, a Snowflake
account host (<account>.snowflakecomputing.com or host), a Databricks
workspace host, or an Iceberg catalog URL. The check runs when the destination
is saved and again before every run. BigQuery and the cloud storage endpoints
the SDKs build themselves aren’t operator-controlled and aren’t checked.
Snowflake PrivateLink hosts resolve to private addresses, so list them here.
To point a destination at a private host on purpose, such as MinIO next to the stack or a Postgres on your LAN, list that host name:
DESTINATIONS_ALLOW_PRIVATE_HOSTS=minio,warehouse.internalEnvironment
Section titled “Environment”| Variable | Default | Effect |
|---|---|---|
DESTINATIONS_WORKER |
on | off stops the scheduler. Manual runs through the API still work. |
DESTINATIONS_WORKER_INTERVAL_MS |
30000 |
How often the scheduler looks for due destinations. |
DESTINATIONS_ALLOW_PRIVATE_HOSTS |
(none) | Host names exempt from the SSRF guard. |
Destination credentials are encrypted with STORE_SECRET in the data-store
database (resources.db on SQLite). When you rotate the secret, pass that file
to pnpm rotate-store-secret too (see persistence).
From the console
Section titled “From the console”Operators with the destinations:manage permission (admins by default) find
Destinations in the admin console’s Admin group:
- List (
/destinations): each destination in your tenant scope with its type, scope, schedule and the status, time and error of its last run. - New destination (
/destinations/new): pick a type, then fill in a form generated from the type catalog. Nested settings, such as a loader’sstage, get their own fieldset. Credential fields are password inputs that are never filled back in. Types marked experimental (Iceberg) carry a badge. The tenant and vertical picker offers only what you may grant: a tenant-restricted operator sees only their own tenants and can’t pick “every tenant”. Export personal data unmasked appears only if you holddata:read_pii. - Detail (
/destinations/:id): the settings, the names of the credential fields that are set, and the run history with the files each run wrote. From here you can Test connection, Run now, change the schedule, Backfill (all, one vertical, or one resource type), replace the credentials, or delete the destination. Backfill, credential replacement and delete ask for confirmation first.
The console calls admin-api’s /api/destinations/* routes, which mirror the
ones below and need the same permission. admin-api sends your scope and, only
if you hold data:read_pii, the right to approve a PII export. It audits every
change as destination.create, destination.update, destination.delete,
destination.run, destination.backfill, destination.credentials_rotate or
destination.test, and credential values never appear in the audit log.
API (integration-server, internal)
Section titled “API (integration-server, internal)”admin-api proxies these routes. Every route needs X-Admin-Secret and, except
/types, the caller’s X-OpenSteward-Data-Scope. Write bodies carry actor,
the admin user id, which is recorded as createdBy and piiApprovedBy.
| Route | Does |
|---|---|
GET /admin/destinations/types |
Type catalog with config and secret JSON Schemas. |
GET /admin/destinations |
Destinations visible to the caller. |
POST /admin/destinations |
Create: { name, type, config, secrets?, scope, includePii?, intervalSec?, enabled?, actor }. Returns 201. |
GET / PATCH / DELETE /admin/destinations/:id |
Read, patch (name, config, scope, includePii, intervalSec, enabled), or delete. |
PUT /admin/destinations/:id/credentials |
Replace (rotate) the secrets: { secrets, actor }. |
POST /admin/destinations/:id/test |
Build the sink and write the probe object, check Postgres CREATE, or round-trip a probe file through a loader’s stage and warehouse. Returns { ok, error?, latencyMs }. |
POST /admin/destinations/:id/run |
Start a run now. Returns 202 { runId }, or 409 RUN_IN_PROGRESS or 409 DESTINATION_DISABLED. |
POST /admin/destinations/:id/backfill |
Reset cursors (all, or { vertical, resourceType? }), then start a run unless run: false. |
GET /admin/destinations/:id/runs?limit= |
Run history, newest first. |
Errors use { error: { code, message, details? } }:
400:INVALID_BODY,INVALID_CONFIG,UNSUPPORTED_TYPE(a type the catalog doesn’t list) orHOST_NOT_ALLOWED.403:SCOPE_EXCEEDS_CALLERorPII_NOT_PERMITTED.404:DESTINATION_NOT_FOUND. This is also the answer for a destination outside the caller’s scope.409:DATA_STORE_DISABLED,RUN_IN_PROGRESSorDESTINATION_DISABLED.
Try it with MinIO and DuckDB
Section titled “Try it with MinIO and DuckDB”docker run -d --name minio -p 9000:9000 minio/minio server /data# DESTINATIONS_ALLOW_PRIVATE_HOSTS=localhost for integration-server, then create:# type s3, config { bucket: "lake", endpoint: "http://localhost:9000",# forcePathStyle: true }, secrets { accessKeyId: "minioadmin",# secretAccessKey: "minioadmin" }duckdb -c " INSTALL httpfs; LOAD httpfs; SET s3_endpoint='localhost:9000'; SET s3_use_ssl=false; SET s3_url_style='path'; SET s3_access_key_id='minioadmin'; SET s3_secret_access_key='minioadmin'; SELECT resource_type, count(*) FROM read_parquet('s3://lake/**/*.parquet', hive_partitioning=1) WHERE vertical IS NOT NULL GROUP BY 1;"opensteward.dev ·Plans and free license ·Terms ·Privacy ·support@opensteward.dev
© 2026 Falco Consulting AB. Built from open-steward/opensteward@a3961d8. This site sets no cookies, runs no analytics and loads nothing from other sites, except platform logos from Brandfetch on the Platforms page.