Management API — contract & automation

Auth & integration assumptions

• Workspace scope: the Bearer token is a workspace API key; every management URL operates on that workspace’s pipes, keys, and limits.
• Ingest path unchanged: POST https://data.batchpipe.com/v1/ingest/:pipe_id (and related ingest docs) stay the data path; management keys can create sibling keys and must be stored like root credentials.

Validation errors (400)

Failed Zod validation returns JSON shaped as:

{
  "error": "validation_error",
  "message": "Request body failed validation",
  "errors": {
    "pipe_name": "Pipe name can only contain letters, numbers, hyphens, and underscores",
    "columns.0.column_name": "Column name is required",
    "destination_config.url": "URL is required for HTTP endpoint destinations"
  }
}

errors is a flat object: keys are Zod issue.path segments joined with . (array indices appear as digits, e.g. columns.2.source_field). Values are human-readable strings (not machine codes). Implementation: middleware/validation.js.

See also the HTTP API — validation section.

Secret redaction (GET pipe / destination payloads)

Before JSON is returned, sensitive fields in destination_config are replaced with the string [redacted] (never omitted — the key stays present so clients know the field exists):

GET /v1/api-keys never returns full secrets — only api_key_prefix and metadata (lib/domain/api_key.js).

Postgres / MySQL destination_config

Delivery reads host, port (default 5432 / 3306), database, username, password, table, and optional dialect / sql_dialect (lib/delivery/database_destination.js). There is no per-destination JSON flag for TLS today: TLS is used when DESTINATION_DB_SSL=true or when host is not localhost / 127.0.0.1, with rejectUnauthorized: false for the client pool.

Column mapping: each row maps a SQL column to a top-level JSON key on ingested records via source_field (lib/delivery/record_value.js — nested JSONPath not supported). column_type is one of the JSON types string|number|boolean|object|array|null before casting to SQL.

PATCH …/destinations/:id — empty string keeps secrets: if destination_config.password (database), secret_key (object store), or auth_header (HTTP) is sent as "", the server keeps the previously stored value after shallow-merge (routes/management_api_routes.js).

Replacing column mappings: sending columns on PATCH replaces all destination columns (delete + insert) for database destinations.

Server defaults (create pipe / destination)

• POST /v1/pipes — pipe_status is always active. pipe_add_ingest_ts defaults true; pipe_add_ingest_ip defaults false; geo dimensions use nullable pipe_ingest_geo_*_field targets (unset = disabled); pipe_ingest_ts_field / pipe_ingest_ip_field default to ingest_ts / ingest_ip; pipe_api_key_required defaults true; pipe_allow_any_browser_origin defaults false (lib/domain/pipe.js).
• Uniqueness: (workspace_id, pipe_name) is unique — duplicate returns 409 duplicate_pipe_name.
• POST destination — destination_status starts active. Delivery retries until success, Skip, or JetStream retention drops the message.
• GET /v1/pipes/:pipe_id — includes each database destination’s columns array (joined server-side) so automation can read mappings without a second round-trip.

Ingest geo enrichment

Each pipe can enable zero or more geo dimensions independently. On POST https://data.batchpipe.com/v1/ingest/:pipe_id, the data plane resolves the client IP (same source as pipe_add_ingest_ip, but geo does not require pipe_add_ingest_ip to be enabled) and, when GeoLite2-City is installed, writes each configured dimension to the target JSON key on the enriched record (lib/ingest_record_enrichment.js, lib/geoip_city_lookup.js).

Deployment: install GeoLite2-City.mmdb on every data-plane instance (npm run geolite:download; optional GEOIP_CITY_DB_PATH). If the database file is missing, geo targets are silently skipped — ingest still succeeds.

Management API: set each pipe_ingest_geo_*_field on POST /v1/pipes or PATCH /v1/pipes/:id to the top-level JSON key name you want on ingested records. Omit the property or send null to disable that dimension. Values are trimmed strings, max 100 characters.

If lookup fails or a dimension has no value, that key is not added to the record (other dimensions and your payload fields are unchanged).

Destination wiring: database destinations map enriched keys with columns[].source_field — use the same string you set on the pipe (e.g. ingest_country_code). See the OpenAPI browserIngestWithGeo + postgresWithGeoColumns examples under POST /v1/pipes and POST …/destinations. Glossary: Ingest geo enrichment.

Minimal end-to-end (conceptual JSON)

Replace UUIDs and secrets; order matters only where a step needs a prior pipe_id / destination_id.

1. POST /v1/pipes — create pipe (see OpenAPI browserIngestWithGeo example for enrichment + geo targets).
2. POST /v1/pipes/{pipe_id}/destinations — attach Postgres destination + columns (see OpenAPI postgresWithGeoColumns when using geo enrichment).
3. PUT /v1/pipes/{pipe_id}/limits — set batching / rate caps.
4. POST /v1/pipes/{pipe_id}/allowed-origins — if the pipe will accept browser ingest without a per-request key.
5. POST /v1/ingest/:pipe_id — send JSON objects whose top-level keys match source_field mappings (see ingest reference).

Keys, limits, rate limits

• Key scoping: prefer a dedicated automation key per environment, name it clearly (api_key_name), rotate by creating a new key and revoking the old one (DELETE /v1/api-keys/:id cannot revoke the key used for the current request).
• Management rate limits: there is no workspace-specific Express rate limiter on /v1/pipes… today (unlike auth forms). Provision flows should still backoff on 429 if introduced at the edge.
• OpenAPI URL: stable path on each deployment: https://batchpipe.com/openapi/management-v1.openapi.yaml