Pipelines

A pipeline is a JSON document, and this page is its language. Webhooks and Request Bins run the same language on the same engine, so what you learn here holds there too, within each product's own header rules. To try any of this with no account, use the playground.

The document

This pipeline keeps paid invoices of $1,000 or more, removes the customer's email and any phone number, adds two fields and a header, and says a bearer token goes on the request:

{
  "version": 1,
  "steps": [
    { "filter": { "type": "invoice.paid", "data": { "amount": { "$gte": 100000 } } }, "note": "Paid invoices of $1,000 or more" },
    { "drop": ["data.customer.email", "**.phone"] },
    { "set": { "event_type": "{{ type }}", "amount_display": "{{ data.amount | money(data.currency) }}" } },
    { "headers": { "set": { "X-Event-Type": "{{ type }}" } } },
    { "auth": { "type": "bearer", "secret": "sec_…" } }
  ]
}

version is the document's format, always 1; it is not the pipeline's version. The steps run in the order written. Each is an object with one step key and, optionally, a note of up to 200 characters, which the console shows on the step and a drop reports. A filter can also take an action.

A run sees one request: a body, any JSON value; headers; and read-only meta. In the run API all three are whatever you send. Each step sees the body as the one before left it. Steps are numbered from 0 in the API and in error text, and from 1 in the console's editor.

filter

Keeps the request only if it matches; otherwise the run answers dropped. With "action": "drop" it works the other way round: what matches is dropped.

{ "filter": { "data": { "status": "paid", "amount": { "$gte": 100000 } } }, "note": "Paid, $1,000 or more" }

• Every key is a condition, and all of them must hold. A nested object tests by shape, as data does here.
• A plain value matches by JSON equality: 1 equals 1.0.
• An array matches when it equals the value or when one of its elements matches: {"data": {"tags": "urgent"}} holds for ["billing", "urgent"], and {"data": {"items": {"sku": "A-1", "qty": 2}}} needs one item with both.
• A field the request doesn't have matches only {"$exists": false} and the negations, such as $neq.
• The operators are $eq, $neq, $gt, $gte, $lt, $lte (numbers with numbers, text with text; ISO timestamps sort by time), $in, $nin, $exists, $startsWith, $endsWith, $contains (text, or an array holding the value), and $and, $or and $not to combine filters.
• $headers, $meta and $body test that part of the request: {"$headers": {"x-github-event": "push"}}.

drop

Removes fields from the body: a list of 1 to 50 paths. * matches any one level and ** any depth, so **.email removes every email field, wherever it is. A path that matches nothing is not an error.

{ "drop": ["data.customer.email", "data.cards[*].number", "**.phone"] }

set

Adds or overwrites fields: 1 to 50 pairs of a path and a value. It creates the objects on the way, and every value reads the body as the step found it. A path that runs into a value that isn't an object fails the run with set_conflict.

{ "set": { "event_type": "{{ type }}", "data.total_cents": { "$ref": "data.amount" } } }

map

Replaces the body with a value you write, usually an object.

{ "map": {
    "invoice_id": "{{ data.id }}",
    "amount": { "$ref": "data.amount" },
    "amount_display": "{{ data.amount | money(data.currency) }}",
    "event": "{{ type }}"
} }

On an invoice.paid request whose data is {"id": "in_42", "amount": 150000, "currency": "usd"}, the body becomes:

{ "invoice_id": "in_42", "amount": 150000, "amount_display": "1,500.00 USD", "event": "invoice.paid" }

headers

Sets headers with set, removes them with remove and renames them with rename, in the order remove, rename, set. Names match whatever their case, and a name that is set replaces every spelling of it.

{ "headers": { "set": { "X-Event-Type": "{{ type }}", "Content-Type": "application/json" } } }

A headers step can't set a header the connection owns, such as Host or Content-Length, or Set-Cookie. It can't set Authorization, Cookie or X-Api-Key either, because the value would sit in the document, which every read returns: a credential goes in an auth step. A product that runs the pipeline by reference also holds it to that product's own header rules.

auth

Says that a credential goes on the request, and names the slot it comes from:

{ "auth": { "type": "api_key", "in": "header", "name": "X-Api-Key", "secret": "sec_…" } }

secret is the id of a slot the workspace holds. It is never a value: a value pasted there is refused with invalid_secret, and the refusal doesn't quote it back. The product that runs the pipeline by reference maps the slot to its own secret (Credentials). Test and the run API show the header or parameter with the value masked.

Auth steps come after every other step, so no step can read, copy or test a credential.

Paths and values

A path names a field of the body:

set takes plain paths, drop takes * and **, and a value reads with * but not **.

A value in set, map or a header is JSON, copied as written, except for two things:

• {{ path }} in a string inserts a value as text. A missing value or null inserts nothing, and an object or a list goes in as JSON. {{ 'text' }} and {{ 42 }} are literals, so a literal {{ is written {{ '{{' }}.
• {"$ref": "path"} inserts the value itself and keeps its type: a number stays a number, a list a list. Add "default" for when the path is missing.

An expression can pass its value through filters, left to right: {{ data.amount | money(data.currency) }} gives 1,500.00 USD for an amount of 150000 in usd.

There are no loops, conditions or sums: a condition is a filter step, and a list becomes text through [*] and join.

Templates

The gallery holds ready-made pipelines to start from. In the console they sit under Start from a template. Through the API, GET /v1/pipelines/templates returns each one with its pipeline. Saving one makes a pipeline that is your own: change it like any other, and nothing refers back to the template.

Each chat template is a map step and a headers step that sets Content-Type: application/json, and it works on any event. A missing type reads "Event" and a missing id or time "—", the data is cut to fit the app's limits, and nothing in the event can format the message, mention anyone or end the code block early.

Limits

A pipeline has no loops, no clock and no network, and fixed caps bound each run. Past one, the run answers failed with the reason, and a document past one is refused when it is saved.

The playground has smaller caps: a 16 KiB document, a 32 KiB body, 16 KiB for one value a step renders, 20 headers, 1 KiB for one header value and 10,000 values read or copied in a run.

What's next

• Quickstart — test a pipeline, save it and run it.
• Transforms reference — every route and field, and what each refusal means.
• Transformations in Webhooks — the same language on a Webhooks endpoint.