Data engineering: reliable pipelines

1. Connections, secrets and the run folder

Set up each warehouse or mailbox once under Settings → Connections. Passwords go under Settings → Secrets and are referenced as {{secret:NAME}}; they are encrypted on disk, swapped in only at run time, and blanked out of logs. Test every connection with its Test button before a pipeline depends on it.

Every run has a folder (FLOWCALL_WORKDIR for Python and command tasks). SQL tasks save <task name>.csv there; API tasks save <task name>.response.txt; uploads live in FLOWCALL_LIBRARY. Nothing is shared between runs, which is why a retried task can never double-load data.

Do this: create a connection, test it, and point the template's SQL task at it.

Example used: Warehouse query to Excel report

2. Test before you schedule

The task editor's Try it button runs one task against a copy of the latest run's files and shows the log and the files it produced. Nothing is scheduled, nothing is recorded. Use it the way you would use a REPL: change the SQL, try it, look at the rows, change it again.

For a whole pipeline, use Run now while the pipeline is paused ("Run on schedule" unchecked). Turn the schedule on only when a manual run has passed end to end.

The example's query has a mistake in it on purpose: use Try it to find it, fix it, and try again.

Do this: change a query in a task, press Try it, and read the preview before saving.

Example used: Fix it before you schedule it

3. Gates, alerts and what to do at 3 a.m.

Put a quality check directly after every ingestion task and treat it as a gate: the pipeline should never publish data that failed a rule. Set alerts to "On failure" with an email and, if you have one, a Teams webhook, so the failure reaches a person with the last three log lines attached.

Retries with backoff handle the transient failures (a warehouse that was busy). Time limits handle the hung ones. Anything else should fail loudly rather than run late and wrong.

Do this: install the template, set alerts to On failure with your email, run it, and read the failure email.

Example used: Daily data quality gate

4. Guardrails that tell you the numbers

A rule that passes tells you nothing about drift. Every quality check now writes a short summary: the row count against the last run and against the usual (the median of the last ten runs, so one bad day does not skew it), any column whose total moved more than 15%, blank rates that jumped, and the rules that failed.

On the check task, 'Email me the numbers' sends that summary on failure, when something moves, or every run. 'When something moves' is the setting most teams keep: quiet on normal days, specific on the day the payments file is half empty.

Do this: install the Morning data guardrail, run it three or four times, then shrink payments.csv in Files and run it again. Read what moved.

Example used: Morning data guardrail

5. Versions, GitHub and CI

Every save is a version; restore any of them from the Pipeline tab. For team work, connect a GitHub repository under Settings → GitHub: pipelines push as JSON files, one per pipeline, and pull back.

For CI/CD: create an API token under Settings → Account, add it to your repository's secrets, and use the workflow file shipped in ci/ — on every pull request it validates every pipeline JSON (schema, loops, rule syntax), and on merge to main it deploys them to your Flowcall server with tools/cadence_ci.py deploy.

Do this: run python tools/cadence_ci.py validate pipelines/ against an exported pipeline.

Example used: Morning data guardrail

6. Reading a mailbox as a data source

Payment processors and partners send files by email; the Read-a-mailbox task turns that into a table. Set up an IMAP connection (Gmail with an app password, or a shared mailbox), filter by sender and subject, save attachments matching .csv, .xlsx, and the task produces messages.csv plus the files in attachments/. Chain a quality check and a Python task to turn the attachment into your extract.

To practise without a mailbox, the example ships sample messages in exactly the shape the mailbox task produces; replace its first step with a Read-a-mailbox task when you are ready.

Do this: add a mailbox connection, test it, and build a three-task pipeline: read mailbox → check → export.

Example used: Turn a billing inbox into a table

7. Know what it costs, and who can change what

Settings → Usage & cost shows what each pipeline consumes on this machine: machine time, peak memory, rows produced and AI tokens, priced at rates you set. Waiting for a person is never counted as machine time. A monthly budget warns you before a runaway model call does.

Settings → Users sets who does what: owners manage people, secrets and the plan; editors build, run and decide; viewers look and download. Approval steps can name exactly who may decide.

Do this: set a monthly budget, add a viewer, and sign in as them to see what they can and cannot touch.

Example used: See what a pipeline costs