Six things in data engineering that actually break in production

A pipeline is idempotent when running it again with the same inputs leaves the world in the same state. Run it once, run it fifty times, the table looks identical.

This sounds academic until 3am, when a task failed halfway through writing, the retry kicked in, and your revenue table now has 1.4x the real revenue because the first attempt committed 40% of its rows before dying.

The only three write modes, ranked by how much they will hurt you

The default should be insert overwrite by partition. It is the cheapest form of correctness available: the write is a full replacement of a bounded slice, so the state after the run does not depend on the state before it.

# Without this, Spark's "overwrite" nukes the ENTIRE table, not just
# the partitions present in your dataframe. This one line has caused
# more data loss than any bug I have seen.
spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")

(df
  .repartition("event_date")          # one writer per partition, fewer small files
  .write
  .mode("overwrite")
  .partitionBy("event_date")
  .saveAsTable("analytics.fct_payment"))

Hidden inputs: the real cause of non-determinism

A pipeline is deterministic when output equals f(input partition). Anything else your code reads is a hidden input, and every hidden input is a future incident.

• current_timestamp(), now(), current_date() in the transformation body. If a backfill for March runs in August, every "days since" column is wrong.
• rand(), uuid(), monotonically_increasing_id(). Rerun, different values, downstream joins silently lose rows.
• Mutable dimension tables read as of now. Joining yesterday's facts to today's customer table gives you a different answer every day you rerun it.
• API or service calls inside the transform. The remote system moved on.
• Ordering assumptions. collect_list without an explicit sort returns whatever the shuffle gives you.

Late data: pick a lookback window, and pick it with evidence

Events arrive late. Mobile clients buffer offline, upstream services replay, a broker had a bad hour. So "compute yesterday, once" is wrong on every real system.

The pattern that works: every run reprocesses a rolling window, not just the newest day. Because the write is an overwrite by partition, reprocessing is free of side effects.

-- Runs every day. Rewrites the last 3 event-days from source truth.
-- Rerunning any date range is safe: same input, same output.
INSERT OVERWRITE TABLE analytics.fct_payment PARTITION (event_date)
SELECT
    payment_id,
    TO_DATE(event_time)          AS event_date,   -- from the DATA
    amount_usd,
    status
FROM raw.payment_events
WHERE TO_DATE(event_time)
      BETWEEN DATE_SUB('${run_date}', 2) AND '${run_date}';

How do you choose the window size? Measure it. Log ingest_time - event_time for a month and take the p99. If p99 is 26 hours, a 3 day lookback covers you with margin. If p99 is 9 days, you have an upstream problem to fix before you widen the window to 10 days and pay for it every single night.

Deduplication that actually holds

At-least-once delivery is the norm in every message system worth using. So dedup is not optional, it is a layer.

WITH ranked AS (
  SELECT *,
    ROW_NUMBER() OVER (
      PARTITION BY payment_id                -- the true business key
      ORDER BY     source_updated_at DESC,   -- newest wins
                   ingest_sequence   DESC    -- tiebreak, never leave ties
    ) AS rn
  FROM raw.payment_events
  WHERE event_date BETWEEN '${start}' AND '${end}'
)
SELECT * EXCEPT (rn) FROM ranked WHERE rn = 1;

Three details separate this from the version that fails in six months:

1. Always include a tiebreaker. If two rows share the same key and same timestamp, ROW_NUMBER picks arbitrarily, and "arbitrarily" changes between runs. That alone destroys idempotency.
2. Order by the source's clock, not yours. Ingest time tells you when you saw it. Source updated time tells you which version is truer. For CDC use the log position (LSN, SCN, binlog offset), which is monotonic by construction.
3. Dedup inside the reprocessing window, not globally. A global dedup scans the whole table on every run and gets slower forever.

Backfills: one code path or none

The single strongest predictor of backfill pain is having a separate backfill script. It drifts from the production job within two sprints, and then your history and your present are computed by different logic.

Three operational rules that make large backfills survivable:

• Chunk by partition and run with bounded parallelism. Fifty concurrent partition rewrites will evict your production queries from the cluster and page someone else.
• Write to a shadow table first for anything above a week. Compare row counts and key aggregates against production, then swap. This is the write, audit, publish pattern from Chapter 5.
• Keep a backfill ledger. One row per partition with status and run id. When someone asks in November whether March 14 was reprocessed after the fix, you want an answer, not a Slack search.

Exactly-once, said honestly

There is no such thing as exactly-once delivery over a network. What exists is at-least-once delivery plus an idempotent sink, which produces exactly-once effects. Every system that markets exactly-once is doing one of these:

The checklist

Write the grain as a sentence before you write DDL

The grain is what one row means. If you cannot say it in one sentence with no "and also", the model is wrong and no amount of indexing will save it.

• Good: "One row per payment attempt."
• Good: "One row per customer per calendar day."
• Bad: "One row per payment, and for refunds one row per refund line, and sometimes a summary row."

Mixed grain is the single most common modeling defect I see in code review. It shows up later as double counting that nobody can explain, because the SUM is technically correct and the table is not.

Four fact table shapes and when each is correct

Additivity is a property you must record

Every measure is one of three things, and the model should tell you which:

• Additive: sums across every dimension. Revenue, quantity, cost.
• Semi-additive: sums across some dimensions but not time. Account balance, headcount, inventory level. Summing 30 daily balances gives a meaningless number.
• Non-additive: ratios, percentages, averages, distinct counts. Never store a ratio. Store the numerator and denominator and let the ratio be computed at query time, otherwise every rollup is wrong.

Slowly changing dimensions, decided in 30 seconds

A customer changes address. Do the old orders belong to the old address or the new one? That question, and only that question, decides your SCD type.

-- Step 1: hash only the columns whose change should create a new version.
-- Include a noisy column like last_login_at and you will version every day.
WITH src AS (
  SELECT customer_id, segment, country, plan_tier,
         SHA2(CONCAT_WS('|', segment, country, plan_tier), 256) AS row_hash,
         source_updated_at
  FROM staging.customer
)
MERGE INTO dim.customer AS t
USING src AS s
  ON t.customer_id = s.customer_id AND t.is_current = true

-- Step 2: close the old version. Do NOT overwrite its attributes.
WHEN MATCHED AND t.row_hash <> s.row_hash THEN UPDATE SET
  t.valid_to = s.source_updated_at,
  t.is_current = false

-- Step 3: insert the new version, open ended.
WHEN NOT MATCHED THEN INSERT (
  customer_sk, customer_id, segment, country, plan_tier,
  row_hash, valid_from, valid_to, is_current
) VALUES (
  SHA2(CONCAT_WS('|', s.customer_id, s.source_updated_at), 256),
  s.customer_id, s.segment, s.country, s.plan_tier,
  s.row_hash, s.source_updated_at, '9999-12-31', true
);

Surrogate keys: hash beats sequence in a distributed system

An auto-increment surrogate key requires a global counter, which means coordination, which means a bottleneck and a non-deterministic value on rerun. A hash key (sha256 of the natural key plus the version timestamp) is computable independently on every executor, is identical on every rerun, and lets you build fact and dimension tables in parallel without waiting for the dimension load to publish keys.

The costs are real and worth naming: 32 bytes instead of 8, no ordering, and no visual debuggability. On columnar storage with dictionary encoding the size cost is usually noise. Take the determinism.

Late-arriving facts and the as-of join

Here is where most SCD2 implementations are quietly wrong. Your fact arrives with an event time. Your dimension has versions. The join must select the dimension version that was in effect at the event time, not the current one.

SELECT f.payment_id, f.amount_usd, d.plan_tier
FROM   fct_payment f
JOIN   dim_customer d
  ON   f.customer_id = d.customer_id
 AND   f.event_time  >= d.valid_from
 AND   f.event_time  <  d.valid_to;   -- half-open interval, never BETWEEN

Use a half-open interval. If you use BETWEEN valid_from AND valid_to and the boundaries touch, an event at the exact boundary matches two versions and you have silently doubled that row.

The dimension that has not arrived yet

A fact references customer 8812. The customer dimension has never seen 8812. Options, in order of how much I like them:

1. Inferred member. Insert a placeholder dimension row with the key and "Unknown" attributes, then update it when the real record lands. The fact joins successfully and nothing is lost.
2. Route to a quarantine table and reprocess after the dimension load. Correct, but it puts business data in a place nobody queries, so it needs an owner and an alarm.
3. Drop the row. Never. This is silent data loss and it will be discovered by finance, not by you.

Star schema or one big table? Both, on purpose

The argument is not really about joins. On columnar engines with predicate pushdown, broadcast joins to small dimensions are close to free. The argument is about where the definition lives.

My rule, and it has held across a payments warehouse and a petabyte-scale analytics estate:

• Star schema is the semantic layer. Business logic, SCD history and conformed dimensions live here. This is what an auditor reads.
• Wide denormalized tables are a cache. Built from the star, for high-traffic dashboards, for ML feature pipelines, and for anyone who is going to write their own SQL anyway. Rebuild them, do not hand-edit them.
• If a metric is defined in the wide table and nowhere else, you have lost. Two teams will define it twice, differently, and both will be defensible.

The snapshot storage math, so you go in with your eyes open

A 100 GB dimension snapshotted daily costs about 36 TB per year before compression gains. Mitigations, in order of effort: shorten retention on old partitions, snapshot weekly beyond 90 days, or replace snapshots with SCD2 plus a view that reconstructs any as-of date. Do the arithmetic before you commit, not in the FinOps review.

What was broken about a Hive table

A Hive table was a convention: a directory, some subdirectories named dt=2026-08-01, and Parquet files inside. The metastore stored the schema and the partition list. That convention breaks in four specific ways.

• No atomic commit. A writer produces files, then the reader sees them mid-write. There is no moment where the table flips from old state to new state.
• Planning means listing. To find files, the engine lists object storage prefixes. On S3 that is a paginated API call per prefix, and a table with 200,000 partitions spends minutes just deciding what to read.
• Schema evolution by position. Rename a column and old files no longer match. Reorder columns and you get silently wrong data, not an error.
• Partitioning leaks into every query. The user has to know that dt exists and write WHERE dt = '2026-08-01' AND event_time >= .... Forget the first predicate and you scan five years.

How Iceberg is put together

Read that diagram right to left and the properties fall out:

1. Manifest files record every data file plus per-column min, max and null counts. The planner can skip a file without opening it.
2. A manifest list is one snapshot: the exact set of manifests, and therefore files, that make up the table at a point in time.
3. metadata.json holds the schema, partition specs, sort orders and the history of snapshots.
4. The catalog holds one pointer to the current metadata file. A commit is a compare-and-swap on that pointer.

Because the commit is optimistic, two concurrent writers can conflict. The loser re-reads the new snapshot and retries. This is fine for a handful of writers and turns into a livelock at high concurrency on the same partitions, which is why streaming writers should target distinct partitions or use a single writer per table.

Hidden partitioning is the feature people underrate

In Hive you materialize a dt column and pray users filter on it. In Iceberg you declare a transform, and the engine derives the partition value from a real column.

CREATE TABLE analytics.fct_payment (
  payment_id   STRING,
  event_time   TIMESTAMP,
  merchant_id  STRING,
  amount_usd   DECIMAL(18,2)
)
PARTITIONED BY (days(event_time), bucket(32, merchant_id));

-- Users just write this. No dt column, no leaked implementation detail.
SELECT * FROM analytics.fct_payment
WHERE event_time >= '2026-08-01';

-- Volume grew 10x. Change the layout without rewriting history.
-- Old files keep the old spec, new files use the new one.
ALTER TABLE analytics.fct_payment
  REPLACE PARTITION FIELD days(event_time) WITH hours(event_time);

Partition evolution is the part with no Hive equivalent. Changing a Hive partition scheme means rewriting the entire table. Iceberg stores the spec id per data file, so old and new layouts coexist and the planner handles both.

Copy on write versus merge on read

Updating a row in an immutable file format means one of two strategies, and the choice is a pure read-versus-write trade.

The small file problem, with the arithmetic

Say a streaming job writes every minute. That is 1,440 files a day per partition. Over a year, half a million files in one table. Each file costs a metadata entry, a task slot, and at least one object storage request.

Target file size is 128 MB to 512 MB for analytics scans. Below about 32 MB, per-file overhead dominates and your job spends more time scheduling than reading.

Delta and Hudi, fairly

Delta Lake keeps a _delta_log directory of ordered JSON commits, with a Parquet checkpoint every ten commits so readers do not replay the whole history. Its strongest cards are deletion vectors (merge on read without rewriting files), OPTIMIZE with Z-ordering for multi-column locality, liquid clustering as the newer alternative to fixed partitioning, and the deepest Spark integration in existence. Its weakness has historically been the tightness of that Databricks coupling, though the open protocol has improved a lot.

Hudi was built for a specific job: high-frequency upserts keyed by a record id, with fast incremental pulls. Its record-level index means it can find the file holding a key without a full scan, which is exactly what a CDC sink needs. It is the most operationally complex of the three and repays that complexity only if your workload really is upsert-dominated.

The four maintenance jobs everyone forgets

-- 1. Compact small files into target-sized ones.
CALL catalog.system.rewrite_data_files(
  table => 'analytics.fct_payment',
  options => map('target-file-size-bytes', '268435456')
);

-- 2. Merge tiny manifests. Planning time is proportional to manifest count.
CALL catalog.system.rewrite_manifests('analytics.fct_payment');

-- 3. Expire old snapshots. This is what actually frees storage.
--    It also sets your time-travel window. Coordinate with compliance.
CALL catalog.system.expire_snapshots(
  table => 'analytics.fct_payment',
  older_than => TIMESTAMP '2026-08-01 00:00:00',
  retain_last => 10
);

-- 4. Delete files no snapshot references (failed writes leave these behind).
CALL catalog.system.remove_orphan_files(
  table => 'analytics.fct_payment',
  older_than => TIMESTAMP '2026-08-20 00:00:00'
);

The three clocks

• Event time. When the thing happened, stamped by the producer. This is the only clock that defines correctness.
• Ingestion time. When the broker received it. Monotonic and cheap, useful for measuring lag, useless for business logic.
• Processing time. When your operator touched it. Fast, non-deterministic, and different on every replay.

If you window by processing time, a replay of yesterday's data produces different results than the original run. That is not a streaming system, that is a random number generator with a Kafka dependency.

A watermark is a claim, not a fact

A watermark of time W is the system asserting: I do not expect any more events with event time earlier than W. It is a heuristic. You choose how aggressive it is, and you are choosing between latency and completeness.

The usual construction is watermark = max_observed_event_time - allowed_out_of_orderness. Set the bound to 5 seconds and results are fast but drop stragglers. Set it to 6 hours and results are complete but nothing emits for 6 hours.

Where late data goes

Three destinations, and you must pick one explicitly:

1. Re-fire the window. The window emits an updated result. Downstream must handle retractions or upserts, not blind appends.
2. Side output. Late events go to a separate stream or table, get counted, and are reconciled by a nightly batch job. This is the lambda-flavoured answer and it is usually the pragmatic one.
3. Drop. Fine, if you emit a metric and alarm on it. Not fine as an accident.

State is the real operational limit

Any stateful operator (window, join, aggregate, dedup) holds data between events. That state is the thing that will page you, not CPU.

• Key skew. State is partitioned by key. One hot key, say a merchant that is 40% of traffic, puts 40% of the state on one task manager. That task lags, alignment stalls, checkpoints time out. Detect it by plotting state size and lag per subtask, not per job.
• Unbounded growth. A dedup keyed on event id with no TTL grows forever. Setting a TTL is not a memory optimization, it is a declaration that duplicates arriving after the TTL will not be caught. That is a correctness statement and it belongs in the contract.
• Restore compatibility. If you do not assign stable operator UIDs, changing the job graph makes your savepoint unrestorable, and your only option is starting from a fresh offset. Assign UIDs on day one.
• Backend choice. Heap state is fast until GC pauses; RocksDB spills to disk and handles state larger than memory at the cost of serialization on every access. Above a few GB per task, RocksDB.

Kafka details that decide your design

Change data capture: the pattern behind most real streams

Most "streaming" in enterprises is CDC: replicating an operational database into analytics in near real time. Query-based CDC (polling WHERE updated_at > last_seen) is easy and wrong in three ways: it misses deletes, it misses intermediate states, and it misses rows committed out of order relative to their timestamp. Log-based CDC reads the database's own write-ahead log, so it catches every change including deletes, in commit order.

The part that bites is the handoff. You need a consistent snapshot of the existing table, plus the stream of changes since, with no gap and no unhandled overlap. Modern connectors do incremental snapshotting with a watermark trick so they never lock the source table, but you should still verify the boundary yourself with row counts and a checksum on a stable key range.

The uncomfortable part: you probably do not need streaming

Streaming costs roughly three to five times more engineering time than the batch equivalent, forever. State management, checkpoint tuning, exactly-once semantics, replay procedures, on-call depth: all of it is permanent overhead.

The question to ask a stakeholder is not "how fresh do you want this data". Everyone answers "real time" to that. Ask instead: what decision will you make differently if this arrives in one minute rather than in one hour? If there is no answer, you have your requirement.

Five ways data goes wrong

That last row is the killer. Upstream redefines "active user" from 30 day to 7 day. No test fires. The schema is identical, the volume is plausible, the nulls are fine. Three weeks later someone notices the board deck disagrees with the product dashboard. Semantic drift is invisible to schema validation, which is exactly why data contracts exist.

A data contract is five things, not a schema file

1. Schema. Fields, types, nullability, and enumerated values.
2. Semantics. What each field means in prose, with the definition of any derived or business term.
3. Guarantees. Uniqueness, referential integrity, value ranges, expected volume band.
4. SLA. Freshness and availability commitments, plus what happens when they are missed.
5. Change policy. Who owns it, what counts as breaking, how much notice consumers get.

Compatibility modes and the deploy order they imply

The data test pyramid

Why your static thresholds fire every Monday

"Alert if row count drops more than 20 percent" is the classic. Weekend traffic is 40 percent below weekday traffic, so it alerts every Saturday until someone mutes it, and then it is not a monitor, it is decoration.

The minimum viable seasonal check compares today to the same weekday over the last four weeks, and flags on median absolute deviation rather than percentage. It is a handful of lines of SQL and it removes most false positives.

WITH daily AS (
  SELECT event_date, COUNT(*) AS rows_loaded
  FROM analytics.fct_payment
  WHERE event_date >= DATE_SUB('${run_date}', 28)
  GROUP BY 1
),
baseline AS (                                   -- same weekday, prior 4 weeks
  SELECT PERCENTILE_CONT(rows_loaded, 0.5) AS med,
         PERCENTILE_CONT(ABS(rows_loaded - AVG(rows_loaded) OVER ()), 0.5) AS mad
  FROM daily
  WHERE DAYOFWEEK(event_date) = DAYOFWEEK(DATE '${run_date}')
    AND event_date < '${run_date}'
)
SELECT
  d.rows_loaded,
  b.med,
  ABS(d.rows_loaded - b.med) / NULLIF(b.mad, 0) AS deviation_score,
  CASE WHEN ABS(d.rows_loaded - b.med) > 4 * b.mad
       THEN 'FAIL' ELSE 'PASS' END AS status
FROM daily d CROSS JOIN baseline b
WHERE d.event_date = '${run_date}';

Write, audit, publish

The pattern that separates teams who ship bad data from teams who catch it: never write directly to the table consumers read.

1. Write to a staging location, or an Iceberg branch, or a snapshot that is not yet the current pointer.
2. Audit by running assertions against that staged output. Row counts, key aggregates, referential checks, comparison to the previous run.
3. Publish by an atomic swap, only if the audit passes. Iceberg branches make this a metadata operation with no data movement.

-- WRITE: land the run on an isolated branch. Readers see nothing.
ALTER TABLE analytics.fct_payment CREATE BRANCH etl_run_8891;
INSERT INTO analytics.fct_payment.branch_etl_run_8891 SELECT ... ;

-- AUDIT: assert against the branch, not against production.
SELECT
  COUNT(*)                                       AS row_count,
  COUNT(*) - COUNT(DISTINCT payment_id)        AS dupes,
  SUM(CASE WHEN amount_usd < 0 THEN 1 ELSE 0 END) AS negatives
FROM analytics.fct_payment.branch_etl_run_8891
WHERE event_date = '${run_date}';

-- PUBLISH: atomic. Consumers move from old state to new in one step.
CALL catalog.system.fast_forward('analytics.fct_payment', 'main', 'etl_run_8891');

Blocking versus alerting: classify every check

If every check blocks the DAG, one flaky assertion halts the warehouse at 4am and someone disables checks entirely. If nothing blocks, garbage publishes. So classify:

• Blocking: violations that make the output actively wrong. Duplicate primary keys, null keys, negative amounts, row count near zero, referential breaks on a join key. Fail the run and page.
• Alerting: violations that are suspicious but survivable. Distribution shift, a new enum value, a slightly late arrival. Publish, notify the owner, review in the morning.
• Quarantine: a subset of rows fails while the rest are fine. Route the bad rows to a quarantine table with the failure reason, publish the good ones, and alarm on quarantine volume. Never silently drop.

Lineage is an on-call tool before it is a governance tool

Two ways to build it. Static lineage parses SQL and DAG definitions: complete coverage of what is declared, blind to anything dynamic. Runtime lineage reads query logs and execution plans: it captures what actually happened, including the ad hoc job someone runs from a laptop, but only after it has run at least once. Serious platforms do both and reconcile.

What makes it worth building is column-level granularity, because that is what answers the two questions that matter at 3am:

• This column is wrong. Which reports and models consume it, and who do I tell?
• I need to change this upstream field. What breaks?

SLOs, not test counts

Borrow the discipline from site reliability engineering. Define per dataset:

• Freshness SLO. "Available by 06:00 local, 99 percent of days in a rolling 30 day window."
• Completeness SLO. "Within 2 percent of the reconciled source count, 99.5 percent of days."
• Error budget. Nine misses a quarter is the budget. Spend it and feature work stops until reliability work lands.

Classify once, at the source, then propagate

Manual classification of derived tables does not scale and is never current. One table gets tagged, the four tables built from it do not, and the dashboard built from those is where the data actually leaks.

The pattern that works: tag columns at the source schema, then propagate tags along column-level lineage, with explicit downgrade rules.

Purpose limitation is a join, and it belongs in the access path

Data collected for fraud prevention should not silently become training data for ad ranking. Encoded as engineering, purpose limitation means every dataset carries allowed-purpose tags, every access request carries a declared purpose, and the policy engine intersects them before the query runs.

# Deny by default. Grant only when purpose, classification and
# the requester's clearance all line up. Every decision is logged.
allow {
    input.dataset.classification == "pii_restricted"
    input.request.purpose in input.dataset.allowed_purposes
    input.user.clearance   >= 3
    input.request.justification != ""
    time.now_ns() < input.user.grant_expires_ns   # access is temporary
}

mask[column] {
    column := input.dataset.columns[_]
    column.tags[_] == "pii_direct"
    not input.user.clearance >= 4              # mask instead of deny
}

Two design points worth stealing. Mask rather than deny where possible, so an analyst can still do their job on the non-sensitive columns instead of filing a ticket and losing a day. And make grants expire, because permanent access granted for one project is the single largest source of over-permissioning in every estate I have seen.

Deletion: the genuinely hard one

A user asks to be deleted. You delete their row from the source database. Then you remember where else it lives.

Whichever you pick, you need a subject-to-storage index: given a subject id, which tables, partitions and systems contain their data. Without it every request is a full scan of the estate, and you cannot meet a 30 day statutory deadline with a full scan.

De-identification, honestly ranked

1. Removing direct identifiers (name, email, phone) is a start, not anonymization. The classic result is that a large share of a population is uniquely identified by postcode, birth date and sex together.
2. k-anonymity requires each combination of quasi-identifiers to appear at least k times. It stops singling out, but if all k people share a sensitive value, you learn it anyway. That is what l-diversity patches.
3. Differential privacy adds calibrated noise so that the presence or absence of any single person barely changes the output, with a formal epsilon budget. It works well for aggregate counts and published statistics. It works badly for row-level analytics, and every query spends budget you cannot get back.

The practical stance: use tokenization and access control for operational data, k-anonymity plus suppression for internal analytical sharing, and differential privacy only where you publish aggregates externally. Anyone selling differential privacy as a general purpose privacy switch has not implemented it.

Consent is a dataset with an SLA

Consent state changes. A user opts out at 09:00. If your consent snapshot refreshes nightly, you have a fifteen hour window where pipelines process data you no longer have a basis to process. That is not a policy gap, it is a data freshness bug with legal consequences.

Treat consent as a first-class dataset: streamed rather than batched, joined at processing time rather than baked into a copy, with its own freshness monitor and an alarm that a compliance owner receives.

What agentic AI changed

Batch governance assumed a human wrote a query you could review. An agent composes tool calls at runtime, so the boundary has to be enforced in the moment.

• Provenance on training and retrieval corpora. Record where every document came from, under what basis, and with what retention. When a source must be removed, you need to know what was built from it.
• Runtime authorization, not design-time. The agent must act with the requesting user's permissions, not a broad service account. A service account with read-all is a data exfiltration path wearing a helpful interface.
• Prompts and outputs are logs containing personal data. They fall under the same retention and access rules as any other table, and they are almost always the least governed store in the company.
• Tenant isolation in retrieval. A vector index shared across customers with filtering applied after retrieval is a leak waiting for a bug. Partition the index, do not filter the results.

The starter set, in build order