Materialization
Materialization turns exact committed source ranges into higher-generation physical representations. It is a background data-plane workflow, not a second logical commit protocol. A task may prepare or publish bytes, but only the generation-index CAS changes ordinary read selection.
Discovery without ownership coupling
Materialization must also process cold or unloaded streams. A sharded stream registration is a bounded discovery hint for scanners; it is not visibility truth and does not authorize deletion. After finding a candidate stream, the scanner rereads authoritative projection/binding, head, generation index, profile, task, and checkpoint state. Registration scans use fixed shards and bounded pages instead of loading every stream into memory.
Planner and durable task
The planner considers committed generation indexes, policy, source size/count/age, trim state, activation, registration, and existing work. It emits an immutable source set and target range with bounded source count, record count, and metadata bytes. Source entries freeze stream, view, range, generation, target identity, checksum, and index metadata version.
The durable task separates workflow progress from read visibility:
| State | Meaning |
|---|---|
PLANNED | Source set and target policy are frozen |
CLAIMED | A worker owns a versioned lease |
OUTPUT_READY | Output passed format and identity verification |
PUBLISHING | Final source/target/authority checks are in progress |
PUBLISHED | Workflow reports publication, but the generation index remains the visibility authority |
RETRY_WAIT | Retry is scheduled under a bounded policy |
FAILED / CANCELLED | No reader selection; evidence remains for recovery/diagnosis |
Multiple workers claim by version-CAS and heartbeat a lease. A new worker reconstructs work from the durable task, source identities, and object metadata; it does not reuse a dead process’s staging handle.
Source protection and exact reads
Before reading, the worker creates durable protection for every source generation/Object/BK range.
Protection includes task identity, source generation, root lifecycle epoch, and identity checksum.
If a generation-0 BK anchor has already retired before dynamic protection can be created, the task
ends as SOURCE_RETIRED rather than retrying an impossible source forever.
Every source read revalidates:
- stream and read view;
- half-open range and generation;
- index key, metadata version, and index checksum;
- exact
ReadTargetand physical checksum; - source root health.
This prevents a planner snapshot from being mixed with bytes from a newer or different generation.
Bounded staging and output identity
Workers use backpressured source streams, a shared staging-byte budget, local spill files, sorted runs, bounded fan-in, and (for semantic compaction) multiple passes. Staging files are process-local resources, not recovery truth; close and cancel paths must return permits and remove them.
The output identity is based on exact content SHA-256 and a durable worker output-attempt ID. The generation number is deliberately not part of the object key: a response-loss retry can reuse a verified immutable output before a generation number is allocated. A key that has been physically deleted is never reused, avoiding late-PUT resurrection.
Guarded PUT and publication
Before and after upload, the worker verifies owner/authority, physical-root permission, key identity, declared and actual length, provider metadata/checksum, complete format, whole-file CRC/SHA, and source lineage. If a PUT succeeded but the response was lost, exact HEAD plus full verification finds the same output; the worker does not blindly write another key.
Publication then follows this order:
- Allocate a positive generation.
- Create a final index record in
PREPAREDstate. - Revalidate source, output root, task owner, activation, and protection.
- CAS
PREPARED -> COMMITTEDfor the exact generation-index key. - Transfer output protection from the task owner to the committed-index owner.
- Advance the advisory materialization checkpoint.
- Release task-temporary protection.
- Retire task metadata only after rereading the committed generation and all references.
Only step 4 changes normal read selection. A task marked PUBLISHED without that index CAS is not a
readable generation.
Checkpoints
Two checkpoint classes have different authority:
| Checkpoint | Purpose | Can replace head/index truth? |
|---|---|---|
| Materialization checkpoint | Reduce scanner/planner repetition and record progress | No; it is an advisory hint |
| Recovery checkpoint | Provide immutable proof sufficient to replace an old commit prefix during replay/index repair | It participates in retirement proof, but remains versioned and head-anchored |
Native Kafka adds NKC1 producer/transaction/segment/index state to accelerate partition open. No checkpoint may lead the current head or become an alternate Kafka internal-topic authority.
Close and drain
Runtime close stops new admission, permits at most one active pass plus one coalesced pending trigger, and waits for accepted provider work to reach a determinate outcome within a shutdown deadline. At the deadline, work is cancelled only through a recoverable path; completion must still be observed. Close is a lifecycle boundary, not a way to abandon an accepted object upload or publication future.