Example: Kafka transaction and DeleteRecords
This example keeps Kafka’s protocol state separate from Nereus’s committed bytes.
Initial state
| Fact | Value |
|---|---|
| Topic ID / partition | T1 / 0 |
| Binding stream | S-kafka-1 |
logStartOffset | 0 |
| Stable end / HW / LSO | 100 / 100 / 100 |
| Request | Open transaction, one RecordBatch with four records |
Produce and transaction visibility
- KRaft confirms the current leader epoch.
- Kafka validates producer ID, epoch, sequence, and transaction state.
- Kafka assigns base offset
100. - The adapter stores the complete RecordBatch as one ranged entry with
recordCount=4and range[100,104). - Nereus WAL and head commit; stable end becomes
104. - Kafka records the open transaction in ProducerStateManager.
- HW may advance to
104, while LSO remains100. - Produce completes.
READ_UNCOMMITTED can read within HW. READ_COMMITTED cannot expose the open transaction until the
Kafka commit marker and transaction state advance LSO. Nereus supplies stable bytes/range; Kafka
decides transaction visibility.
Broker takeover
The new leader obtains a higher KRaft epoch and Nereus append authority, hydrates an NKC1 checkpoint, replays its covered tail to the current head, and rebuilds producer/transaction/virtual-segment state. The old leader’s later head CAS is fenced. Any physical bytes it wrote without a commit do not advance LEO.
DeleteRecords to offset 60
- Kafka retention publishes an NKC1 checkpoint covering the pre-trim state.
- Nereus advances logical trim to
60. - Binding and local log start become
60. - Fetch below
60returns Kafka out-of-range/trim semantics. - Objects and ledgers remain until source retirement and physical GC.
If the Broker is killed after trim, a new Broker reloads head, trim, and checkpoint. Retrying target
60 does not publish a second checkpoint or modify the head again.