Stream, record, entry, and offset
Why provider coordinates are not enough
Pulsar ledgers and Kafka segments are useful physical or protocol structures, but neither is a universal coordinate for a storage layer shared by both systems. A broker can roll a ledger, replace an object, or change a Kafka segment boundary without changing the logical stream range.
Nereus therefore names the logical stream explicitly and treats provider coordinates as physical targets behind that model.
Definitions
| Term | Meaning in Nereus |
|---|---|
stream | The append-only logical sequence owned by one storage profile and one lifecycle. |
record | The smallest logical unit that consumes one Nereus offset. |
entry | An indivisible physical/protocol payload submitted to a provider. |
batch | A group of entries submitted by one API operation. |
offset | A logical position in a stream, represented as a half-open range for reads and writes. |
The mapping from an entry to records is protocol-specific. Nereus does not force Pulsar and Kafka to pretend that their batches have the same shape.
Offset ranges are half-open
An append or read range uses:
[startOffset, endOffset)
The start is included and the end is excluded. This makes adjacent ranges composable and allows a committed head to be represented as the first offset after the committed data.
Pulsar mapping
For the Pulsar projection, one complete Entry is the normal unit mapped to one Nereus offset. The adapter preserves the stable MessageId/Position relationship while storing the logical range below it.
Batch-message details can require a projection of sub-message acknowledgement state; they do not change the stream's logical append order.
Kafka mapping
For Native Kafka, one RecordBatch is represented as a ranged entry. A single physical entry can therefore cover multiple Kafka offsets. The adapter validates Kafka batch boundaries and maps the range to streamId + offset without making the Kafka segment identity part of the core coordinate.
What this model is not
offsetis not a BookKeeper entry ID.offsetis not an object byte position.entryis not always one record.- A higher physical generation does not contain newer logical messages; it is another representation of an existing range.