Native Kafka integration
Native Kafka is a Kafka Broker fork path, not a Kafka request projected through Pulsar. The Kafka
client reaches the native Broker, ReplicaManager/Partition/UnifiedLog seams, the Nereus Kafka
adapter, and then StreamStorage. Kafka retains KRaft, leader, producer, transaction, consumer-group,
and internal-topic semantics; Nereus replaces the partition log’s durable data plane.
Authority split
| Authority | Owns |
|---|---|
| KRaft | Topic/partition metadata, leader and leader epoch, broker membership, ISR/reassignment, feature level, config history |
| Nereus/Oxia | topicId + partition -> streamId binding, append session, stream head, commit chain, generation index, profile, trim, physical references |
| Kafka internal topics | Consumer-group and transaction coordinator state in __consumer_offsets and __transaction_state |
The internal-topic bytes may use Nereus storage, but Oxia does not become a parallel coordinator truth.
Partition binding and activation
The durable binding key is Kafka cluster identity plus topicId + partition. It records stream ID,
profile, lifecycle/incarnation, observed offsets, checkpoint references, and activation facts. An
observed offset accelerates startup only; the current Nereus head remains authoritative.
Cluster activation is a capability barrier:
- Validate Kafka and Nereus configuration.
- Establish Oxia, Object Store, and optional BookKeeper runtimes.
- Exchange exact profile/capability digests among Brokers.
- Let the KRaft controller aggregate the active Broker set and metadata snapshot.
- Create or recover the Oxia activation record.
- Wait for
ACTIVE, readiness, and source-proof agreement. - Install the Nereus-backed log seams before allowing leader traffic.
Activation proves that the cluster understands the same durable contract; it is not a stream append commit.
Leader open
When KRaft assigns a partition leader, the adapter resolves the binding, obtains an append session with the KRaft leader authority, reads the stable stream head, validates the recovery checkpoint, replays the committed tail, and rebuilds Kafka producer/transaction/epoch/index state. Only then is the Nereus-backed log published as writable.
After leadership loss, the old Broker stops new admission. Provider completions are revalidated against the KRaft authority and Nereus head CAS, so a stale leader cannot publish a logical append.
LEO, HW, and LSO
These offsets remain distinct:
- LEO is the next record offset and is backed by the stable Nereus append end.
- HW is Kafka’s consumer-visible replicated boundary; the initial Nereus model uses RF=1, with Kafka state advancing after a stable append and derived-state update.
- LSO is the
READ_COMMITTEDtransaction boundary and may trail HW while a transaction is open.
Storage returns committed bytes and coverage. The Kafka adapter still applies HW, LSO, and aborted transaction ranges before building a Fetch response.
Initial compatibility boundary
The current Native Kafka model is KRaft-only with RF=1 as the primary Nereus data-replica model. Profile choice is immutable after stream creation; implicit stock-log/Nereus mixing and online local log migration are not supported. Stock Kafka remote log/tiered storage is disabled for Nereus partitions. Delete, compact, and compact+delete use the Nereus retention/materialization paths, while consumer groups and transactions continue to use native internal topics.