Skip to main content
adb’s architecture follows from a single decision: separate storage from compute. Data lives in object storage as open, columnar files; query engines are lightweight and stateless. Because the engines hold no permanent state, they can scale up under load, scale down when idle, and move between machines without relocating or losing data.

​
Two write paths: streaming and batch

Data reaches storage along one of two paths, depending on how it arrives. Streaming path, for live events. Incoming events land in the query layer (the tier that also serves historical reads), where they’re materialized as in-memory Arrow data so they’re queryable immediately. Periodically, that data is flushed down into columnar files in object storage. This path is what makes new traces show up within seconds. Batch path, for large backfills. Big uploads bypass the query layer entirely and write Parquet/Arrow files directly to blob storage. An indexing service then generates the metadata needed so those files are queryable through the same interface as streamed data. Because backfills don’t pass through the query layer, they don’t block or slow down live ingestion.
Both paths converge on the same open file formats in object storage, so a query sees streamed events and backfilled history as one unified dataset.

​
Apache Flight for zero-copy access

adb uses Apache Flight to move data to query engines and clients. Because Arrow data has the same representation in memory and on the wire, Flight transfers it with no serialization step, enabling zero-copy downloads of just-updated Arrow files at scale.

​
Compaction

Ingest patterns vary enormously, from a trickle of tiny individual events to single attributes holding two-million-token payloads. A one-size compaction strategy can’t handle both, so adb compacts dynamically, merging and reorganizing segments differently depending on their size and shape to keep files efficient to scan.

​
Cache management

adb can hold petabytes in object storage while pulling only what a query needs onto the query servers. A cache-management layer controls the boundary between “hot” data (kept close to compute for fast access) and “cold” data (left in cheaper object storage). This is what lets adb offer petabyte-scale storage without paying to keep all of it on fast compute nodes.

Next: Real-time ingestion