You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/platform/understanding-airbyte/high-level-view.md
+37-1Lines changed: 37 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,7 +10,36 @@ The platform provides all the horizontal services required to configure and run
10
10
11
11
Connectors are independent modules which push/pull data to/from sources and destinations. Connectors are built in accordance with the [Airbyte Specification](./airbyte-protocol.md), which describes the interface with which data can be moved between a source and a destination using Airbyte. Connectors are packaged as Docker images, which allows total flexibility over the technologies used to implement them.
12
12
13
-
A more concrete diagram can be seen below:
13
+
## Data Transfer Modes
14
+
15
+
Airbyte supports two data transfer modes that are automatically selected based on connector capabilities:
16
+
17
+
-**Socket Mode**: Records flow directly from source to destination via Unix domain sockets, enabling high-throughput parallel data transfer. A lightweight bookkeeper process handles control messages, state, and logs.
18
+
-**Legacy Mode**: Records flow through an orchestrator middleware that sits between source and destination, using standard input/output streams.
19
+
20
+
Socket mode is used when both source and destination connectors support it, providing significantly higher performance for data movement operations.
A more concrete diagram of the platform orchestration can be seen below:
14
43
15
44
```mermaid
16
45
---
@@ -45,6 +74,13 @@ flowchart LR
45
74
-**Workload API**[`airbyte-workload-api-server`]: The HTTP interface for enqueuing workloads — the discrete pods that run the connector operations.
46
75
-**Launcher**[`airbyte-workload-launcher`]: Consumes events from the workload API and interfaces with k8s to launch workloads.
47
76
77
+
### Data Transfer Middleware
78
+
79
+
Within connector operation pods, Airbyte runs middleware containers to process connector output:
80
+
81
+
-**Bookkeeper**[`airbyte-bookkeeper`]: Used in socket mode. Processes control messages, state, and logs while records flow directly between connectors via sockets.
82
+
-**Container Orchestrator**[`airbyte-container-orchestrator`]: Used in legacy mode. Sits between source and destination connectors, processing all data and control messages.
83
+
48
84
The diagram shows the steady-state operation of Airbyte, there are components not described you'll see in your deployment:
49
85
50
86
-**Cron**[`airbyte-cron`]: Clean the server and sync logs (when using local logs). Regularly updates connector definitions and sweeps old workloads ensuring eventual consenus.
0 commit comments