Skip to content

Deployment

The modules of a pipeline do not know where the others run. They only talk through the event bus and through the storage they share, so the same pipeline can run in one process or in several, and moving from one to the other means changing the infrastructure block of the descriptor, not the code.

One nodeinboxfolderdata lakefolderone Java processcrawleringesterjanitormounterIN_MEMORY event busSeveral nodesinboxS3data lakeshared foldernode Acrawlernode Bingesterjanitornode CmounterKAFKA event bus
The modules are the same in both cases. What changes is the infrastructure in the descriptor, with an in-memory bus and folders for one process, and Kafka and shared storage when the modules run in different nodes.

Every module runs in one node, with the IN_MEMORY event bus and folders for the inbox and the data lake. Nothing else has to be installed, which makes it the way to develop and test a pipeline. Build your first pipeline runs like this.

The same layout works in production when one machine is enough. The node runs as one service, and the inbox and the data lake can stay in folders on a persistent disk or move to S3. Because the bus is in memory, every module must stay in that process.

Each node runs the modules you choose, for example the crawlers in one node, the ingester and the janitor in another and the mounter in a third. The nodes need to share three things.

Piece What to use Why
Event bus KAFKA Events must reach modules in other nodes.
Inbox S3, or a folder every node can reach The crawler writes envelopes that the ingester and the janitor read.
Data lake A folder every node can reach The ingester writes it and the mounter reads it. In this version the mounter can only read a data lake on the file system.

Each node has its own descriptor, with the same infrastructure block and only its own modules.

Every node opens port 8080 for its health endpoint. Nodes built from a descriptor always use that port, so on the same machine each one needs its own container. A node built in Java can take another port with SpoolNode.create(port).

Wherever the nodes run, every module reports through OpenTelemetry, and the watchdog receives a heartbeat from each one. Operations explains how to see it all in Grafana.