Build your first pipeline
This guide builds a node that reads a list of orders from an HTTP endpoint, stores each order in an inbox and writes it to a data lake on the file system. Everything runs in one process on your machine, with no other service.
You need a Maven project with the runtime, as described in Install Spool.
1. Serve some data
Section titled “1. Serve some data”The crawler needs a source to read. Save this file as orders.json.
[ { "order_id": "A-1001", "customer": "ACME", "amount": 120.50, "currency": "EUR" }, { "order_id": "A-1002", "customer": "Globex", "amount": 75.00, "currency": "EUR" }, { "order_id": "A-1003", "customer": "Initech", "amount": 310.20, "currency": "EUR" }]Serve it on port 8000 from the folder that contains it. Any HTTP server that returns a JSON array works, and this one comes with Python.
python -m http.server 80002. Describe the node
Section titled “2. Describe the node”A node is described in a YAML file that the runtime reads from the classpath. Create src/main/resources/Spool.yaml.
infrastructure: eventBus: type: IN_MEMORY
inbox: type: FILE_SYSTEM configuration: path: "spool-data/inbox"
dataLake: type: FILE_SYSTEM configuration: path: "spool-data/datalake"
modules: - crawler: type: POLL id: orders-crawler source: type: HTTP configuration: sourceId: orders-api url: "http://localhost:8000/orders.json" scheduleMilliseconds: 60000 mediaType: JSON_ARRAY eventMapping: namingConvention: SNAKE_CASE
- ingester: id: orders-ingester type: REACTIVE
- janitor: id: orders-janitor configuration: milliseconds: 5000 millisecondsThreshold: 10000 millisecondsTTL: 60000The infrastructure block declares what the modules share. The event bus lives in memory, which is enough when every module runs in the same process. The inbox and the data lake are folders, relative to the directory where you start the program.
The three modules do the work.
- The crawler asks the URL for data once a minute (
scheduleMilliseconds). Because the response is a JSON array, it splits it into one record per element.sourceIdnames the source, and that name ends up in the data lake path. - The ingester writes each record to the data lake as soon as the crawler has stored it.
- The janitor checks the inbox every 5 seconds. It records which envelopes are already in the data lake, publishes again the ones that have waited longer than
millisecondsThreshold, and deletes from the inbox the ones that have been in the data lake for longer thanmillisecondsTTL.
The DSL reference lists every key.
3. Start the node
Section titled “3. Start the node”Create the entry point of the program.
package com.example;
import software.spool.runtime.OpenTelemetryConfiguration;import software.spool.runtime.SpoolRuntime;
public class Main { public static void main(String[] args) { SpoolRuntime.builder() .OpenTelemetryConfiguration(OpenTelemetryConfiguration.builder() .serviceName("first-pipeline") .build()) .withNodeFromDSL("/Spool.yaml") .build() .start(); }}The OpenTelemetry configuration is required. With only a service name, the node tries to send its telemetry to a collector on the default local address and logs Failed to export when there is none. Those messages do not affect the pipeline, and Operations explains how to point it at a real collector.
Run Main from your IDE or with Maven. The node keeps running until you stop it.
4. Check the result
Section titled “4. Check the result”After a few seconds, the working directory contains this.
spool-data/├── inbox/│ └── PERSISTED/│ ├── ca769770….json│ ├── cb5ca79e….json│ └── d0beb143….json└── datalake/ └── bronze/year=2026/month=11/day=21/hour=20/source=orders-api/ ├── ca769770….json ├── cb5ca79e….json └── d0beb143….jsonEach file is an envelope, which holds one order together with the information Spool needs to trace it. The file name is its idempotency key, a SHA-256 of the source and the content, so the same order read twice is stored once.
{ "idempotencyKey": { "value": "ca76977059863551b1096539f28eb289f5e59899cfac8ba4843ea1155ac55771" }, "metadata": { "metadata": { "SOURCE": "orders-api", "CORRELATION_ID": "97d873eb-82d9-465f-a2c1-47faee9620fe", "PARTITION_SCHEMA": "{\"sourceId\":\"orders-api\",\"eventType\":\"java.lang.Void\",\"attributes\":[]}" } }, "mediaType": { "value": "JSON_ARRAY" }, "payload": "eyJvcmRlcl9pZCI6IkEtMTAwMyIsImN1c3RvbWVyIjoiSW5pdGVjaCIsImFtb3VudCI6MzEwLjIsImN1cnJlbmN5IjoiRVVSIn0=", "status": "CAPTURED", "retries": 0, "capturedAt": 1795292136.6882117, "updatedAt": null}The payload is the original order encoded in Base64.
{"order_id":"A-1003","customer":"Initech","amount":310.2,"currency":"EUR"}The inbox keeps one folder per state. An envelope starts in CAPTURED and moves to PERSISTED on the next run of the janitor after the ingester has written it. The data lake partitions records by the hour they were captured and by source. Envelope lifecycle explains each state.
The node also answers on port 8080 with the health of its modules.
curl http://localhost:8080/spool/healthNext steps
Section titled “Next steps”- How Spool works explains the modules and the events between them.
- SEC EDGAR filings shows a pipeline over real public data, with a source written for it.