Modules
Each module does one job and talks to the others only through events. A node can run any combination of them.
| Module | What it does | Descriptor | Java builder |
|---|---|---|---|
| crawler | Reads a source and stores each record in the inbox. | Yes | CrawlerBuilderFactory |
| ingester | Validates records and writes them to the data lake. | Yes | IngesterBuilderFactory |
| janitor | Keeps the inbox up to date, retries stuck records and deletes old ones. | Yes | JanitorBuilderFactory |
| validator | Finds your validation rules and runs them for the ingester. | Not needed | Not needed |
| mounter | Aggregates data lake partitions into a data mart. | No | MounterBuilderFactory |
| watchdog | Receives heartbeats and reports the health of every module. | No | Runs as its own service |
A module declared in the descriptor gets its infrastructure from the infrastructure block. A module built in Java receives its ports directly, and is registered in a SpoolNode with register before the node starts.