Skip to content

Envelope lifecycle

Every record that enters Spool travels inside an envelope. The envelope holds the original data, unchanged, together with what Spool needs to trace it. That is an idempotency key, the source it came from, a correlation identifier, its state, the number of retries and the time it was captured.

CAPTUREDPERSISTEDQUARANTINEDdeletedpublished again after the thresholdEnvelopePersistedEnvelopeQuarantinedor three retriesafter the TTL
The janitor moves an envelope between states when it sees the ingester's events. It also publishes again the envelopes that wait too long and deletes the ones already in the data lake once their TTL has passed.
State Set by Meaning
CAPTURED crawler The record has been read and stored in the inbox, and is waiting to be ingested.
PERSISTED janitor The ingester has written the record to the data lake.
QUARANTINED janitor The record broke a validation rule, or it was still waiting after the last retry.

The janitor changes the state when it sees the EnvelopePersisted and EnvelopeQuarantined events, so there can be a few seconds between the ingester writing a record and the inbox showing it as PERSISTED.

The state model also includes INGESTED, but no module sets it in this version.

The idempotency key is a SHA-256 of the source identifier and the content of the record. The inbox stores each envelope under its key, so reading the same record twice does not create a second envelope.

An envelope that stays in CAPTURED longer than the janitor’s threshold is published again, and its retry counter goes up by one. When the counter reaches three, the janitor stops publishing it and sets it to QUARANTINED.

Descriptor key Meaning Default
milliseconds How often the janitor runs. 5000
millisecondsThreshold How long an envelope can stay in CAPTURED before it is published again. 3 minutes
millisecondsTTL How long a PERSISTED envelope stays in the inbox before it is deleted. 1 day

The number of retries can only be changed when the janitor is built in Java, with withMaxRetries.

Once a record is in the data lake, the inbox no longer needs it. The janitor deletes PERSISTED envelopes that have not changed for longer than millisecondsTTL. Envelopes in QUARANTINED are kept.

When the ingester finds a record that breaks a validation rule, it does not write it to the data lake. It sends the original data and the list of violations to a quarantine store, and publishes EnvelopeQuarantined.

The quarantine store is a port of the ingester, QuarantineStore, and has to be provided when the ingester is built in Java. A node built from a descriptor has no quarantine store in this version.