Skip to content

crawler

The crawler reads an external source, splits what it receives into records and stores each record in the inbox as an envelope in the CAPTURED state. For each record it publishes SourcePayloadCaptured and EnvelopeStored.

The crawler artifact, version 1.2.1, is included in the runtime.

A crawler has one of two types.

  • POLL asks the source for data on a schedule. With scheduleMilliseconds greater than zero it repeats at that interval, and without it, or with 0, it reads once.
  • STREAM receives records as the source pushes them.

The source is a plugin chosen with source.type. Spool includes HTTP for polling a URL and IN_MEMORY for streams created in the same process. You can add your own, as the SEC EDGAR example does with SEC_DAILY_INDEX. See Ports and backends.

The crawler turns the response into records in three steps.

  1. rootPath points to the part of the response that holds the records, with dots between keys and numbers for list positions. For example, data.items or data.sumario.diario.0.items.
  2. mediaType says how to split that part into records. JSON_ARRAY makes one record per element and JSON treats the whole object as one record. EVENT_CLASS and EVENT_CLASS_ARRAY are for sources that deliver Java objects instead of bytes, such as the in-memory stream, and turn the object into JSON first. PDF keeps the document as one record.
  3. Each enrichment rule copies a value from anywhere in the response into every record. source is the path of the value and target the name it gets in the record. Without target, the name stays the same.
source:
type: HTTP
configuration:
sourceId: gazette
url: "https://example.com/summary"
rootPath: data.items
mediaType: JSON_ARRAY
enrichment:
- source: data.date
target: publish_date

eventMapping says how to read each record as a Java class.

  • namingConvention is how the fields are written in the source, as SNAKE_CASE, CAMEL_CASE, PASCAL_CASE or KEBAB_CASE. It is required.
  • domainMappingList lists the classes of your records. Each class must be in the events package and implement software.spool.core.model.Event. When a class is listed, the data lake partition includes eventType=<class>. With several classes, each record gets the first one that can read it, so keep one class per crawler unless the classes reject each other’s records.
  • attributeList names fields of the record that become partition keys in the data lake, as field=value. A record without one of those fields cannot be partitioned and fails with a PartitionKeyException.

By default, an error while reading the source is logged and the crawler tries again on its next run. errorRouter lets you choose another router plugin by its type, with its own configuration. Spool includes DEFAULT.

- crawler:
type: POLL
id: orders-crawler
source:
type: HTTP
configuration:
sourceId: orders-api
url: "http://localhost:8000/orders.json"
scheduleMilliseconds: 60000
mediaType: JSON_ARRAY
eventMapping:
namingConvention: SNAKE_CASE

The DSL reference lists every key.