crawler
The crawler reads an external source, splits what it receives into records and stores each record in the inbox as an envelope in the CAPTURED state. For each record it publishes SourcePayloadCaptured and EnvelopeStored.
The crawler artifact, version 1.2.1, is included in the runtime.
Poll and stream
Section titled “Poll and stream”A crawler has one of two types.
POLLasks the source for data on a schedule. WithscheduleMillisecondsgreater than zero it repeats at that interval, and without it, or with0, it reads once.STREAMreceives records as the source pushes them.
The source is a plugin chosen with source.type. Spool includes HTTP for polling a URL and IN_MEMORY for streams created in the same process. You can add your own, as the SEC EDGAR example does with SEC_DAILY_INDEX. See Ports and backends.
From response to records
Section titled “From response to records”The crawler turns the response into records in three steps.
rootPathpoints to the part of the response that holds the records, with dots between keys and numbers for list positions. For example,data.itemsordata.sumario.diario.0.items.mediaTypesays how to split that part into records.JSON_ARRAYmakes one record per element andJSONtreats the whole object as one record.EVENT_CLASSandEVENT_CLASS_ARRAYare for sources that deliver Java objects instead of bytes, such as the in-memory stream, and turn the object into JSON first.PDFkeeps the document as one record.- Each
enrichmentrule copies a value from anywhere in the response into every record.sourceis the path of the value andtargetthe name it gets in the record. Withouttarget, the name stays the same.
source: type: HTTP configuration: sourceId: gazette url: "https://example.com/summary" rootPath: data.items mediaType: JSON_ARRAY enrichment: - source: data.date target: publish_dateEvent mapping
Section titled “Event mapping”eventMapping says how to read each record as a Java class.
namingConventionis how the fields are written in the source, asSNAKE_CASE,CAMEL_CASE,PASCAL_CASEorKEBAB_CASE. It is required.domainMappingListlists the classes of your records. Each class must be in theeventspackage and implementsoftware.spool.core.model.Event. When a class is listed, the data lake partition includeseventType=<class>. With several classes, each record gets the first one that can read it, so keep one class per crawler unless the classes reject each other’s records.attributeListnames fields of the record that become partition keys in the data lake, asfield=value. A record without one of those fields cannot be partitioned and fails with aPartitionKeyException.
Errors
Section titled “Errors”By default, an error while reading the source is logged and the crawler tries again on its next run. errorRouter lets you choose another router plugin by its type, with its own configuration. Spool includes DEFAULT.
In the descriptor
Section titled “In the descriptor”- crawler: type: POLL id: orders-crawler source: type: HTTP configuration: sourceId: orders-api url: "http://localhost:8000/orders.json" scheduleMilliseconds: 60000 mediaType: JSON_ARRAY eventMapping: namingConvention: SNAKE_CASEThe DSL reference lists every key.