Data Processor

The Data Processor step runs a data processor job as part of a
workflow — custom Python, Node.js, or Spark code, against files in the
Intely File System.

Typical use: retrieve a large file from a vendor, run a Spark job to aggregate it, then load
the aggregated result onward.

Configuration

1. Select the job to run.
2. Bind each of the job's input files to a file or directory — either a fixed IFS path or
   a file produced by an earlier step.
3. Supply values for the job's arguments.
4. Choose the resource tier for the run: Small, Medium, Large or Extra Large.

Output

The job's output files are available to later steps, and are also written into IFS at
system/temp/job-execution-<executionId>/. The execution log is written alongside them.

Notes

• Data processor runs are the slowest step type in most workflows. Where the job doesn't need
  to block the rest of the workflow, consider whether it belongs in a separate scheduled
  integration instead.
• Match the tier to the work. Running a small transformation on an Extra Large tier consumes
  resource without going faster.