Data Processor

Data Processor runs custom code inside Intely's infrastructure. When a transformation is
too complex for a mapping — heavy computation, a specialized library, a large-scale
aggregation — write it as a data processor job.

Find it under Connect → Data Processor.

Job types

Each type corresponds to a prepared execution environment. If you need a library that isn't
available in it, raise it with your Intely contact rather than assuming it is present.

How a job is put together

A job points at code and files that already live in the Intely File System.

1. Source directory — an IFS directory containing your code. Its contents are copied into
   the job's source/ directory at run time.
2. Main file — the file within that directory to execute. For Java and Scala Spark jobs,
   also supply a main class.
3. Input files — named slots your code reads from. Each input is bound at run time to
   either a single IFS file or an IFS directory (all files directly inside it, flat —
   subdirectories are not walked). Everything bound is copied into the job's input/
   directory. You can give an input a reference file name so your code can hard-code the
   name it expects.
4. Output files — named slots your code writes to. Each is defined by a match
   expression (a regular expression) applied to the job's output/ directory. Anything
   matching is collected and written back into IFS.
5. Arguments — command-line arguments passed to your program. Each is either a value
   (--name value) or a switch (--flag), in long (--) or short (-) form.

Mark inputs and outputs required where the job can't succeed without them.

The execution directory

At run time your job gets an isolated working directory:

source/    your code, copied from the source directory
input/     the files bound to your input slots
output/    write your results here

Write everything you want kept to output/. Files matching an output definition are copied
into IFS under system/temp/job-execution-<executionId>/; anything else is discarded when
the container exits.

Running a job

Choose a resource tier for the run:

Provide values for any inputs and arguments, then run.

Execution statuses

Pending → Waiting → Executing → Success or Failed.

A run can also end as User Terminated (you stopped it), System Terminated (Intely
stopped it) or Not Executed. Terminating is the transitional state after a stop is
requested.

Logs and outputs

Each execution keeps its full log, written into IFS alongside the outputs at
system/temp/job-execution-<executionId>/log-<executionId>.log. Open an execution to see
its log, its arguments, and links to every output file it produced.

Edits and run history

Each execution records the job as it was configured at the time, so a run's history stays
meaningful after the job is edited.

Using a job inside an integration

The Integration Builder has a Data Processor step, so a job can run as part of a larger
workflow — for example, pull a file from a vendor, process it with Spark, then post the
results onward. See Data Processor (step).