.. Pipeline configuration documentation

.. module:: xmm.pipeline

XMM Pipelines
=============

.. autoclass:: Pipeline

The XMM project contains an easily configurable
:class:`Pipeline` runner. It can do almost any task you can
think of, given the steps necessary are implemented
and compatible.

Pipeline format
---------------

Pipelines can be configured using JSON::

   [
       ["FirstStep", {
           "options": "go here",
       }],
       ["SecondStep", {
           "can": [
               "have",
               "different",
               "options"
           ]
       }],
       ["NoOptionStep", {}]
  ]

Steps
-----

.. module:: xmm.pipeline.steps

Every pipeline consists of one or more steps. A step is defined
as follows:

1. A function that accepts a ``state`` and a ``context`` variable.
   This function must return a tuple with the modified ``state`` and ``context``.
2. A class that may have its ``__init__`` options configured by the JSON pipeline
   configuration. Such Step classes should inherit from the base :class:`PipelineStep`.

The ``state`` variable should always be in a form of a Python :term:`iterable`.
This way each step can walk over datasets in the state lazily without using up too many system resources.

At the start of every pipeline, the `context` will be populated with a
:class:`~datetime.datetime` instance with the current date and time in the ``start`` key.

There are currently a number of steps available. See
the table below for requirements and configuration options.

.. toctree::

   import
   export
   generic
   readers
   loaders
   writers
   transfers

.. autosummary::
   PipelineStep
   ReadStep
   ImportStep
   ValueMapStep
   DbWriteStep

   ExportStep
   ConvertStep
   FlattenStep
   FormatStep
   FileWriteStep
   RenameStep
   XSLTExportStep
   XSLTImportStep
   ZipStep

   TransferStep
   NotifyStep
   ProgressStep