.. Pipeline Export documentation

Export steps
============

.. module:: xmm.pipeline.steps

Steps required to export data to various destinations.

ExportStep
~~~~~~~~~~

.. autoclass:: ExportStep
   :members: __init__

In an export pipeline, the first step will usually be the ExportStep itself.
This step gathers data from the database to be converted and formatted further
at a later point in the pipeline.

You can either supply a ``query`` argument containing a MongoEngine query
or a ``search`` argument with options used to construct an ElasticSearch query.

**Arguments:**

- `str` `model`: A model name or identifier, as used in the generic RESTful API endoints.
- `list` `query`: A list of actions to perform on the model class directly as queries. Example::

        "query": [
            ["objects", {
                "name__startswith": "A",
            }],
            ["order_by", ["last_login"]]
        ]

- dict `search`: An ElasticSearch query object

**Added context:**

- str `mode`: Will be set to ``'export'``
- dict `export`
   - type `model`: Will be set to the model class
   - str `model_name`: Will be set to the model name (str)

ConvertStep
~~~~~~~~~~~

.. autoclass:: ConvertStep
   :members: __init__

The converter step is responsible for transforming the acquired data
to a more appropriate format, only containing fields we actually requested.

**Arguments:**

- list `attributes`: A list of attributes to extract. Can be a dotted-path to fetch
   values from a related model field. E.g. ``['foo', 'user.name']``

**Added context:**

- dict `convert`
   - list `attributes`: Will be a copy of the supplied `arguments` list.

FormatStep
~~~~~~~~~~

.. autoclass:: FormatStep
   :members: __init__

Formatting options can be supplied to this step, so data will look just like you requested.

**Arguments:**

- target, depth, collapse, value_map, \**options
- str `target`: (optional) A string identifying a known text format to export to, possible values are:
   - ``'text'`` This will coerce all data with ``str()`` before writing
   - ``'json'`` The JSON format will leave some types alone, so they can be exported
     with their appropriate type still intact in a JSON file.
- int `depth`: (optional) Define how deep nested attributes will be parsed,
   0 means no nesting, -1 will be unlimited.
- bool `collapse`: If the collapse flag is set and a relationship in an object
   resolves to None, the complete dict will be set to None instead
   of their individual sub-properties (has no effect when depth=0).
- dict `value_map`: (optional) A dict or list of pairs containing replacement values for
   specific other values found in attributes. Use this to show 'YES' for values like True. Example::

      "value_map": [
          [null, "[NULL]"],
          [true, "[TRUE]"],
          [false, "[FALSE]"]
      ],

- \**options: Further options can be supplied, they will be passed down to the individual datatype
   formatters. Currently supported are for example:

   - `dateformat`
   - `datetimeformat`
   - `timeformat`

**Added context:**

- dict `format`
   - int `depth`: Depth of data structures
   - bool `flat`: Will be true if depth is 0

FileWriteStep
~~~~~~~~~~~~~

.. class:: FileWriteStep

.. autoclass:: FileWriteStep
   :members: __init__

This step will write the current state to a file using a writer of your choice.

.. seealso:: :ref:`writers`

**Arguments:**

- str `load_class`: The name of a writer class to use. Can also be
   a fully qualified import path.
- dict `options`: Options to be passed down to the writer class
   - str `filename`: Filename to save to (relative path)

**Added context:**

- dict `write`:
   - Writer `writer`: The writer class that was used
   - str `filename`: The absolute path of the file that was written to
   - str `mime_type`: MIME type of the written file

RenameStep
~~~~~~~~~~

.. autoclass:: RenameStep
   :members: __init__

**Added context:**

This step adds no context.

ZipStep
~~~~~~~

.. autoclass:: ZipStep
   :members: __init__

This step will zip the file found at the current state.

**Arguments:**

This step takes no arguments.

**Added context:**

- dict `zip`:
   - str `filename`: The new filename for the zipped file.

XSLTExportStep
~~~~~~~~~~~~~~

.. autoclass:: XSLTExportStep
   :members: __init__

This step gets a XSLT template as string and uses it for XML transformation.

**Arguments:**

- str `xslt`: XSLT transformation template.

**Added context:**

This step adds no context.

FilterStep
~~~~~~~~~~

.. autoclass:: FilterStep
   :members: __init__

This step filters documents based on a filter function given to it.

**Arguments:**

- func `filter_func`: Gets the function that is used to filter documents.

**Added context:**

This step adds no context.