Generic steps

PipelineStep

class PipelineStep

A single step in a pipeline.

DebugStep

class DebugStep(state=False, context=False, err=False)

Debug step.

Print parts of or the entire current state and or context.

NotifyStep

class NotifyStep(template=None, template_error=None)

Send a notification.

Parameters

• template (str) – A format string that receives the context for building the success notification message.

• template_error (str) – A format string that receives the context for building the error notification message.

__init__(template=None, template_error=None)

Initialize self. See help(type(self)) for accurate signature.

Sends a notification about the successful import or export.

Arguments:

This step takes no arguments.

Added context:

• bool notify: Will set notify to True.

ProgressStep

class ProgressStep(step_size=5)

Progress step.

Shows a progressbar when stderr is a terminal, or the progress in percent when stderr is a file.

Parameters

step_size (int) – Step size in percent (defaults to 5).

__init__(step_size=5)

Initialize self. See help(type(self)) for accurate signature.

Shows a progressbar when stderr is a terminal, or the progress in percent when stderr is a file.

Arguments:

• int step: Step size in percent (defaults to 5).

Added context:

This step adds no context.

FlattenStep

class FlattenStep(depth=-1)

Convert nested dicts to flat dicts.

Parameters

depth (int) – Number of nested levels to flatten. Set to -1 for infinite depth.

Examples:

Input:

{
  'fields': {
    'name': {
      'de': 'Deutsch',
      'en': 'English',
    },
  },
}

Output:

{
  'fields.name.de': 'Deutsch',
  'fields.name.en': 'English',
}

ExpandStep

class ExpandStep(depth=-1)

Convert flattened dicts into nested dicts.

Parameters

depth (int) – Number of levels to expand. Set to -1 for infinite depth.

Examples:

Input:

{
  'fields.name.de': 'Deutsch',
  'fields.name.en': 'English',
}

Output:

{
  'fields': {
    'name': {
      'de': 'Deutsch',
      'en': 'English',
    },
  },
}

RebuildIndexStep

class RebuildIndexStep(model, raw=False)

RebuildIndex step.

Rebuilds the Elasticsearch index for a given model.

Parameters

• model (str|type) – Model class name or concrete class.

• raw (bool) – Use raw method to index data. Please use with caution, model links are not indexed!

SplitStep

class SplitStep(pipelines=[])

SwitchStep

class SwitchStep(func, pipelines)