Skip to content
Processing Framework

Referencedocs

tdsplitengine

A framework that simplifies the development of an engine container component of a split engine.

Business logic should be implemented in a separate module, where each Operation has a different method. These are then ‘registered’ by using the @dispatcher.operation decorator. This will map the method name to the operation name

./my_module.py

import tdsplitengine
my_dispatcher = tdsplitengine.Dispatcher()


@my_dispatcher.operation
def my_operation(task):
    # Perform business logic here
    return

The operation method can optionally return output parameters, rescheduling information, and/or a success details string. The easiest way to do this is to use the TaskResult object.

@my_dispatcher.operation
def my_operation(task):
    # Perform business logic here
    return tdsplitengine.TaskResult(
        output_parameters={"status_code": 0},
        reschedule=None,
        success_details='Operation succeeded'
    )

class tdsplitengine.Dispatcher

This class contains boilerplate code that simplifies running the operation specified by the task json generated by an orchestrator container.

operation(_func=None, *, operation_name=None)

Decorator registers the decorated function as implementing an operation.

  • Parameters:
    • _func (Operation_Signature) – The Function that implements the operation.
    • operation_name (str | None) – The name that will be used to specify the operation in Task Json. If None then the function name is used as the operation name.
  • Returns: The function being decorated. This decorator does not wrap the function.
  • Return type: Operation_Signature

register_operation(implementing_func, operation_name=None)

Register an operation in the dispatch table.

As an alternative to decorating an operation function with the operation decorator it can be explicitly registered by calling this function.

  • Parameters:
    • implementing_func (Operation_Signature) – The function that implements the operation.
    • operation_name (str) – The name of the operation.
  • Raises: RegistrationError – If the operation name is already in use.
  • Returns: None
  • Return type: None

dispatch_operation(operation_name, task)

Call the function registered to handle the operation named operation_name.

  • Parameters:
    • operation_name (str) – The name of the operation to execute.
    • task (Dict) – The task json that specifies the operation’s parameters.
  • Raises: TaskError – If the operation_name is not registered.
  • Returns: The value returned by the registered function that handles the operation.
  • Return type: Operation_Result

execute(task=None)

Execute the operation specified in a “task JSON” dictionary.

Normally the Task JSON is read from the file specified by the PF_TASK_JSON_PATH environment variable. It is expected that PF_TASK_JSON_PATH will be set by the container orchestrator when launching the container. If the task parameter is non None it will be used instead of reading task JSON from the file indicated by the PF_TASK_JSON_PATH environment variable.

  • Parameters: task (Dict | None) – An optional dictionary that when specified overrides task JSON file specified by the PF_TASK_JSON_PATH environment variable.
  • Returns: None
  • Return type: None

exception tdsplitengine.RegistrationError(operation_name, msg=None)

Indicates a problem registering an operation.

Most engines will not attempt to process an operation if this exception is encountered.

  • Variables:
    • operation_name (str) – The name of the operation that could not be registered.
    • msg (str) – An optional message to include in the exception.
  • Parameters:
    • operation_name (str)
    • msg (str | None)

class tdsplitengine.Reschedule(wait_time=0, properties=None)

An operation method can return a tuple with an instance of this data class as the second element to request that the decider reschedule this operation and try it again.

  • Parameters:
    • wait_time (int) – The number of seconds to wait before retrying the operation, defaults to 0
    • properties (dict) – An optional dictionary of properties that can be used to provide additional information

exception tdsplitengine.TaskError(reason, details=”)

This exception indicates that an engine operation has failed. Its reason and details properties specify values for the like named SWF taskFailure result.

Instantiate a new TaskError instance

  • Parameters:
    • reason (str) – The reason that caused the exception to be triggered. Truncated to 256 characters.
    • details (str) – Optional string with more details about the exception. Truncated to 32768 characters. Note: This will be truncated to 1024 characters within the Operation Status Event.

class tdsplitengine.TaskResult(output_parameters=None, reschedule=None, details=None)

Complete TaskResult object

An operation can return a TaskResult object to set output parameters, request rescheduling and provide a success details message.

  • Parameters:
    • output_parameters (dict) – A dictionary of output parameters to set in the task result JSON.
    • reschedule (Reschedule) – An Reschedule object to request a rescheduling of the operation.
    • details (str) – An string with additional information about a success sent to the Event Service.

tdsplitengine.dispatcher.Operation_Result

A method implementing an operation can return: None indicating it has neither set output parameters, requested a reschedule, nor has details Returning a Dict indicates a desire to set output parameters. Returning a tuple sets both output parameters and requests a rescheduling of the operation. Returning a TaskResult object is used to optionally set output parameters, request rescheduling and/or provide a success details message.

alias of None | Dict | Tuple[Dict | None, Reschedule | None] | TaskResult

tdsplitengine.dispatcher.Operation_Signature

Operation_Signature defines the expected signature of a function that implements an operation

alias of Callable[[Dict], None | Dict | Tuple[Dict | None, Reschedule | None] | TaskResult]

class tdsplitengine.settings.ConfigOption(section, option_name, fallback, required=False)

Bases: object

Define an expected configuration file setting. option_name: The name of the option. fallback: Value to assign if the option isn’t defined in the configuration file. required: Raise an exception if the option is not defined in the specified section. If fallback is none the option will never be undefined.

  • Parameters:
    • section (str) – The section of the configuration file where the option is defined.
    • option_name (str) – The name of the option.
    • fallback (str) – The value to assign if the option isn’t defined in the configuration file.
    • required (bool) – Raise an exception if the option is not defined in the specified section, defaults to False

class tdsplitengine.settings.Settings

Bases: object

Dynamic properties are added to this class to store the values passed to the initialize_settings method.

After this class is instantiated, the properties defined in the manifest can be accessed as attributes of the class.

>>> settings.option_name

classmethod read(conf_path, manifest)

Construct a new Settings object by reading the settings specified in options from the file specified in conf_file.

The expected format of the configuration file is as follows:

[section]
foo=bar
baz=bat
  • Parameters:
    • conf_path (Path) – Configuration file to read settings from.
    • manifest (Iterable [ConfigOption ]) – An iterable of ConfigOption objects the define the options to read from the configuration file.
  • Returns: An instantiated Settings object with the properties defined by the variables property.
  • Return type: Settings

classmethod read_file(file, manifest)

Construct a new Settings object by reading the settings specified in.

  • Parameters:
    • file (TextIO) – Stream of the configuration file to read settings from.
    • manifest (Iterable [ConfigOption ]) – An iterable of ConfigOption objects the define the options.
  • Raises: KeyError – If a required option is not defined in the configuration file.
  • Returns: An instantiated Settings object with the properties defined.
  • Return type: Settings