Skip to content
Processing Framework

Operation Implementation

An Engine implements one or more Operations. Engines are deployed as Docker images. An Integrator is responsible for building and testing the Docker image before supplying it to the PF team for deployment.

Executing an Operation

Once the Processing Framework determines which container image should execute an Operation, it sets up an execution environment for the Operation, and then creates a container from the image to process the Operation. By design the Engine container does not need to be aware of cloud providers and will not receive credentials necessary to access the cloud provider services.

The container is expected to execute a single Operation and exit. A new container is created for each Operation processed.

The Processing Framework communicates with the Engine container through a shared Docker volume mounted on the container. The Processing Framework injects the environment variable PF_TASK_JSON_PATH into the Engine container at launch. The value of PF_TASK_JSON_PATH is the absolute path of a request JSON file that specifies the details of the Operation to process. This request JSON file conforms to the task request schema.

The results_json key in the request JSON file specifies an absolute path for the results JSON file that the Engine must write to communicate the success of the Operation. If the Engine container exits without writing the results JSON file the Operation is assumed to have failed. The results JSON file must conform to the task result schema or the Operation is assumed to have failed.

Task Request JSON Contents

The schema of the task request JSON as well as the layout of the execution environment in the shared volume is governed by the Operation definition JSON used to define the Operation to the API.

For example when a procedure includes the following compute_checksum Operation:

{
  "Operation": {
    "identifier": "compute_checksum",
    "name": "Compute Checksum",
    "description": "Compute a checksum for each file file in the input port",
    "Engine_name": "integrity_checker",
    "version": 1,
    "parameters": {
      "algorithm": {
        "type": "single_choice",
        "optional": false,
        "description": "indicate which checksum algorithm to use",
        "options": [
          "SHA256",
          "MD5"
        ]
      }
    },
    "inputs": {
      "sources": {
        "name": "sources",
        "optional": false,
        "data_types": [
          "*"
        ],
        "description": "Files to hash"
      }
    },
    "outputs": {
      "hashes": {
        "data_type": "data",
        "description": "A text file with one line for each file in the input port"
      }
    },
    "output_parameters": {
      "hashhash": {
        "type": "string",
        "description": "Hash of the hash file in output port. Used to verify the hash file hasn't been modified",
        "optional": false
      },
      "count": {
        "type": "string",
        "description": "The number of files hashed"
      }
    }
  }
}

Execution of the compute_checksum Operation defined above results in the sending the following task request JSON to a Linux Engine.

{
  "results_json": "/taskscratch/result_33c1a9e6-06d6-11ec-8e65-0242ac110002.json",
  "secondary_logs": "/taskscratch/secondary_logs33c1a9e6-06d6-11ec-8e65-0242ac110002",
  "tertiary_logs": "/taskscratch/tertiary_logs33c1a9e6-06d6-11ec-8e65-0242ac110002",
  "Operation":
  {
    "id": "f07b13e0-980d-4388-9c0b-c2d81d499578",
    "identifier": "compute_checksum",
    "version": 1
  },
  "parameters":
  {
    "algorithm": "sha256"
  },
  "inputs":
  {
    "sources":
    [
      {
        "base-directory": "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data",
        "data-directory": "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/rare/throat",
        "type": "file",
        "hint": null,
        "data-size": 3423,
        "file-extensions":
        [
          "txt",
          "env",
          "sql",
          "lua"
        ],
        "local-paths":
        [
          "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/rare/throat/Operation run duration.sql",
          "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/rare/throat/recover_lf.lua",
          "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/rare/throat/level2/make_dev.sql",
          "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/rare/throat/level2/make_utest.sql",
          "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/rare/throat/level2/level3/dev shuffle.txt",
          "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/rare/throat/level2/level3/run_on_localhost.env"
        ]
      },
      {
        "base-directory": "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data",
        "data-directory": "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/nicosia/campfire",
        "type": "file",
        "hint": null,
        "data-size": 1556,
        "file-extensions":
        [
          "json"
        ],
        "local-paths":
        [
          "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/nicosia/campfire/sample task input.json",
          "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/nicosia/campfire/v4_app_wkt.json"
        ]
      }
    ]
  },
  "outputs":
  {
    "hashes": "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/hashes"
  },
  "tds":
  {
    "execution-id": "33c1a9e6-06d6-11ec-8e65-0242ac110002",
    "Operation-id": "jostle",
    "tid_owner_id": "test_fixture",
    "tid-owner-uuid": "00000000-0000-0000-0000-000000000000"
  },
  "reschedule":
  {
    "wait-time": null,
    "properties":
    {}
  }
}

The Request JSON’s results_json key specifies the absolute path of the JSON file the Operation must write to specify the Operation’s execution status and output parameters.

The Request JSON’s secondary_logs and tertiary_logs keys specify the absolute paths of directories to write log files that will be made available for export using the API’s Request log contents resource. Secondary logs are also streamed to Sumologic at the end of an Operation where tertiary logs are not. Tertiary logs are intended for log files that either already are streamed in some other way to avoid duplication in Sumologic or for logs which an integrator simply chooses not to stream. Further explanation can be found on the Engine logging page.

The Request JSON’s Operation key specifies which of the Operations supported by the Engine to execute. The identifier and version keys correspond to the keys of the same name in the Operation definition. The value of the id key is generated by the API when the Operation is defined.

The Operation definition specifies a single input parameter named algorithm.

    "parameters": {
      "algorithm": {
        "type": "single_choice",
        "optional": false,
        "description": "indicate which checksum algorithm to use",
        "options": [
          "SHA256",
          "MD5"
        ]
      }
    },

This results in a corresponding parameters section in the request JSON file. In this case the client application specified a value of “true” for the parameter when the execution was created.

  "parameters":
  {
    "algorithm": "sha256"
  },

Since algorithm is not defined as optional a value for it will always be included in the request JSON. If the parameter was optional it would be omitted from the request JSON if a value wasn’t supplied when the execution was created.

The Operation definition specifies a single input port named sources.

"inputs": {
    "sources": {
    "name": "sources",
    "optional": false,
    "data_types": [
        "*"
    ],
    "description": "Files to hash"
    }
},

Which results in the corresponding inputs section in the request JSON file.

"inputs":
{
"sources":
[
    {
    "base-directory": "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data",
    "data-directory": "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/rare/throat",
    "type": "file",
    "hint": null,
    "data-size": 3423,
    "file-extensions":
    [
        "txt",
        "env",
        "sql",
        "lua"
    ],
    "local-paths":
    [
        "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/rare/throat/Operation run duration.sql",
        "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/rare/throat/recover_lf.lua",
        "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/rare/throat/level2/make_dev.sql",
        "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/rare/throat/level2/make_utest.sql",
        "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/rare/throat/level2/level3/dev shuffle.txt",
        "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/rare/throat/level2/level3/run_on_localhost.env"
    ]
    },
    {
    "base-directory": "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data",
    "data-directory": "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/nicosia/campfire",
    "type": "file",
    "hint": null,
    "data-size": 1556,
    "file-extensions":
    [
        "json"
    ],
    "local-paths":
    [
        "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/nicosia/campfire/sample task input.json",
        "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/nicosia/campfire/v4_app_wkt.json"
    ]
    }
]
},

The Operation definition defined an input port named sources so the request JSON inputs object has a corresponding sources key. A procedure can feed the output ports of multiple Operations into a single input port. This example execution chained two output ports into the sources input port so the value of the sources key is a list with two entries.

The value of the local-paths key is a list of absolute file paths with an entry for each file in the port. These absolute paths will be routed in the shared volume mounted on the container.

An input port can contain a complex nested directory structure. The value of the data-directory key is absolute path of input port root. Subtracting data-directory from a local-paths value to determine where the input fits in the input port’s directory structure.

The Operation definition specifies a single output port named hashes.

  "outputs":
  {
    "hashes": "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/hashes"
  },

Which results in the corresponding outputs section in the request JSON file.

"outputs":
{
  "hashes": "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/hashes"
},

The Operation definition defined an output port named hashes so the request JSON outputs object has a corresponding hashes key. The value of the hashes key is the absolute path of a directory where the contents of the Operation’s hashes port should be written. The Processing Framework will save the contents of the directory and make it available to the subsequent Operations.

The request JSON’s tds key provides details of the procedure execution run containing the Operation execution. The execution-id key is the identifier assigned by the API to the execution when the execution was created. Including it in log messages simplifies associating the log messages with an execution. The Operation-id key is the key assigned to the Operation in the procedure definition. The tid_owner_id and tid_owner-uuid keys specify the application that created the execution.