Skip to content
Processing Framework

Task Request JSON

An operation’s definition in the processing API as well as the details of the execution are reflected in the contents of the task request JSON. The task request always conforms to the schema defined in task request schema.

Posting the following JSON to the processing API defines the compute_checksum operation.

{
  "operation": {
    "identifier": "compute_checksum",
    "name": "Compute Checksum",
    "description": "Compute a checksum for each file file in the input port",
    "engine_name": "integrity_checker",
    "version": 1,
    "parameters": {
      "algorithm": {
        "type": "single_choice",
        "optional": false,
        "description": "indicate which checksum algorithm to use",
        "options": [
          "SHA256",
          "MD5"
        ]
      }
    },
    "inputs": {
      "sources": {
        "name": "sources",
        "optional": false,
        "data_types": [
          "*"
        ],
        "description": "Files to hash"
      }
    },
    "outputs": {
      "hashes": {
        "data_type": "data",
        "description": "A text file with one line for each file in the input port"
      }
    },
    "output_parameters": {
      "hashhash": {
        "type": "string",
        "description": "Hash of the hash file in output port. Used to verify the hash file hasn't been modified",
        "optional": false
      },
      "count": {
        "type": "string",
        "description": "The number of files hashed"
      }
    }
  }
}

The following task request JSON resulted from executing a procedure that included the compute_checksum operation defined above.

{
  "results_json": "/taskscratch/result_33c1a9e6-06d6-11ec-8e65-0242ac110002.json",
  "secondary_logs": "/taskscratch/secondary_logs33c1a9e6-06d6-11ec-8e65-0242ac110002",
  "tertiary_logs":  "/taskscratch/tertiary_logs33c1a9e6-06d6-11ec-8e65-0242ac110002",
  "operation":
  {
    "id": "f07b13e0-980d-4388-9c0b-c2d81d499578",
    "identifier": "compute_checksum",
    "version": 1
  },
  "parameters":
  {
    "algorithm": "sha256"
  },
  "inputs":
  {
    "sources":
    [
      {
        "base-directory": "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data",
        "data-directory": "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/rare/throat",
        "type": "file",
        "hint": null,
        "data-size": 3423,
        "file-extensions":
        [
          "txt",
          "env",
          "sql",
          "lua"
        ],
        "local-paths":
        [
          "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/rare/throat/operation run duration.sql",
          "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/rare/throat/recover_lf.lua",
          "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/rare/throat/level2/make_dev.sql",
          "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/rare/throat/level2/make_utest.sql",
          "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/rare/throat/level2/level3/dev shuffle.txt",
          "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/rare/throat/level2/level3/run_on_localhost.env"
        ]
      },
      {
        "base-directory": "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data",
        "data-directory": "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/nicosia/campfire",
        "type": "file",
        "hint": null,
        "data-size": 1556,
        "file-extensions":
        [
          "json"
        ],
        "local-paths":
        [
          "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/nicosia/campfire/sample task input.json",
          "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/nicosia/campfire/v4_app_wkt.json"
        ]
      }
    ]
  },
  "outputs":
  {
    "hashes": "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/hashes"
  },
  "tds":
  {
    "execution-id": "33c1a9e6-06d6-11ec-8e65-0242ac110002",
    "operation-id": "jostle",
    "tid_owner_id": "test_fixture",
    "tid-owner-uuid": "00000000-0000-0000-0000-000000000000"
  },
  "reschedule":
  {
    "wait-time": null,
    "properties":
    {}
  }
}

Task Request Annotation

results_json specifies the absolute path of the JSON file the operation must write to specify the operation’s execution status and output parameters. The required contents are specified by result-json.md.

secondary_logs specifies the absolute directory path where log files from the engine containers entry point executable should be written. Log files written in this directory will be available for export using the API’s Request log contents resource.

tertiary_logs specifies the absolute directory path where log files from any additional executable in the container should be written. Log files written in this directory will be available for export using the API’s Request log contents resource.

operation specifies which of the engines operations to execute. The identifier and version keys correspond to the keys of the same name in the operation definition. The value of the id key is generated by the API when the operation is defined.

The compute_checksum operation definition specifies a single input parameter named algorithm.

    "parameters": {
      "algorithm": {
        "type": "single_choice",
        "optional": false,
        "description": "indicate which checksum algorithm to use",
        "options": [
          "SHA256",
          "MD5"
        ]
      }
    },

This results in a corresponding parameters section in the request JSON file. In this case the client application specified a value of “true” for the parameter when the execution was created.

  "parameters":
  {
    "algorithm": "sha256"
  },

Since algorithm is not defined as optional a value for it will always be included in the request JSON. If the parameter was optional it would be omitted from the request JSON if a value wasn’t supplied when the execution was created.

The operation definition specifies a single input port named sources.

"inputs": {
    "sources": {
    "name": "sources",
    "optional": false,
    "data_types": [
        "*"
    ],
    "description": "Files to hash"
    }
},

Which results in the corresponding inputs section in the request JSON file.

"inputs":
{
"sources":
[
    {
    "base-directory": "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data",
    "data-directory": "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/rare/throat",
    "type": "file",
    "hint": null,
    "data-size": 3423,
    "file-extensions":
    [
        "txt",
        "env",
        "sql",
        "lua"
    ],
    "local-paths":
    [
        "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/rare/throat/operation run duration.sql",
        "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/rare/throat/recover_lf.lua",
        "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/rare/throat/level2/make_dev.sql",
        "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/rare/throat/level2/make_utest.sql",
        "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/rare/throat/level2/level3/dev shuffle.txt",
        "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/rare/throat/level2/level3/run_on_localhost.env"
    ]
    },
    {
    "base-directory": "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data",
    "data-directory": "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/nicosia/campfire",
    "type": "file",
    "hint": null,
    "data-size": 1556,
    "file-extensions":
    [
        "json"
    ],
    "local-paths":
    [
        "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/nicosia/campfire/sample task input.json",
        "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/inputs/sources/data/nicosia/campfire/v4_app_wkt.json"
    ]
    }
]
},

The operation definition defined an input port named sources so the request JSON inputs object has a corresponding sources key. A procedure can feed the output ports of multiple operations into a single input port. This example execution chained two output ports into the sources input port so the value of the sources key is a list with two entries.

The value of the local-paths key is a list of absolute file paths with an entry for each file in the port.

An input port can contain a complex nested directory structure. The value of the data-directory key is absolute path of input port root. Subtracting data-directory from a local-paths value to determine the path the file had in the output port of the operation supplying the file.

The operation definition specifies a single output port named hashes.

    "outputs": {
      "hashes": {
        "data_type": "data",
        "description": "A text file with one line for each file in the input port"
      }
    },

Which results in the corresponding outputs section in the request JSON file.

"outputs":
{
  "hashes": "/taskscratch/33c1a9e6-06d6-11ec-8e65-0242ac110002/jostle/hashes"
},

The value of the hashes key is the absolute path of a directory where the contents of the operation’s hashes port should be written.

The request JSON’s tds key contains metadata from the procedure execution generating the current operation execution request. execution-id indicates the execution causing the current operation execution. Including the ID it in log messages simplifies associating the log messages with an execution. The operation-id key is the key assigned to the operation in the procedure definition. The tid_owner_id and tid_owner-uuid keys specify the application that created the execution.