Engine Logging

This section will briefly outline the two ways in which engine logs are handled, as streamed logs and as log files. Logs in the Processing Framework are sent to Sumologic, a log management service, where they are queryable.

Streamed Logs

Streamed logs are primarily used by the Processing team for support and are not readily accessible to the users who launch an Execution.

When a log is written to a Docker container’s stdout that message is captured by the Docker log driver and forwarded to sumologic. It is then accessible by the Processing team for debugging and support. Viewing logs in Sumologic is a primary way the Processing team diagnoses issues with engines and having clear and informative logs is critical.

Streamed logs are especially valuable for debugging containers that fail partway into an operation. Since they are captured and forwarded live they are more likely to be available even if the engine crashes in such a way that the log files are unrecoverable.

Log Files

Log files are available through the Log Export API to the user who launches an execution. When an engine container is launched and the taskscratch volume attached there are directories for log files specified in the task JSON. They are referred to as Secondary Logs and Tertiary Logs.

{
  ...
  "secondary_logs": "/taskscratch/secondary_logs33c1a9e6-06d6-11ec-8e65-0242ac110002",
  "tertiary_logs": "/taskscratch/tertiary_logs33c1a9e6-06d6-11ec-8e65-0242ac110002",
  ...
}

They are both locations to write or copy log files from the operation with slightly different use cases. The files are collected and handled after the engine container shuts down. Any log files not at one of these two locations when the container shuts down will not be available for retrieval using the Log Export API or available in Sumologic.

Secondary Logs

When these log files are collected at container shutdown they are copied to the log export storage location and then the contents of each file are streamed to Sumologic.

Tertiary Logs

These log files are also copied to the log export storage location but are not streamed to sumologic. This location is appropriate for logs already streamed by some other method or that are not intended to be accessible by the Processing team in sumologic.

When a log is both streamed to the container’s I/O and written to a file in secondary logs the result is that it will be captured in Sumologic twice. This should be avoided, both to have lower overall log volume and cleaner logs.

Retrieving Logs

Log files that have been saved from executions are retrievable using the Log Export API.

Processing Framework Request log contents API