Skip to content
Processing Framework

Suggestions for Building an Engine Container

As long as an Engine container implements the required external interface, the internal details of the container are the purview of the integrator. The integrator can choose the tools and techniques used to build the container image. With that in mind, this section provides some suggestions for developers who are new to Python and Docker.

The integritychecker engine used as an example in this document is available in this repository. If you do not already have access to this repository, please request access through Cloud Engagement Team.

The engine is implemented as a Python module which the container will launch with the command pythom -m integritychecker. Implementing the engine as a module simplifies writing unit tests for the module as well as installing it in a container.

{gitRoot} will represent the root of the engine’s git repository. This directory structure below works well for Python engines.

{gitRoot}
│   .gitignore
│   README.md
│
└───container
    │   .dockerignore
    │   dockerfile
    │   requirements.txt
    │
    └───app
        │   pyproject.toml
        │   setup.cfg
        │   setup.py
        │
        └───integritychecker
                __init__.py
                __main__.py

  • The container directory is used as the engine’s docker build context.

  • container/app is the engine’s Python package build directory

  • container/app/integritychecker contains the Python module that implements the engine. This directory name will be used as the Python package name.

Note: It may be useful to include your executable within this repo so it can easily be incorporated into the docker image.

See below for more information

Python Module Implementation

An empty container/app/integritychecker/__init__.py file is required to identify the container/app/integritychecker/ directory as a Python package.

The container/app/integritychecker/__main__.py the_dispatcher.py file contains the actual engine implementation. The section Explanation of __main__.py discusses the contents of the file.

Python Module Packaging

The pyproject.toml file is a declarative definition of the Python package.

[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"

The setup.cfg file is a declarative definition of the engine Python package. The file’s format is documented at https://setuptools.readthedocs.io/en/latest/userguide/declarative_config.html. For reference the contents of the setup.cfg file is replicated below.

[metadata]
name = integritychecker
version = 1.0.0
author = TrimbleData:Processing
author_email = data@trimbledata.com
description = Engine that computes file hashes developed to serve as a training sample for engagement
requires_python = >=3.8
url = https://bitbucket.trimble.tools/users/kevin_chapman_trimble.com/repos/split-f3dconverter


[options]
packages = integritychecker
install_requires =
    tdsplitengine>=1.1.0

Explanation of the setup.cfg file

name = integritychecker

The name element of the [metadata] section specifies the name of the Python package built by this configuration. This should have the same value as the directory name containing the package’s Python code.

packages = integritychecker

The packages element of the [options] sections specifies the directory that contains the package’s Python code.

install_requires =
    tdsplitengine>=1.1.0

Lists the Python packages imported by this package. This requirements list is also used when building the requirements.txt file.

[options.package_data]
* = *.conf, *.yaml

This simple engine doesn’t have support files. An options.package_data section can be used to include non Python files that the engine needs. A common example is a configuration file.

Explanation of the setup.py file

When a setup.cfg file is present the setup.py file is not involved in the building or installation of the package. A setup.py file is only included to support an editable install of the package. An editable package install is useful for running unit tests and supporting IDE code completion.

For reference the contents of the setup.py file is replicated below.

import setuptools


if __name__ == "__main__":
    setuptools.setup()

Define a Docker Image

Specify Engine Prerequisites Using a requirements.txt File

Installing the prerequisite packages using a requirements.txt file installs them on an early layer that is expected to be more static than the engine code. A requirements.txt file also creates reproducible builds because it freezes the prerequisites until explicitly updated.

The requirements.txt file can be generated using the pip-compile command from the pip-tools Python package. You should already be using a virtual environment to develop the engine and pip-tools must be installed into the same virtual environment. Install pip-tools into the virtual environment using the command:

Terminal window
pip install pip-tools

Change the current directory to the container/app/ directory.

Build a requirements.txt file with the command:

Terminal window
pip-compile -U --output-file='..\requirements.txt' 'setup.cfg'

The requirements file should be built on the same operating system that the engine container runs. Package dependencies are not necessarily the same across operating systems and can cause issues during build if incorrectly specified.

Exclude unneeded content using .dockerignore

Files in the build context that are not needed while building the image can be excluded using the .dockerignore file. Files listed in .dockerignore will not be copied to the build host. At the minimum dockerfile and .dockerignore can be excluded. For reference

.dockerignore
dockerfile

Create a dockerfile

Using a dockerfile to build an image is the preferred way to create an image. A dockerfile allows reproducibility of images that allows them to be rebuilt with minimal changes. See here for more information regarding docker image best practices.

The steps required to build a docker image containing the engine are defined in a dockerfile. The directory containing the dockerfile is used as the build context. The docker agent only has access to the directory containing the dockerfile and its children.

Create the file at container/dockerfile.For example the contents of the integritychecker engine’s docker file follows:

FROM python:3.8-slim-buster


LABEL maintainer="data@trimbledata.com"


COPY requirements.txt /pf_build/


# Older versions of pip don't support environment variable injection so pip
# must be upgraded after the virtual environment is created.
RUN /usr/local/bin/python -m venv /opt/engine_venv && \
    /opt/engine_venv/bin/python -m pip install --no-cache-dir --upgrade pip && \
    /opt/engine_venv/bin/python -m pip install --no-cache-dir -r /pf_build/requirements.txt


# Prefixing path with the venv to make manual exploration easier. Not needed to run the container
ENV PATH="/opt/engine_venv/bin:$PATH"


COPY ./app /pf_build/
RUN /opt/engine_venv/bin/python -m pip install --no-cache-dir /pf_build


WORKDIR /
USER 65534
CMD ["/opt/engine_venv/bin/python", "-m", "integritychecker"]

The sha256sum and md5sum command are included in the python:3.8-slim-buster base image so this dockerfile doesn’t need to install them. Most engine images will need to install additional executables on the image. There are several common approaches for installing the additional executables.

  • An installer can be saved under the container directory and then use a COPY statement to copy the installer into the image.
  • Use a RUN command to run an apt-get command to install the executable on the image.
  • Use a RUN command to download an installer from Trimble eTools.
  • Use a RUN command to download an installer using curl or Invoke-RestMethod.
COPY requirements.txt /pf_build/

This copy copies a requirements.txt file into the docker image so that it can be referenced in the subsequent RUN command. Requirements.txt is copied early in the dockerfile and intendent of the rest of the build context to speed up subsequent builds. The Python requirements will probably change less frequently than the engine code so copying just requirements.txt allows the prerequisites to be cached.

RUN /usr/local/bin/python -m venv /opt/engine_venv && \
    /opt/engine_venv/bin/python -m pip install --no-cache-dir --upgrade pip && \
    /opt/engine_venv/bin/python -m pip install --no-cache-dir -r /pf_build/requirements.txt

This run command creates a virtual Python environment and installs the engine’s prerequisites defined in the requirements.txt file into the virtual environment.

Note: It is not strictly necessary to use a virtual Python environment, but it is recommended

See below for more information.

COPY ./app /pf_build/

This copy command copies the entire contents of the app directory except for files explicitly ignored in the .dockerignore file into the image’s /pf_build directory.

RUN /opt/engine_venv/bin/python -m pip install --no-cache-dir /pf_build

This RUN command installs the engine app form source. The {gitRoot}/container/app/setup.cfg file defines the actual details of the install.

WORKDIR /

Make sure /pf_build isn’t the current directory when the engine is started. The Python interpreter includes the current directory in sys.path and the engine should be loaded from the virtual environment instead of /pf_build.

USER 65534

ISO prohibits running engines under the Administrator account. This command causes the engine to run under user id 65534. This value must be an integer, non-zero, user id in Linux containers. A Windows container should run under the containeruser account. In both cases the Processing system will reject containers that use an Administrator account. See Choosing a User ID for more details.

CMD ["/opt/engine_venv/bin/python", "-m", "integritychecker"]

Specifies that the integritychecker package should be executed when the container is run.

Building the Docker Image

Set the current directory to the container directory. This directory should contain the dockerfile. Build the image with the command:

docker build -t int-engine/<engine>-linux:latest .

Where int-engine/<engine>-linux specifies the image name. And :latest specifies a tag to assign the image. When creating a new version of your image, convention is to use the same image name with an updated image tag.

Note the trailing . which defined our current directory as the build context

Notes

Python Virtual Environment

Benefits

Virtual environments let you have a stable, reproducible, and portable environment. Virtual environments give control which packages versions are installed and when they are upgraded. This helps avoid potentially versioning issues.

It is highly recommended to use a virtual environment

Issues

There are rare occurences where the use of a virtual environment causes issues running the split engine package. Issues can stem from a few reasons:

  • Permissions issues
  • Faulty virtual environment creation
  • Faulty python installation

If this is the case, avoiding a virtual environment may solve the issues.

Including your executable

The example used in this tutorial follows using a standard executables sha256sum and md5sum. Many times, the image will need to include a custom executable to run business logic.

Storing your executable

It may be beneficial to include this within your git repository. This would be included in the container directory described above:

{gitRoot}
│   .gitignore
│   README.md
│
└───container
    │   .dockerignore
    │   dockerfile
    │   requirements.txt
    │
    └───custom_executable/
    │       the_executable
    │
    └───app
        │   pyproject.toml
        │   setup.cfg
        │   setup.py
        │
        └───integritychecker
                __init__.py
                __main__.py

Adding your executable

If the executable is stored within the git repository, it can be included to your image within the dockerfile by adding a COPY command to the dockerfile:

COPY ./custom_executable/ /custom_executable/