Docker

Prerequisites

To run Docker agents locally, you must have the Docker engine installed on your machine. More information can be found here.

Configuring your Docker agent

Docker agents are configured using a YML file:

# docker_agent.yml

agent:
  type: docker
  image: <base image>
  server_url:  # URL to the docker server
  requirements:  # path to requirements.txt, relative to this file
  env:
    <env var 1>: 'env var 1 value'

The YML file has a single top-level key, agent. This key contains all the configurations needed for your agent. Specifically:

type: this will always be docker for Docker agents
image: the base image for your custom, Prism Docker image. If this is not specified, then Prism defaults to python:3.10.8-slim-bullseye.
server_url: URL to the Docker server. If this is not specified, then Prism defaults to unix://var/run/docker.sock
requirements: path to the requirements.txt file. This path must be relative to the path of the agent YML file.
env: {key,value} pairs representing environment variables to add to your Docker image

Example agent

For the remainder of this guide, let's assume that we have the following project.

etl_project/
  ├── prism_project.py
  ├── tasks/
  │   ├── extract.py
  │   ├── transfrom.py
  │   └── load.py
  ├── docker_agent.yml
  ├── requirements.txt
  └── triggers.yml

And, let's assume that we defined our docker_agent.yml as follows:

# docker_agent.yml

agent:
  type: docker
  requirements: ./requirements.txt
  env:
    AWS_ACCESS_KEY_ID: "{{ env('AWS_ACCESS_KEY_ID') }}"
    AWS_SECRET_ACCESS_KEY: "{{ env('AWS_SECRET_ACCESS_KEY') }}"

Note the way we specified the requirements key. We can do this because docker_agent.yml and requirements.txt are in the same directory.

Creating your Docker agent

You can build your Docker agent using the command: prism agent apply -f <path to agent YML>:

$ prism agent apply -f ./docker_agent.yml
--------------------------------------------------------------------------------
<HH:MM:DD> | INFO  | Creating agent...
<HH:MM:DD> | INFO  | Did not find `server_url` in configuration...defaulting to unix://var/run/docker.sock

etl-project-docker-agent:1.0[build] | Step 1/7 : FROM python:3.10.8-slim-bullseye
etl-project-docker-agent:1.0[build] |  ---> db19fdf3337d
etl-project-docker-agent:1.0[build] | Step 2/7 : COPY etl_project/requirements.txt ./etl_project/requirements.txt
etl-project-docker-agent:1.0[build] |  ---> Using cache
etl-project-docker-agent:1.0[build] |  ---> 94b39b4230b9
etl-project-docker-agent:1.0[build] | Step 3/7 : RUN pip install --upgrade pip && pip install -r ./etl_project/requirements.txt
etl-project-docker-agent:1.0[build] |  ---> Using cache
etl-project-docker-agent:1.0[build] |  ---> 354ee738fadb
etl-project-docker-agent:1.0[build] | Step 4/7 : COPY etl_project ./etl_project
etl-project-docker-agent:1.0[build] |  ---> 46d1a5fc033c
etl-project-docker-agent:1.0[build] | Step 5/7 : WORKDIR ./etl_project
etl-project-docker-agent:1.0[build] |  ---> Running in dfef365c56da
etl-project-docker-agent:1.0[build] | Removing intermediate container dfef365c56da
etl-project-docker-agent:1.0[build] |  ---> a86e524ab13d
etl-project-docker-agent:1.0[build] | Step 6/7 : ENV AWS_ACCESS_KEY_ID=*****************
etl-project-docker-agent:1.0[build] |  ---> Running in afc0f4cae407
etl-project-docker-agent:1.0[build] | Removing intermediate container afc0f4cae407
etl-project-docker-agent:1.0[build] |  ---> 9a5cfa367c08
etl-project-docker-agent:1.0[build] | Step 7/7 : ENV AWS_SECRET_ACCESS_KEY=**********************************
etl-project-docker-agent:1.0[build] |  ---> Running in 11a198388a11
etl-project-docker-agent:1.0[build] | Removing intermediate container 11a198388a11
etl-project-docker-agent:1.0[build] |  ---> ef1ea0d753fb
etl-project-docker-agent:1.0[build] | Successfully built ef1ea0d753fb
etl-project-docker-agent:1.0[build] | Successfully tagged prism-docker-etl-project-docker-agent:1.0

This commands builds a custom Docker image. In the example above, this image is called etl-project-docker-agent:1.0.

Multiple prism agent apply calls will increment the tag associated with the custom Docker image. For example, if we were to call prism agent apply again, then Prism would re-build the Docker image and name it etl-project-docker-agent:1.1.

Running your Docker agent

There are two commands you can use to run your Docker agent.

prism agent run

The prism agent run command runs a transient container with your Docker image. This container is automatically removed on daemon side when the container’s process exits.

$ prism agent run -f ./docker_agent.yml
--------------------------------------------------------------------------------
<HH:MM:DD> | INFO  | Streaming agent logs...

etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | Running with prism v0.2.1...
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | Found project directory at /etl_project
etl-project-docker-agent:1.0[run] |
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | RUNNING EVENT 'parsing prism_project.py'................................................ [RUN]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | FINISHED EVENT 'parsing prism_project.py'............................................... [DONE in 1.42s]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | RUNNING EVENT 'task DAG'................................................................ [RUN]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | FINISHED EVENT 'task DAG'............................................................... [DONE in 0.01s]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | RUNNING EVENT 'creating pipeline, DAG executor'......................................... [RUN]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | FINISHED EVENT 'creating pipeline, DAG executor'........................................ [DONE in 0.01s]
etl-project-docker-agent:1.0[run] |
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | ======================= tasks 'finicky-macaw-JaEvjyMWtb' =======================
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | 1 of 3 RUNNING EVENT 'extract.Extract'...................................................[RUN]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | 1 of 3 FINISHED EVENT 'extract.Extract'................................................. [DONE in 120.01s]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | 2 of 3 RUNNING EVENT 'transform.Transform'.............................................. [RUN]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | 2 of 3 FINISHED EVENT 'transform.Transform'............................................. [DONE in 791.38s]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | 3 of 3 RUNNING EVENT 'load.Load'........................................................ [RUN]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | 3 of 3 FINISHED EVENT 'load.Load'....................................................... [DONE in 7.84s]

<HH:MM:DD> | INFO  | Done streaming agent logs...
--------------------------------------------------------------------------------

prism agent build

The prism agent build command re-builds your Docker image and then runs a transient container with your newly built image. As with prism agent run, this container is automatically removed on daemon side when the container’s process exits.

$ prism agent build -f ./docker_agent.yml
--------------------------------------------------------------------------------
<HH:MM:DD> | INFO  | Creating agent...
<HH:MM:DD> | INFO  | Did not find `server_url` in configuration...defaulting to unix://var/run/docker.sock

etl-project-docker-agent:1.1[build] | Step 1/7 : FROM python:3.10.8-slim-bullseye
etl-project-docker-agent:1.1[build] |  ---> db19fdf3337d
etl-project-docker-agent:1.1[build] | Step 2/7 : COPY etl_project/requirements.txt ./etl_project/requirements.txt
etl-project-docker-agent:1.1[build] |  ---> Using cache
etl-project-docker-agent:1.1[build] |  ---> 94b39b4230b9
etl-project-docker-agent:1.1[build] | Step 3/7 : RUN pip install --upgrade pip && pip install -r ./etl_project/requirements.txt
etl-project-docker-agent:1.1[build] |  ---> Using cache
etl-project-docker-agent:1.1[build] |  ---> 354ee738fadb
etl-project-docker-agent:1.1[build] | Step 4/7 : COPY etl_project ./etl_project
etl-project-docker-agent:1.1[build] |  ---> 46d1a5fc033c
etl-project-docker-agent:1.1[build] | Step 5/7 : WORKDIR ./etl_project
etl-project-docker-agent:1.1[build] |  ---> Running in dfef365c56da
etl-project-docker-agent:1.1[build] | Removing intermediate container dfef365c56da
etl-project-docker-agent:1.1[build] |  ---> a86e524ab13d
etl-project-docker-agent:1.1[build] | Step 6/7 : ENV AWS_ACCESS_KEY_ID=*****************
etl-project-docker-agent:1.1[build] |  ---> Running in afc0f4cae407
etl-project-docker-agent:1.1[build] | Removing intermediate container afc0f4cae407
etl-project-docker-agent:1.1[build] |  ---> 9a5cfa367c08
etl-project-docker-agent:1.1[build] | Step 7/7 : ENV AWS_SECRET_ACCESS_KEY=**********************************
etl-project-docker-agent:1.1[build] |  ---> Running in 11a198388a11
etl-project-docker-agent:1.1[build] | Removing intermediate container 11a198388a11
etl-project-docker-agent:1.1[build] |  ---> ef1ea0d753fb
etl-project-docker-agent:1.1[build] | Successfully built ef1ea0d753fb
etl-project-docker-agent:1.1[build] | Successfully tagged etl-project-docker-agent:1.1

<HH:MM:DD> | INFO  | Streaming agent logs...

etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | Running with prism v0.2.1...
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | Found project directory at /etl_project
etl-project-docker-agent:1.0[run] |
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | RUNNING EVENT 'parsing prism_project.py'................................................ [RUN]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | FINISHED EVENT 'parsing prism_project.py'............................................... [DONE in 1.42s]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | RUNNING EVENT 'task DAG'................................................................ [RUN]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | FINISHED EVENT 'task DAG'............................................................... [DONE in 0.01s]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | RUNNING EVENT 'creating pipeline, DAG executor'......................................... [RUN]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | FINISHED EVENT 'creating pipeline, DAG executor'........................................ [DONE in 0.01s]
etl-project-docker-agent:1.0[run] |
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | ======================= tasks 'finicky-macaw-JaEvjyMWtb' =======================
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | 1 of 3 RUNNING EVENT 'extract.Extract'...................................................[RUN]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | 1 of 3 FINISHED EVENT 'extract.Extract'................................................. [DONE in 120.01s]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | 2 of 3 RUNNING EVENT 'transform.Transform'.............................................. [RUN]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | 2 of 3 FINISHED EVENT 'transform.Transform'............................................. [DONE in 791.38s]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | 3 of 3 RUNNING EVENT 'load.Load'........................................................ [RUN]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | 3 of 3 FINISHED EVENT 'load.Load'....................................................... [DONE in 7.84s]

<HH:MM:DD> | INFO  | Done streaming agent logs...

Notice that this command re-built the image: now, the image tag is 1.1, not 1.0.

More information on these commands can be found in the CLI section.

PreviousOverview NextEC2

Last updated 1 year ago