Prism
v0.1.9rc2
v0.1.9rc2
  • 👋Welcome to Prism!
  • Getting Started
    • Installation
    • Creating your first project
    • Why Prism?
  • Fundamentals
    • Tasks
      • tasks
      • hooks
        • hooks.sql
        • hooks.spark
        • hooks.dbt_ref
    • Targets
      • Multiple targets
    • Config files
      • prism_project.py
        • RUN_ID / SLUG
        • SYS_PATH_CONF
        • THREADS
        • PROFILE_YML_PATH / PROFILE
        • PRISM_LOGGER
        • TRIGGERS_YML_PATH / TRIGGERS
      • Profile YML
      • Triggers YML
    • Jinja
      • __file__ and Path
      • prism_project
      • wkdir
      • parent_dir
      • concat
      • env
  • Adapters
    • Overview
    • sql
      • BigQuery
      • Postgres
      • Redshift
      • Snowflake
      • Trino
    • PySpark
    • dbt
  • Agents
    • Overview
    • Docker
    • EC2
  • CLI
    • Command Line Interface
    • init
    • compile
    • connect
    • create
      • agent
      • task
      • trigger
    • graph
    • run
    • spark-submit
    • agent
      • apply
      • run
      • build
      • delete
  • Advanced features
    • Concurrency
    • Logging
    • Triggers
    • Retries
    • Python Client
  • API Reference
    • prism.task.PrismTask
    • @task(...)
    • @target(...)
    • @target_iterator(...)
    • tasks.ref(...)
    • hooks.sql(...)
    • hooks.dbt_ref(...)
  • Use Cases
    • Analytics on top of dbt
    • Machine Learning
  • Wiki
    • DAGs
Powered by GitBook
On this page
  • Prerequisites
  • Configuring your Docker agent
  • Example agent
  • Creating your Docker agent
  • Running your Docker agent
  1. Agents

Docker

PreviousOverviewNextEC2

Last updated 1 year ago

Prerequisites

To run Docker agents locally, you must have the Docker engine installed on your machine. More information can be found .

Configuring your Docker agent

Docker agents are configured using a YML file:

# docker_agent.yml

agent:
  type: docker
  image: <base image>
  server_url:  # URL to the docker server
  requirements:  # path to requirements.txt, relative to this file
  env:
    <env var 1>: 'env var 1 value'

The YML file has a single top-level key, agent. This key contains all the configurations needed for your agent. Specifically:

  • type: this will always be docker for Docker agents

  • image: the base image for your custom, Prism Docker image. If this is not specified, then Prism defaults to .

  • server_url: URL to the Docker server. If this is not specified, then Prism defaults to unix://var/run/docker.sock

  • requirements: path to the requirements.txt file. This path must be relative to the path of the agent YML file.

  • env: {key,value} pairs representing environment variables to add to your Docker image

Example agent

For the remainder of this guide, let's assume that we have the following project.

etl_project/
  ├── prism_project.py
  ├── modules/
  │   ├── extract.py
  │   ├── transfrom.py
  │   └── load.py
  ├── docker_agent.yml
  ├── requirements.txt
  └── triggers.yml

And, let's assume that we defined our docker_agent.yml as follows:

# docker_agent.yml

agent:
  type: docker
  requirements: ./requirements.txt
  env:
    AWS_ACCESS_KEY_ID: "{{ env('AWS_ACCESS_KEY_ID') }}"
    AWS_SECRET_ACCESS_KEY: "{{ env('AWS_SECRET_ACCESS_KEY') }}"

Note the way we specified the requirements key. We can do this because docker_agent.yml and requirements.txt are in the same directory.

Creating your Docker agent

You can build your Docker agent using the command: prism agent apply -f <path to agent YML>:

$ prism agent apply -f ./docker_agent.yml
--------------------------------------------------------------------------------
<HH:MM:DD> | INFO  | Creating agent...
<HH:MM:DD> | INFO  | Did not find `server_url` in configuration...defaulting to unix://var/run/docker.sock

etl-project-docker-agent:1.0[build] | Step 1/7 : FROM python:3.10.8-slim-bullseye
etl-project-docker-agent:1.0[build] |  ---> db19fdf3337d
etl-project-docker-agent:1.0[build] | Step 2/7 : COPY etl_project/requirements.txt ./etl_project/requirements.txt
etl-project-docker-agent:1.0[build] |  ---> Using cache
etl-project-docker-agent:1.0[build] |  ---> 94b39b4230b9
etl-project-docker-agent:1.0[build] | Step 3/7 : RUN pip install --upgrade pip && pip install -r ./etl_project/requirements.txt
etl-project-docker-agent:1.0[build] |  ---> Using cache
etl-project-docker-agent:1.0[build] |  ---> 354ee738fadb
etl-project-docker-agent:1.0[build] | Step 4/7 : COPY etl_project ./etl_project
etl-project-docker-agent:1.0[build] |  ---> 46d1a5fc033c
etl-project-docker-agent:1.0[build] | Step 5/7 : WORKDIR ./etl_project
etl-project-docker-agent:1.0[build] |  ---> Running in dfef365c56da
etl-project-docker-agent:1.0[build] | Removing intermediate container dfef365c56da
etl-project-docker-agent:1.0[build] |  ---> a86e524ab13d
etl-project-docker-agent:1.0[build] | Step 6/7 : ENV AWS_ACCESS_KEY_ID=*****************
etl-project-docker-agent:1.0[build] |  ---> Running in afc0f4cae407
etl-project-docker-agent:1.0[build] | Removing intermediate container afc0f4cae407
etl-project-docker-agent:1.0[build] |  ---> 9a5cfa367c08
etl-project-docker-agent:1.0[build] | Step 7/7 : ENV AWS_SECRET_ACCESS_KEY=**********************************
etl-project-docker-agent:1.0[build] |  ---> Running in 11a198388a11
etl-project-docker-agent:1.0[build] | Removing intermediate container 11a198388a11
etl-project-docker-agent:1.0[build] |  ---> ef1ea0d753fb
etl-project-docker-agent:1.0[build] | Successfully built ef1ea0d753fb
etl-project-docker-agent:1.0[build] | Successfully tagged prism-docker-etl-project-docker-agent:1.0

This commands builds a custom Docker image. In the example above, this image is called etl-project-docker-agent:1.0.

Multiple prism agent apply calls will increment the tag associated with the custom Docker image. For example, if we were to call prism agent apply again, then Prism would re-build the Docker image and name it etl-project-docker-agent:1.1.

Running your Docker agent

There are two commands you can use to run your Docker agent.

prism agent run

The prism agent run command runs a transient container with your Docker image. This container is automatically removed on daemon side when the container’s process exits.

$ prism agent run -f ./docker_agent.yml
--------------------------------------------------------------------------------
<HH:MM:DD> | INFO  | Streaming agent logs...

etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | Running with prism v0.1.9rc2...
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | Found project directory at /etl_project
etl-project-docker-agent:1.0[run] |
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | RUNNING EVENT 'parsing prism_project.py'................................................ [RUN]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | FINISHED EVENT 'parsing prism_project.py'............................................... [DONE in 1.42s]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | RUNNING EVENT 'module DAG'.............................................................. [RUN]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | FINISHED EVENT 'module DAG'............................................................. [DONE in 0.01s]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | RUNNING EVENT 'creating pipeline, DAG executor'......................................... [RUN]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | FINISHED EVENT 'creating pipeline, DAG executor'........................................ [DONE in 0.01s]
etl-project-docker-agent:1.0[run] |
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | ======================= tasks 'finicky-macaw-JaEvjyMWtb' =======================
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | 1 of 3 RUNNING EVENT 'extract.py'....................................................... [RUN]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | 1 of 3 FINISHED EVENT 'extract.py'...................................................... [DONE in 120.01s]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | 2 of 3 RUNNING EVENT 'transform.py'..................................................... [RUN]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | 2 of 3 FINISHED EVENT 'transform.py'.................................................... [DONE in 791.38s]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | 3 of 3 RUNNING EVENT 'load.py'.......................................................... [RUN]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | 3 of 3 FINISHED EVENT 'load.py'......................................................... [DONE in 7.84s]

<HH:MM:DD> | INFO  | Done streaming agent logs...
--------------------------------------------------------------------------------
prism agent build

The prism agent build command re-builds your Docker image and then runs a transient container with your newly built image. As with prism agent run, this container is automatically removed on daemon side when the container’s process exits.

$ prism agent build -f ./docker_agent.yml
--------------------------------------------------------------------------------
<HH:MM:DD> | INFO  | Creating agent...
<HH:MM:DD> | INFO  | Did not find `server_url` in configuration...defaulting to unix://var/run/docker.sock

etl-project-docker-agent:1.1[build] | Step 1/7 : FROM python:3.10.8-slim-bullseye
etl-project-docker-agent:1.1[build] |  ---> db19fdf3337d
etl-project-docker-agent:1.1[build] | Step 2/7 : COPY etl_project/requirements.txt ./etl_project/requirements.txt
etl-project-docker-agent:1.1[build] |  ---> Using cache
etl-project-docker-agent:1.1[build] |  ---> 94b39b4230b9
etl-project-docker-agent:1.1[build] | Step 3/7 : RUN pip install --upgrade pip && pip install -r ./etl_project/requirements.txt
etl-project-docker-agent:1.1[build] |  ---> Using cache
etl-project-docker-agent:1.1[build] |  ---> 354ee738fadb
etl-project-docker-agent:1.1[build] | Step 4/7 : COPY etl_project ./etl_project
etl-project-docker-agent:1.1[build] |  ---> 46d1a5fc033c
etl-project-docker-agent:1.1[build] | Step 5/7 : WORKDIR ./etl_project
etl-project-docker-agent:1.1[build] |  ---> Running in dfef365c56da
etl-project-docker-agent:1.1[build] | Removing intermediate container dfef365c56da
etl-project-docker-agent:1.1[build] |  ---> a86e524ab13d
etl-project-docker-agent:1.1[build] | Step 6/7 : ENV AWS_ACCESS_KEY_ID=*****************
etl-project-docker-agent:1.1[build] |  ---> Running in afc0f4cae407
etl-project-docker-agent:1.1[build] | Removing intermediate container afc0f4cae407
etl-project-docker-agent:1.1[build] |  ---> 9a5cfa367c08
etl-project-docker-agent:1.1[build] | Step 7/7 : ENV AWS_SECRET_ACCESS_KEY=**********************************
etl-project-docker-agent:1.1[build] |  ---> Running in 11a198388a11
etl-project-docker-agent:1.1[build] | Removing intermediate container 11a198388a11
etl-project-docker-agent:1.1[build] |  ---> ef1ea0d753fb
etl-project-docker-agent:1.1[build] | Successfully built ef1ea0d753fb
etl-project-docker-agent:1.1[build] | Successfully tagged etl-project-docker-agent:1.1

<HH:MM:DD> | INFO  | Streaming agent logs...

etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | Running with prism v0.1.9rc2...
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | Found project directory at /etl_project
etl-project-docker-agent:1.0[run] |
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | RUNNING EVENT 'parsing prism_project.py'................................................ [RUN]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | FINISHED EVENT 'parsing prism_project.py'............................................... [DONE in 1.42s]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | RUNNING EVENT 'module DAG'.............................................................. [RUN]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | FINISHED EVENT 'module DAG'............................................................. [DONE in 0.01s]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | RUNNING EVENT 'creating pipeline, DAG executor'......................................... [RUN]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | FINISHED EVENT 'creating pipeline, DAG executor'........................................ [DONE in 0.01s]
etl-project-docker-agent:1.0[run] |
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | ======================= tasks 'finicky-macaw-JaEvjyMWtb' =======================
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | 1 of 3 RUNNING EVENT 'extract.py'....................................................... [RUN]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | 1 of 3 FINISHED EVENT 'extract.py'...................................................... [DONE in 120.01s]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | 2 of 3 RUNNING EVENT 'transform.py'..................................................... [RUN]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | 2 of 3 FINISHED EVENT 'transform.py'.................................................... [DONE in 791.38s]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | 3 of 3 RUNNING EVENT 'load.py'.......................................................... [RUN]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO  | 3 of 3 FINISHED EVENT 'load.py'......................................................... [DONE in 7.84s]

<HH:MM:DD> | INFO  | Done streaming agent logs...

Notice that this command re-built the image: now, the image tag is 1.1, not 1.0.

More information on these commands can be found in the section.

here
python:3.10.8-slim-bullseye
CLI