apply

Usage

prism agent apply constructs your agent using the information in your agent's configuration YML. For example, for Docker agents, prism apply constructs a custom Docker image containing your project data.

Usage: prism agent apply [-h] -f FILE [--full-tb] [-l]

Build your agent using a configuration YML

Options:
  -h, --help            show this help message and exit

Subcommand Options:
  -f, --file FILE       Path to agent configuration YML

General Options:
  --full-tb             Display the full traceback for errors in the project; default is False
  -l, --log-level       Log level, must be one of `info`, `warn`, `error`, or `debug`. Default is `info`

Here's what prism agent apply does under the hood:

  1. Parses the project's prism_project.py file and identifies all paths containing project related data (e.g., SYS_PATH_CONF, PROFILE_YML_PATH, TRIGGERS_YML_PATH, and so on).

  2. Securely builds the agent computing environment with all the data identified in step 1. For example:

    • For Docker agents, Prism constructs a transient build context and uses Docker's Python SDK to create an image.

    • For EC2 instances / EMR clusters (coming soon!), Prism uses the SSH protocol to securely copy all project data into the virtual machine.

Example

Here's what the output looks like in Terminal (using a Docker agent as an example):

$ prism agent apply -f ./docker_agent.yml
--------------------------------------------------------------------------------
<HH:MM:DD> | INFO  | Creating agent...
<HH:MM:DD> | INFO  | Did not find `server_url` in configuration...defaulting to unix://var/run/docker.sock

etl-project-docker-agent:1.0[build] | Step 1/7 : FROM python:3.10.8-slim-bullseye
etl-project-docker-agent:1.0[build] |  ---> db19fdf3337d
etl-project-docker-agent:1.0[build] | Step 2/7 : COPY etl_project/requirements.txt ./etl_project/requirements.txt
etl-project-docker-agent:1.0[build] |  ---> Using cache
etl-project-docker-agent:1.0[build] |  ---> 94b39b4230b9
etl-project-docker-agent:1.0[build] | Step 3/7 : RUN pip install --upgrade pip && pip install -r ./etl_project/requirements.txt
etl-project-docker-agent:1.0[build] |  ---> Using cache
etl-project-docker-agent:1.0[build] |  ---> 354ee738fadb
etl-project-docker-agent:1.0[build] | Step 4/7 : COPY etl_project ./etl_project
etl-project-docker-agent:1.0[build] |  ---> 46d1a5fc033c
etl-project-docker-agent:1.0[build] | Step 5/7 : WORKDIR ./etl_project
etl-project-docker-agent:1.0[build] |  ---> Running in dfef365c56da
etl-project-docker-agent:1.0[build] | Removing intermediate container dfef365c56da
etl-project-docker-agent:1.0[build] |  ---> a86e524ab13d
etl-project-docker-agent:1.0[build] | Step 6/7 : ENV AWS_ACCESS_KEY_ID=*****************
etl-project-docker-agent:1.0[build] |  ---> Running in afc0f4cae407
etl-project-docker-agent:1.0[build] | Removing intermediate container afc0f4cae407
etl-project-docker-agent:1.0[build] |  ---> 9a5cfa367c08
etl-project-docker-agent:1.0[build] | Step 7/7 : ENV AWS_SECRET_ACCESS_KEY=**********************************
etl-project-docker-agent:1.0[build] |  ---> Running in 11a198388a11
etl-project-docker-agent:1.0[build] | Removing intermediate container 11a198388a11
etl-project-docker-agent:1.0[build] |  ---> ef1ea0d753fb
etl-project-docker-agent:1.0[build] | Successfully built ef1ea0d753fb
etl-project-docker-agent:1.0[build] | Successfully tagged prism-docker-etl-project-docker-agent:1.0

Last updated