run
Usage
prism agent run executes your project using the most recent version of the agent. For example, for Docker agents, prism run creates a transient Docker container for your custom Docker image and executes the inputted command.
Usage: prism agent run [OPTIONS]
Run your project using an agent.
Examples:
• prism agent run -f ./ec2.yml --module module01.py --module module02.py
• prism agent run -f ./docker.yml --m module01
• prism agent run -f /Users/docker.yml --vars VAR1=VALUE1
╭─ Options ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ * --file -f TEXT Path to agent configuration YML │
│ [required] │
│ --module -m TEXT Modules to execute. You can specify multiple modules with as follows: -m │
│ <your_first_module> -m <your_second_module>. │
│ --all-downstream Execute all tasks downstream of modules specified with --module │
│ --all-upstream Execute all tasks upstream of modules specified with --module │
│ --log-level -l [info|warn|error|debug] Set the log level │
│ --full-tb Show the full traceback when an error occurs │
│ --vars -v TEXT Variables as key value pairs. These overwrite variables in prism_project.py. │
│ All values are intepreted as strings. │
│ --context TEXT Context as a dictionary. Must be a valid JSON. These overwrite variables in │
│ prism_project.py │
│ --help Show this message and exit. │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯Example
Here's what the output looks like in Terminal (using a Docker agent as an example):
$ prism agent run -f ./docker_agent.yml
--------------------------------------------------------------------------------
<HH:MM:DD> | INFO | Streaming agent logs...
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO | Running with prism v0.2.1...
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO | Found project directory at /etl_project
etl-project-docker-agent:1.0[run] |
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO | RUNNING EVENT 'parsing prism_project.py'................................................ [RUN]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO | FINISHED EVENT 'parsing prism_project.py'............................................... [DONE in 1.42s]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO | RUNNING EVENT 'task DAG'................................................................ [RUN]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO | FINISHED EVENT 'task DAG'............................................................... [DONE in 0.01s]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO | RUNNING EVENT 'creating pipeline, DAG executor'......................................... [RUN]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO | FINISHED EVENT 'creating pipeline, DAG executor'........................................ [DONE in 0.01s]
etl-project-docker-agent:1.0[run] |
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO | ======================= tasks 'finicky-macaw-JaEvjyMWtb' =======================
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO | 1 of 3 RUNNING EVENT 'extract.Extract'...................................................[RUN]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO | 1 of 3 FINISHED EVENT 'extract.Extract'................................................. [DONE in 120.01s]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO | 2 of 3 RUNNING EVENT 'transform.Transform'.............................................. [RUN]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO | 2 of 3 FINISHED EVENT 'transform.Transform'............................................. [DONE in 791.38s]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO | 3 of 3 RUNNING EVENT 'load.Load'........................................................ [RUN]
etl-project-docker-agent:1.0[run] | <HH:MM:DD> | INFO | 3 of 3 FINISHED EVENT 'load.Load'....................................................... [DONE in 7.84s]
<HH:MM:DD> | INFO | Done streaming agent logs...
--------------------------------------------------------------------------------Last updated