Prism
v0.1.9rc2
v0.1.9rc2
  • 👋Welcome to Prism!
  • Getting Started
    • Installation
    • Creating your first project
    • Why Prism?
  • Fundamentals
    • Tasks
      • tasks
      • hooks
        • hooks.sql
        • hooks.spark
        • hooks.dbt_ref
    • Targets
      • Multiple targets
    • Config files
      • prism_project.py
        • RUN_ID / SLUG
        • SYS_PATH_CONF
        • THREADS
        • PROFILE_YML_PATH / PROFILE
        • PRISM_LOGGER
        • TRIGGERS_YML_PATH / TRIGGERS
      • Profile YML
      • Triggers YML
    • Jinja
      • __file__ and Path
      • prism_project
      • wkdir
      • parent_dir
      • concat
      • env
  • Adapters
    • Overview
    • sql
      • BigQuery
      • Postgres
      • Redshift
      • Snowflake
      • Trino
    • PySpark
    • dbt
  • Agents
    • Overview
    • Docker
    • EC2
  • CLI
    • Command Line Interface
    • init
    • compile
    • connect
    • create
      • agent
      • task
      • trigger
    • graph
    • run
    • spark-submit
    • agent
      • apply
      • run
      • build
      • delete
  • Advanced features
    • Concurrency
    • Logging
    • Triggers
    • Retries
    • Python Client
  • API Reference
    • prism.task.PrismTask
    • @task(...)
    • @target(...)
    • @target_iterator(...)
    • tasks.ref(...)
    • hooks.sql(...)
    • hooks.dbt_ref(...)
  • Use Cases
    • Analytics on top of dbt
    • Machine Learning
  • Wiki
    • DAGs
Powered by GitBook
On this page
  • Usage
  • Example
  • Required arguments
  • Optional arguments
  1. CLI

run

PreviousgraphNextspark-submit

Last updated 1 year ago

Usage

prism run is used to execute a Python-based Prism project. This is distinct from a PySpark-based Prism project, in which case you would need to use the command.

Usage: prism run [-h] [--modules [X.py Y.py ...]] [--all-upstream] [--all-downstream] [--full-tb] [-l] [--vars [A=a B=b ...]] [--context '{}']

Execute your Prism project

Options:
  -h, --help            show this help message and exit

Command Options:
  --modules [X.py Y.py ...]
                        Path to script(s) that you want to run; if not specified, all modules in project are run
  --all-upstream        Run all modules upstream of --modules
  --all-downstream      Run all modules downstream of --modules

General Options:
  --full-tb             Display the full traceback for errors in the project; default is False
  -l, --log-level       Log level, must be one of `info`, `warn`, `error`, or `debug`. Default is `info`
  --vars [A=a B=b ...]  Prism variables as key-value pairs `key=value`. These overwrite any variable definitions in `prism_project.py`. All values are read as strings.
  --context '{}'        Prism variables as JSON. Cannot co-exist with --vars. These overwrite any variable definitions in `prism_project.py`.

Here's what prism run does under the hood:

  1. It creates a pipeline consisting of the global variables from prism_project.py and the adapters from profile.yml.

Example

Here's what the output looks like in Terminal:

$ prism run
--------------------------------------------------------------------------------
<HH:MM:SS> | INFO  | Running with prism v0.1.9rc2...
<HH:MM:SS> | INFO  | Found project directory at /Users/my_first_project
 
<HH:MM:SS> | INFO  | RUNNING EVENT 'parsing prism_project.py'................................................ [RUN]
<HH:MM:SS> | INFO  | FINISHED EVENT 'parsing prism_project.py'............................................... [DONE in 0.03s]
<HH:MM:SS> | INFO  | RUNNING EVENT 'module DAG'.............................................................. [RUN]
<HH:MM:SS> | INFO  | FINISHED EVENT 'module DAG'............................................................. [DONE in 0.01s]
<HH:MM:SS> | INFO  | RUNNING EVENT 'creating pipeline, DAG executor'......................................... [RUN]
<HH:MM:SS> | INFO  | FINISHED EVENT 'creating pipeline, DAG executor'........................................ [DONE in 0.01s]
 
<HH:MM:SS> | INFO  | ===================== tasks 'vermilion-hornet-Gyycw4kRWG' =====================
<HH:MM:SS> | INFO  | 1 of 1 RUNNING EVENT 'module01.py'...................................................... [RUN]
<HH:MM:SS> | INFO  | 1 of 1 FINISHED EVENT 'module01.py'..................................................... [DONE in 0.01s]
 
<HH:MM:SS> | INFO  | Done!
--------------------------------------------------------------------------------

Required arguments

There are no required arguments for run.

Optional arguments

Here are the optional arguments you can run with run:

  • --full-tb : Display full traceback if errors arise at any stage of the pipeline

  • --log-level: Log level, one of info, warn, error, or critical.

  • --vars: Prism variables as key-value pairs key=value. These overwrite any variable definitions in prism_project.py. All values are read as strings.

  • --context: Prism variables as JSON. Cannot co-exist with --vars. These overwrite any variable definitions in prism_project.py.

  • --modules: Paths to modules that you want to run; if not specified, all modules in pipeline are run. Paths should be specified relative to the modules folder.

  • --all-upstream: Run all modules upstream of those specified in --modules

  • --all-downstream: Run all modules downstream of those specified in --modules

It first executes the command.

Recall that this command parses the tasks contained in the modules/ folder for tasks.ref(...) calls, creates the dependency graph, and stores the project metadata (e.g., the configuration, the tasks.ref(...) calls, the targets, and the ) in a manifest.

It parses the prism_project.py file. It also parses the configuration files listed within prism_project.py (e.g., , ).

It executes the tasks in a .

spark-submit
prism compile
topological sort
PROFILE_YML_PATH
TRIGGERS_YML_PATH
topological order