Prism
v0.3.0
v0.3.0
  • 👋Welcome to Prism!
  • Getting Started
    • Installation
    • Creating your first project
    • Why Prism?
  • Fundamentals
    • PrismProject API
      • PrismProject().run
      • PrismProject().graph
    • Tasks
    • Targets
      • Multiple targets
    • CurrentRun API
      • CurrentRun.ref()
      • CurrentRun.conn()
      • CurrentRun.ctx()
  • Connectors
    • Overview
    • BigQueryConnector
    • PostgresConnector
    • RedshiftConnector
    • SnowflakeConnector
    • TrinoConnector
    • PrestoConnector
  • CLI
    • Command Line Interface
    • graph
    • init
    • run
  • Advanced features
    • Concurrency
    • Logging
    • Callbacks
    • Retries
    • Skipping tasks
  • API Reference
    • prism.task.PrismTask
    • @task(...)
    • @target(...)
    • @target_iterator(...)
    • prism.target.PrismTarget
  • Use Cases
    • Analytics on top of dbt
    • Machine Learning
  • Wiki
    • DAGs
Powered by GitBook
On this page
  1. Advanced features

Concurrency

PreviousrunNextLogging

Last updated 1 year ago

Overview

Prism supports concurrency via the concurrency keyword argument in the class. This variable tells Prism how many workers should execute the project.

The default value for concurrency is 1. This means that only one worker is used to execute the tasks in the Prism project, and so non-dependent tasks cannot be run in parallel.

Setting concurrency = 2 will allow Prism to use two worker processes to execute the tasks. This means that two tasks can be run simultaneously (assuming that those two tasks are not dependent on one another). For example:

For maximum concurrency, you can set THREADS = os.cpu_count():

# example_project.py

import os

from prism.client import PrismProject

project = PrismProject(concurrency=os.cpu_count())
PrismProject
Concurrency example.