👋Welcome to Prism!
These docs current for version v0.2.7
.
Prism is the easiest way to create data pipelines in Python. With it, users can break down their data flows into modular tasks, manage dependencies, and execute complex computations in sequence.
CHANGELOG
Here are the main differences between this version and the previous version (v0.2.6
):
Why use Prism?
Prism was built to streamline the development and deployment of complex data pipelines. Here are some of its main features:
Real-time dependency declaration: With Prism, users can declare dependencies using a simple function call. No need to explicitly keep track of the pipeline order — at runtime, Prism automatically parses the function calls and builds the dependency graph.
Intuitive logging: Prism automatically logs events for parsing the configuration files, compiling the tasks and creating the DAG, and executing the tasks. No configuration is required.
Flexible CLI: Users can instantiate, compile, and run projects using a simple, but powerful command-line interface.
“Batteries included”: Prism comes with all the essentials needed to get up and running quickly. Users can create and run their first DAG in less than 2 minutes.
Integrations: Prism integrates with several tools that are popular in the data community, including Snowflake, Google BigQuery, Redshift, PySpark, and dbt. We're adding more integrations every day, so let us know what you'd like to see!
What is a Prism project?
At minimum, a Prism project must contain two things:
A directory of Python tasks called
tasks
, andA
prism_project.py
file for managing your Python environment
Important: in previous versions of Prism, the tasks
folder was called modules
. Version v0.2.0
still allows your folder to be called modules
, but you will see the following warning:
`modules` should be renamed to `tasks`...this will be an error in a future version of Prism
Here's how a typical Prism project is structured.
As you can see, projects can contain other folders, like a data
folder, an output
folder, and a profile.yml
file for connecting your project to external sources (more on this later).
Guides: Jump right in
Follow our handy guides to get started on the basics as quickly as possible:
Getting StartedFundamentalsCLIAPI ReferenceIf you have any feedback about the product or the docs, please let us know!
Last updated