Why Prism?

Prism vs. Airflow

In the world of data management, orchestrators are a critical tool for building and operating data pipelines. One of the most popular orchestrators is Apache Airflow, thanks to its Python-based design and web interface.

However, while Airflow effectively executes tasks in the correct order, it may not be the best choice for supporting the entire process of creating and running data pipelines. Airflow's design originated in a time when software engineering principles weren't yet fully integrated into the world of data management. As a result, it may not fully meet the needs of modern data teams.

For example, here are a few differences between Airflow and Prism.

1. Local development Airflow has an extremely abrasive local development environment. Users generally have to spin up a web server and a scheduler, which itself often requires knowledge of Docker, Redis and Celery. As a result, it's difficult to develop, test, or review pipelines outside a production deployment. Conversely, local development with Prism is easy. You can launch a run immediately using the CLI in a lightweight way.

2. Data as a first-class citizen In order to move data between tasks. users can either write the data to S3 (or some other cloud platform) or use XCOMs. These have a ton of limitations (e.g., you have to write custom logic to serialize / deserialize data, large XCOMs cause the web server to crash, etc.). Prism doesn't share these limitations. Tasks can return any object, and the tasks.ref(...) call will reference that object exactly without any additional logic needed.

3. Imperative vs declarative Airflow's DAG are imperative: you have to keep track of the pipeline order and provide instructions for which task should run when. In contrast, Prism projects are declarative. The pipeline order is automatically inferred by looking at function calls.

This list is not exhaustive, and the full slate of differences between Prism and Airflow can fill its own blog post.

TL;DR: Prism was designed to make data practitioners more productive. We'd love for you to join our community!