👋Welcome to Prism!

These docs current for version v0.2.5.

Prism is the easiest way to create data pipelines in Python. With it, users can break down their data flows into modular tasks, manage dependencies, and execute complex computations in sequence.

CHANGELOG

Here are the main differences between this version and the previous version (v0.2.4):

tasks.ref() opens a target instead of just returning its Path

• In version v0.2.4 and before, retrieving the output of a task with a target would return the target path.

  • For example, consider the following task:

  # tasks/hello_world.py
  
  import prism.task
  import prism.target
  import prism.decorators
  
  class HelloWorld(prism.task.PrismTask):
      
      @prism.decorators.target(
          type=prism.target.Txt, 
          loc="/Users/hello_world.txt", 
          **kwargs
      )
      def run(self, tasks, hooks):
          test_str = "Hello, world!"
          return test_str

• In previous versions, tasks.ref("hello_world") would return "/Users/hello_world.txt". Users would have to write custom logic to open their target.

• Now, tasks.ref(...) automatically opens the target and returns its contents. In the above example, tasks.ref("hello_world") would return the contents of the hello_world.txt target—"Hello, world!".

IMPORTANT: this only applies to @target calls. The @target_iterator still returns the parent path to which the targets are saved.

PrismTarget class definition

• In line with the previous update, PrismTarget child classes have a class method called open. This class method defines how tasks.ref() should open the target when it encounters one.

• For example, here is the exact code used for the Txt target.

class Txt(PrismTarget):

    def save(self, **kwargs):
        with open(self.loc, "w") as f:
            f.write(self.obj, **kwargs)
        f.close()
    
    @classmethod  # <-- this is new! 
    def open(cls, loc, hooks):
        with open(loc, 'r') as f:
            obj = f.read()
        return cls(obj, loc, hooks)

Why use Prism?

Prism was built to streamline the development and deployment of complex data pipelines. Here are some of its main features:

• Real-time dependency declaration: With Prism, users can declare dependencies using a simple function call. No need to explicitly keep track of the pipeline order — at runtime, Prism automatically parses the function calls and builds the dependency graph.

• Intuitive logging: Prism automatically logs events for parsing the configuration files, compiling the tasks and creating the DAG, and executing the tasks. No configuration is required.

• Flexible CLI: Users can instantiate, compile, and run projects using a simple, but powerful command-line interface.

• “Batteries included”: Prism comes with all the essentials needed to get up and running quickly. Users can create and run their first DAG in less than 2 minutes.

• Integrations: Prism integrates with several tools that are popular in the data community, including Snowflake, Google BigQuery, Redshift, PySpark, and dbt. We're adding more integrations every day, so let us know what you'd like to see!

What is a Prism project?

At minimum, a Prism project must contain two things:

1. A directory of Python tasks called tasks, and

2. A prism_project.py file for managing your Python environment

Important: in previous versions of Prism, the tasks folder was called modules. Version v0.2.0 still allows your folder to be called modules, but you will see the following warning:

`modules` should be renamed to `tasks`...this will be an error in a future version of Prism

Here's how a typical Prism project is structured.

prism_project/
  ├── data/
  ├── dev/
  │   └── dev.ipynb
  ├── output/
  ├── tasks/
  │   ├── extract.py
  │   ├── transform.py
  │   └── load.py
  ├── prism_project.py
  ├── profile.yml
  └── triggers.yml

As you can see, projects can contain other folders, like a data folder, an output folder, and a profile.yml file for connecting your project to external sources (more on this later).

Guides: Jump right in

Follow our handy guides to get started on the basics as quickly as possible:

Getting StartedFundamentalsCLIAPI Reference

If you have any feedback about the product or the docs, please let us know!