Targets

There are two fundamental building blocks to a Prism project: tasks and targets.

What are targets?

The second fundamental building block to Prism projects are targets.

Targets enable you to cache the results of your tasks. Put differently, targets are used to store the results of your task at an external location (e.g., a CSV on your local machine, a table in your data warehouse, a file in an S3 bucket, and so on). In doing so, they prevent repetitive and costly task re-runs.

For example, let's say you have a project with two tasks: long, and short, and that short depends on the output of long. Based on prior runs, we know that long takes 10 minutes to execute. If we don't want to re-run long every time we make updates to short, we can specify a target for long to save its output to an external location for easy access.

How do you use targets?

To specify a target for a task, use the prism.decorators.target decorator. This decorator takes two required keyword arguments:

• type: a valid PrismTarget instance. This specifies how the object should be saved (e.g., as a .txt file, as a .csv file. etc.)

• loc: a string or pathlib.Path object denoting where the object should be saved

In addition, you can add additional keyword arguments to customize the target's saving behavior (e.g., removing the index from CSVs when saving a Pandas DataFrame)

Incorporating targets into tasks

For class-based tasks, you directly decorate the run function. For function-based tasks, you place the target decorator call inside the targets keyword argument of the task decorator.

Don't worry if this sounds confusing, we'll show you exactly what this looks like.

Class-based tasks

For class-based tasks, you directly decorate the run function as follows:

# modules/hello_world.py

import prism.task
import prism.target
import prism.decorators

class HelloWorld(prism.task.PrismTask):
    
    @prism.decorators.target(
        type=prism.target.Txt, 
        loc="/Users/hello_world.txt", 
        **kwargs
    )
    def run(self, tasks, hooks):
        test_str = "Hello, world!"
        return test_str

Critical: specifying a target changes the output of the task. Usually, the task output is some sort of object (e.g., a DataFrame). However, targets change it to instead be the location where the object is stored. Put differently, targets cause the output of the task to be the loc parameter of the prism.decorators.target decorator.

In the example above, the output of task HelloWorld is the path "/Users/hello_world.txt". This will be reflected in downstream tasks.ref(...) calls:

# modules/second_task.py

import prism.task
import prism.target
import prism.decorators

class SecondTask(prism.task.PrismTask):

    def run(self, tasks, hooks):
        tasks.ref("hello_world.py")  # returns "/Users/hello_world.txt"

Function-based tasks

For function-based tasks, you still use the prism.decorators.target decorator, but you place these inside the targets keyword argument of the @task decorator. Here's what that looks like:

# modules/hello_world.py

from prism.decorators import task, target
import prism.target

@task(
    targets=[
        target(type=prism.target.Txt, loc="/Users/hello_world.txt", **kwargs)
    ]
)
def hello_world(tasks, hooks)
    test_str = "Hello, world!"
    return test_str

The arguments for the target decorator are the exact same as before; only its location has changed.

As with class-based tasks, calling tasks.ref("hello_world.py") in a downstream task will return the path "/Users/hello_world.txt".

Check out the API reference documentation for more information.

What kinds of targets are available?

There are several targets available out-of-the-box. These include Txt, NumpyTxt, PandasCsv, and PysparkParquet. We're always looking to add targets and improve the Prism functionality, so please let us know if there's a target you want us to include in the next update!

If the pre-defined targets are not sufficient for your use case, then you can define your own PrismTarget class. These classes are pretty simple. They have three attributes: obj (i.e., the output to save), loc (the location to save the output), and hooks (see here for more information). And, they have one method called save that specifies how obj should be saved to loc.

For reference, here is the full code for the prism.target.Txt class:

class Txt(PrismTarget):

    def save(self, **kwargs):
        with open(self.loc, "w") as f:
            f.write(self.obj, **kwargs)
        f.close()