Triggers YML

Overview

Triggers are used to run specific functions after the success and/or failure of your project run. The triggers YML file points your project to the specific functions you would like to run.

Here is an example of a valid triggers YML and the accompanying prism_project.py. Assume the following project structure:

common/
  ├── utils.py
etl_project/
  ├── prism_project.py
  ├── tasks/
  │   ├── extract.py
  │   ├── transfrom.py
  │   └── load.py
  └── triggers.yml

prism_project.py

# etl_project/prism_project.py

...
# Triggers
TRIGGERS_YML_PATH = Path(__file__).parent / 'triggers.yml'
TRIGGERS = {
    'on_success': ["slack_trigger"],
}
...

triggers.yml

# etl_project/triggers.yml

include:
  - "{{ Path(__file__).parent.parent }}"

triggers:
  slack_trigger:
    type: function
    function: common.utils.slack_alert
    kwargs:
      token: "{{ env('SLACK_TOKEN'} }}"
      project_name: "{{ prism_project.NAME }}"

Let's break down this example.

• In prism_project.py, we see that the user wishes to run the slack_trigger trigger upon the project's success. This trigger is specified in the triggers.yml file in the project directory.

• In triggers.yml, we see that the slack_trigger trigger will run the function common.utils.slack_alert.

• This function takes two inputs: token (which the user passes using an environment variable), and project_name (which the user passes using the prism_project Jinja function.

• Notably, the common folder lives outside the user's project directory. How does the project access this directory? Via include. The paths specified in include are added to a project's sys.path (similar to SYS_PATH_CONF) at runtime.

  • The path listed in include is the parent directory of common/. Therefore, the project can access common.utils.slack_alert.

YML Structure

As we can see from the example above, the structure of the triggers YML is as follows:

# triggers.yml

include:
  - "{{ Path(__file__).parent }}"
  
triggers:
  <trigger name here>:
    type: function
    function: <import path to function>
    kwargs:
      arg1: value1

The trigger name you enter should match the trigger names you mention in prism_project.py. Each trigger name should have three nested keys:

• type: this should be the trigger type, and as of now, only type function are supported

• function: this should be the import path to your function. Make sure your project has access to this import path using include! Otherwise, you'll encounter an ModuleNotFoundError.

• kwargs: this is an optional set of key-value pairs that represent the functions keyword arguments.

More examples

Triggers are a fairly advanced Prism feature. Let's look at a couple more examples.

Trigger function lives within prism_project.py

If you want to define the trigger function themselves within prism_project.py, then you won't need to modify anything. Your project will automatically be able to import any module already within the project directory. So, your triggers.yml file can look like this:

# We don't need `include`, since the prism_project is always
# accessible

triggers:
  test_trigger:
    type: function
    function: prism_project.<function_name_here>

Important: we don't recommend defining trigger functions inside if prism_project.py. This file should primarily be used to configure and parametrize your project. We recommend defining your trigger functions in external modules. This also ensured reusability across different projects.

Trigger function lives within the project directory

Suppose your Prism project has the following structure:

prism_project/
  ├── prism_project.py
  ├── tasks/
  │   ├── extract.py
  │   ├── transfrom.py
  │   └── load.py
  ├── utils.py
  └── triggers.yml

Let's say your trigger function lives in utils.py. It still lives inside the project directory, just not inside prism_project.py. In this case, you still won't need to modify anything. Your project will automatically be able to import any module already within the project directory. So, your triggers.yml file can look like this:

# We don't need `include`, since the project directory is always
# accessible

triggers:
  test_trigger:
    type: function
    function: utils.<function_name_here>