# EC2 ## Prerequisites EC2 agents rely on the `boto3` and `botocore` packages. These require you to set up authentication credentials for your AWS account. More information can be found [here](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/quickstart.html#configuration). In addition, you'll need to make sure your account has the necessary privileges to create EC2 instances, security groups, and key-pairs. Contact your account administrator to help out with this. ## Configuring your EC2 agent Docker agents are configured using a YML file: ```yaml # ec2_agent.yml agent: type: ec2 instance_type: # instance type, e.g., t2.micro requirements: # path to requirements.txt, relative to this file env: : 'env var 1 value' ``` The YML file has a single top-level key, `agent`. This key contains all the configurations needed for your agent. Specifically: * `type`: this will always be `ec2` for EC2 agents * `instance_type`: the EC2 instance type. Note that different instance types are optimized for different use cases. The full list of EC2 instance types can be found [here](https://aws.amazon.com/ec2/instance-types/). * `requirements`: path to the `requirements.txt` file. This path must be *relative to the path of the agent YML file.* * `env`: `{key,value}` pairs representing environment variables to add to your Docker image ## Example agent For the remainder of this guide, let's assume that we have the following project. ``` etl_project/ ├── prism_project.py ├── modules/ │ ├── extract.py │ ├── transfrom.py │ └── load.py ├── ec2_agent.yml ├── requirements.txt └── triggers.yml ``` And, let's assume that we defined our `ec2_agent.yml` as follows: ```yaml # ec2_agent.yml agent: type: ec2 instance_type: m7g.medium requirements: ./requirements.txt env: AWS_ACCESS_KEY_ID: "{{ env('AWS_ACCESS_KEY_ID') }}" AWS_SECRET_ACCESS_KEY: "{{ env('AWS_SECRET_ACCESS_KEY') }}" ``` Note the way we specified the `requirements` key. We can do this because `ec2_agent.yml` and `requirements.txt` are in the same directory. ## Creating your EC2 agent You can build your EC2 agent using the command: `prism agent apply -f ` (note: logs are truncated for visibility): ``` $ prism agent apply -f ./ec2_agent.yml -------------------------------------------------------------------------------- | INFO | Creating agent... etl_project_ec2_agent[build] | Created key pair etl_project_ec2_agent etl_project_ec2_agent[build] | Created security group with ID sg-XXXXXXXXXXXXXXXXX in VPC vpc-XXXXXXXXXXXXXXXXX etl_project_ec2_agent[build] | Created EC2 instance with ID i-XXXXXXXXXXXXXXXXX etl_project_ec2_agent[build] | Instance i-XXXXXXXXXXXXXXXXX is `pending`... checking again in 5 seconds etl_project_ec2_agent[build] | Instance i-XXXXXXXXXXXXXXXXX is `pending`... checking again in 5 seconds etl_project_ec2_agent[build] | Instance i-XXXXXXXXXXXXXXXXX is `pending`... checking again in 5 seconds etl_project_ec2_agent[build] | Instance i-XXXXXXXXXXXXXXXXX is `pending`... checking again in 5 seconds etl_project_ec2_agent[build] | Instance i-XXXXXXXXXXXXXXXXX is `pending`... checking again in 5 seconds etl_project_ec2_agent[build] | Instance i-XXXXXXXXXXXXXXXXX is `pending`... checking again in 5 seconds etl_project_ec2_agent[build] | Instance i-XXXXXXXXXXXXXXXXX is `pending`... checking again in 5 seconds etl_project_ec2_agent[build] | ssh: connect to host ec2-XX-XXX-XXX-XXX.compute-1.amazonaws.com port 22: Connection refused etl_project_ec2_agent[build] | SSH connection failed. Retrying in 5 seconds... The authenticity of host 'ec2-XX-XXX-XXX-XXX.compute-1.amazonaws.com (XX.XXX.XXX.XXX)' can't be established. ED25519 key fingerprint is SHA256:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX. This key is not known by any other names Are you sure you want to continue connecting (yes/no/[fingerprint])? yes etl_project_ec2_agent[build] | Warning: Permanently added 'ec2-XX-XXX-XXX-XXX.compute-1.amazonaws.com' (ED25519) to the list of known hosts. etl_project_ec2_agent[build] | SSH connection succeeded! etl_project_ec2_agent[build] | , #_ etl_project_ec2_agent[build] | ~\_ ####_ Amazon Linux 2023 etl_project_ec2_agent[build] | ~~ \_#####\ etl_project_ec2_agent[build] | ~~ \###| etl_project_ec2_agent[build] | ~~ \#/ ___ https://aws.amazon.com/linux/amazon-linux-2023 etl_project_ec2_agent[build] | ~~ V~' '-> etl_project_ec2_agent[build] | ~~~ / etl_project_ec2_agent[build] | ~~._. _/ etl_project_ec2_agent[build] | _/ _/ etl_project_ec2_agent[build] | _/m/' etl_project_ec2_agent[build] | Requirement already satisfied: pip in ./.venv/etl_project/lib/python3.9/site-packages (21.3.1) etl_project_ec2_agent[build] | ... etl_project_ec2_agent[build] | ... etl_project_ec2_agent[build] | ... etl_project_ec2_agent[build] | Successfully installed ... etl_project_ec2_agent[build] | Updating remote project and file paths etl_project_ec2_agent[build] | Done updating remote project and file paths -------------------------------------------------------------------------------- ``` If you check your EC2 console, you'll see an EC2 instance called `etl_project_ec2_agent`! ## Running your Docker agent There are two commands you can use to run your Docker agent.
prism agent run The `prism agent run` command streams your project onto your EC2 instance: ``` $ prism agent run -f ./ec2_agent.yml -------------------------------------------------------------------------------- | INFO | Streaming agent logs... etl_project_ec2_agent[run] | | INFO | Running with prism v0.1.9rc1... etl_project_ec2_agent[run] | | INFO | Found project directory at /etl_project etl_project_ec2_agent[run] | etl_project_ec2_agent[run] | | INFO | RUNNING EVENT 'parsing prism_project.py'................................................ [RUN] etl_project_ec2_agent[run] | | INFO | FINISHED EVENT 'parsing prism_project.py'............................................... [DONE in 1.42s] etl_project_ec2_agent[run] | | INFO | RUNNING EVENT 'module DAG'.............................................................. [RUN] etl_project_ec2_agent[run] | | INFO | FINISHED EVENT 'module DAG'............................................................. [DONE in 0.01s] etl_project_ec2_agent[run] | | INFO | RUNNING EVENT 'creating pipeline, DAG executor'......................................... [RUN] etl_project_ec2_agent[run] | | INFO | FINISHED EVENT 'creating pipeline, DAG executor'........................................ [DONE in 0.01s] etl_project_ec2_agent[run] | etl_project_ec2_agent[run] | | INFO | ======================= tasks 'finicky-macaw-JaEvjyMWtb' ======================= etl_project_ec2_agent[run] | | INFO | 1 of 3 RUNNING EVENT 'extract.py'....................................................... [RUN] etl_project_ec2_agent[run] | | INFO | 1 of 3 FINISHED EVENT 'extract.py'...................................................... [DONE in 120.01s] etl_project_ec2_agent[run] | | INFO | 2 of 3 RUNNING EVENT 'transform.py'..................................................... [RUN] etl_project_ec2_agent[run] | | INFO | 2 of 3 FINISHED EVENT 'transform.py'.................................................... [DONE in 791.38s] etl_project_ec2_agent[run] | | INFO | 3 of 3 RUNNING EVENT 'load.py'.......................................................... [RUN] etl_project_ec2_agent[run] | | INFO | 3 of 3 FINISHED EVENT 'load.py'......................................................... [DONE in 7.84s] | INFO | Done streaming agent logs... -------------------------------------------------------------------------------- ```
prism agent build The `prism agent build` command updates your project files within your EC2 instance and executes the project onto it. ``` $ prism agent build -f ./ec2_agent.yml -------------------------------------------------------------------------------- | INFO | Creating agent... etl_project_ec2_agent[build] | Created key pair etl_project_ec2_agent etl_project_ec2_agent[build] | Created security group with ID sg-XXXXXXXXXXXXXXXXX in VPC vpc-XXXXXXXXXXXXXXXXX etl_project_ec2_agent[build] | Created EC2 instance with ID i-XXXXXXXXXXXXXXXXX etl_project_ec2_agent[build] | Instance i-XXXXXXXXXXXXXXXXX is `pending`... checking again in 5 seconds etl_project_ec2_agent[build] | Instance i-XXXXXXXXXXXXXXXXX is `pending`... checking again in 5 seconds etl_project_ec2_agent[build] | Instance i-XXXXXXXXXXXXXXXXX is `pending`... checking again in 5 seconds etl_project_ec2_agent[build] | Instance i-XXXXXXXXXXXXXXXXX is `pending`... checking again in 5 seconds etl_project_ec2_agent[build] | Instance i-XXXXXXXXXXXXXXXXX is `pending`... checking again in 5 seconds etl_project_ec2_agent[build] | Instance i-XXXXXXXXXXXXXXXXX is `pending`... checking again in 5 seconds etl_project_ec2_agent[build] | Instance i-XXXXXXXXXXXXXXXXX is `pending`... checking again in 5 seconds etl_project_ec2_agent[build] | ssh: connect to host ec2-XX-XXX-XXX-XXX.compute-1.amazonaws.com port 22: Connection refused etl_project_ec2_agent[build] | SSH connection failed. Retrying in 5 seconds... The authenticity of host 'ec2-XX-XXX-XXX-XXX.compute-1.amazonaws.com (XX.XXX.XXX.XXX)' can't be established. ED25519 key fingerprint is SHA256:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX. This key is not known by any other names Are you sure you want to continue connecting (yes/no/[fingerprint])? yes etl_project_ec2_agent[build] | Warning: Permanently added 'ec2-XX-XXX-XXX-XXX.compute-1.amazonaws.com' (ED25519) to the list of known hosts. etl_project_ec2_agent[build] | SSH connection succeeded! etl_project_ec2_agent[build] | , #_ etl_project_ec2_agent[build] | ~\_ ####_ Amazon Linux 2023 etl_project_ec2_agent[build] | ~~ \_#####\ etl_project_ec2_agent[build] | ~~ \###| etl_project_ec2_agent[build] | ~~ \#/ ___ https://aws.amazon.com/linux/amazon-linux-2023 etl_project_ec2_agent[build] | ~~ V~' '-> etl_project_ec2_agent[build] | ~~~ / etl_project_ec2_agent[build] | ~~._. _/ etl_project_ec2_agent[build] | _/ _/ etl_project_ec2_agent[build] | _/m/' etl_project_ec2_agent[build] | Requirement already satisfied: pip in ./.venv/etl_project/lib/python3.9/site-packages (21.3.1) etl_project_ec2_agent[build] | ... etl_project_ec2_agent[build] | ... etl_project_ec2_agent[build] | ... etl_project_ec2_agent[build] | Successfully installed ... etl_project_ec2_agent[build] | Updating remote project and file paths etl_project_ec2_agent[build] | Done updating remote project and file paths | INFO | Streaming agent logs... etl_project_ec2_agent[run] | | INFO | Running with prism v0.1.9rc1... etl_project_ec2_agent[run] | | INFO | Found project directory at /etl_project etl_project_ec2_agent[run] | etl_project_ec2_agent[run] | | INFO | RUNNING EVENT 'parsing prism_project.py'................................................ [RUN] etl_project_ec2_agent[run] | | INFO | FINISHED EVENT 'parsing prism_project.py'............................................... [DONE in 1.42s] etl_project_ec2_agent[run] | | INFO | RUNNING EVENT 'module DAG'.............................................................. [RUN] etl_project_ec2_agent[run] | | INFO | FINISHED EVENT 'module DAG'............................................................. [DONE in 0.01s] etl_project_ec2_agent[run] | | INFO | RUNNING EVENT 'creating pipeline, DAG executor'......................................... [RUN] etl_project_ec2_agent[run] | | INFO | FINISHED EVENT 'creating pipeline, DAG executor'........................................ [DONE in 0.01s] etl_project_ec2_agent[run] | etl_project_ec2_agent[run] | | INFO | ======================= tasks 'finicky-macaw-JaEvjyMWtb' ======================= etl_project_ec2_agent[run] | | INFO | 1 of 3 RUNNING EVENT 'extract.py'....................................................... [RUN] etl_project_ec2_agent[run] | | INFO | 1 of 3 FINISHED EVENT 'extract.py'...................................................... [DONE in 120.01s] etl_project_ec2_agent[run] | | INFO | 2 of 3 RUNNING EVENT 'transform.py'..................................................... [RUN] etl_project_ec2_agent[run] | | INFO | 2 of 3 FINISHED EVENT 'transform.py'.................................................... [DONE in 791.38s] etl_project_ec2_agent[run] | | INFO | 3 of 3 RUNNING EVENT 'load.py'.......................................................... [RUN] etl_project_ec2_agent[run] | | INFO | 3 of 3 FINISHED EVENT 'load.py'......................................................... [DONE in 7.84s] | INFO | Done streaming agent logs... -------------------------------------------------------------------------------- ```