Skip to content
0:00 / 0:00

#awsglue#awsstepfunctions#iam

Orchestrate Glue Jobs With Step Functions

DataEng Uncomplicated

23K subscribers • 1 video on Hallway

YouTube
  • Recommended by 1 person: 1 on Stack Overflow and its sister sites.

Intermediate. Assumes working knowledge.

22K views4 years ago1 recommendation

Cloud and DevOpsDatabases and dataAWSData engineering and warehouses

A step-by-step tutorial on configuring AWS Step Functions to orchestrate Glue jobs, including setting up the necessary IAM roles and permissions.

This is a step-by-step tutorial on how to create a step function to orchestrate a single or multiple glue jobs and configure the I am role. #aws #awsglue #stepfunctions IAM Permission Link: docs.aws.amazon.com/step-functions/latest/dg/glue-iam.html

Why it's here

1 person recommends this

What people say. Recommended as a guide for configuring IAM permissions and callbacks when orchestrating AWS Glue jobs with Step Functions.

Geocoderon Stack Overflow • Recommends

On “aws glue job dependency in step function”

We need to first make sure our IAM Step Function Role has the ability to trigger our glue job to run as well as get a "call back" from the aws glue job when the job has been completed or failed. The policy for your step function should include this: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "glue:StartJobRun", "glue:GetJobRun", "glue:GetJobRuns", "glue:BatchStopJobRun" ], "Resource": "*" } ] } 2. Now we can add the "StartJobRun" Step to trigger our Glue job. We need to enable "wait for job to complete" which appears in the visual workflow editor. This will make ensure that our step function will wait for this task to be completed before go…