Data Engineering Portfolio Project Part (2/5) - Setting Up Airflow Tasks To Connect Postgres And S3

Ben Rogojan shows how to set up Apache Airflow tasks to connect a Postgres database with AWS S3.

Data engineering projects can be a great way to show off your skills. But they can be hard to put together. The purpose of this series is to help you put together a data engineering project. For this video you will need to have an EC2 instance set up as well as an RDS Postgres instance and S3 on AWS. In addition we will use Airflow for managing our tasks. For the commands to set-up a connection between Postgres and S3. gist.github.com/bAcheron/e5def0429d79c9312fd1806912b3cbcc Video Breakdown 0:00 Intro 1:52 Setting Up Postgres to S3 Connection 5:20 Airflow Set Up 8:50 Airflow - Writing Your First Pipeline 13:26 Airflow - Setting Up A Postgres Connection If you enjoyed this video, check out some of my other top videos. What Skills Do Data Engineers Need? youtube.com/watch?v=LgSHaOvNodA Data Engineering Project Ideas youtu.be/LJkVvNWlO0g If you want to learn more about machine learning, check out DataCamps Machine Learining Course bit.ly/3BeLEml If you'd like to read up on my updates about the data field, then you can sign up for our newsletter here. seattledataguy.substack.com/​​ Or check out my blog theseattledataguy.com/ Tags: Data engineering projects, Data engineer project ideas, data project sources, data analytics project sources, data project portfolio _____________________________________________________________ Subscribe: youtube.com/channel/UCmLGJ3VYBcfRaWbP6JLJcpA?sub_confirmatio… _____________________________________________________________ About me: I have spent my career focused on all forms of data. I have focused on developing algorithms to detect fraud, reduce patient readmission and redesign insurance provider policy to help reduce the overall cost of healthcare. I have also helped develop analytics for marketing and IT operations in order to optimize limited resources such as employees and budget. I privately consults on data science and engineering problems both solo as well as with a company called Acheron Analytics. I have experience both working hands-on with technical problems as well as helping leadership teams develop strategies to maximize their data. *I do participate in affiliate programs, if a link has an "*" by it, then I may receive a small portion of the proceeds at no extra cost to you.