Skip to content
Hallway
0:00 / 0:00

#apacheairflow#docker#docker-compose

Running Airflow 2.0 with Docker in 5 mins

Data with Marc

41K subscribers • 4 videos on Hallway

YouTube
  • Recommended by 3 people: 3 in engineering communities.
  • Presented by Marc Lamberti, Airflow Instructor
  • May be out of date. The tools, versions or practices it shows have likely moved on.

Beginner. No background in the topic needed.

194K views5 years ago3 recommendations

Databases and dataCloud and DevOpsData engineering and warehousesDocker and containers

Marc Lamberti demonstrates how to quickly set up and run Apache Airflow 2.0 locally using Docker Compose.

Airflow 2.0 is out! How it works, what are the new features, what can do with your DAGs, to answer all those questions, you need to run Airflow 2.0. What is the easiest and fastest way to do it? By using Docker! Let's discover how to run Apache Airflow 2.0 with the CeleryExecutor locally by using Docker! 👍 Smash the like button to become an Airflow Super Hero! ❤️ Subscribe to my channel to become a master of Airflow 🏆 Take my course : udemy.com/course/the-ultimate-hands-on-course-to-master-apac… to join the legends of Airflow 🚨 My Patreon: patreon.com/marclamberti to support my work and be friend for life The docker-compose file: airflow.apache.org/docs/apache-airflow/stable/docker-compose…

Why it's here

3 people recommend this

What people say. Recommended as an easy-to-follow guide for setting up Airflow locally with Docker.

u/surf_ocean_beachin r/dataengineering • Recommends

…# https://preview.redd.it/dhy74eykhp091.png?width=1039&format=png&auto=webp&s=cbed0caf7e89198aa0d68a96a254fc846ba13212 # Data Dashboard https://preview.redd.it/896bizwnhp091.png?width=627&format=png&auto=webp&s=86cadc6e6ab8fa594e49e0a3fca56df2d9c72740 # Learning Resources Airflow Basics: \Airflow DAG: Coding your first DAG for Beginners\([video \Running Airflow 2.0 with Docker in 5 mins\([video S3 Basics: \Setting Up Airflow Tasks To Connect Postgres And S3\([video \How to Upload files to AWS S3 using Python and Boto3\([video \Download files from S3\(https://www.stackvidhya.com/download-files-from-s3-using-boto3/) Docker Basics: \[Docker Tutorial for Beginners…
178View on Reddit

u/marclambertiin r/dataengineering • Recommends

Running Airflow 2.0 with the CeleryExecutor and Docker in 5 mins
3View on Reddit

u/snip3r77in r/dataengineering • Recommends

cloud airflow setup for personal project I managed to deploy docker postgres persists mode locally using WSL2/Ubuntu and managed to ETL to postgres too. But this task requires hourly fetch from API and ingest. I know I can do via crontab but I want to use airflow. I think at most I will use 2 dags. 1 for raw and 2nd to transformed 1. In my previous company I was using Airflow but not exactly doing the setup per se. I want to use cloud as I want to learn installing and using it. So what's my next step? Should I use docker to install airflow locally first and ensure it's working before I spin a micro EC2 at AWS( Sorry I can't afford managed airflow ). If I were to use docker installation, h…
3View on Reddit