Photo by Daniel Korpai on Unsplash

Install Airflow on MacOS (Guide)

For those looking to get started on their journey becoming an Airflow Developer please take a look at this course from Alexandra Abbas https://www.udemy.com/course/apache-airflow-course/

Follow these steps to install Airflow on Mac.

1. Open a Terminal window.
2. Navigate to your Desktop in your Terminal. We are going to create a working directory here but you can create anywhere else in your file system if you like.

cd Desktop

3. Create a working directory here called airflow-tutorial

mkdir airflow-tutorial

4. Change to airflow-tutorial directory in your Terminal.

cd airflow-tutorial

5. Create a virtual environment using coda (or any other tool). Install Python 3.7 in your virtual environment. I’m going to call the environment airflow-tutorial.

conda create --name airflow-tutorial python=3.7

6. Activate the virtual environment

conda activate airflow-tutorial

7. Print the absolute path to your working directory by typing pwd.

I get /Users/arpitrana/Desktop/airflow-tutorial but you might get something different. Copy this path

8. Set the path as the AIRFLOW_HOME environment variable. Note that you have to do this every time you open a new Terminal window and wish to use the Airflow CLI. Alternatively, you can set a permanent environment variable in your bash_profile.

export AIRFLOW_HOME=/Users/arpitrana/Desktop/airflow-tutorial

By default, airflow uses ~/airflow as it’s AIRFLOW_HOME directory. We can overwrite this by setting a different path. Airflow will initialize the airflow.cfg file here along with the logs folder. We’ll store our dags and plugins in this directory.

9. Install Airflow 1.10.10 + extras using pip

pip install apache-airflow[gcp,statsd,sentry]==1.10.10

If you’re using zsh like me then you need to put apache-airflow[gcp,statsd,sentry] in quotes as shown below.

pip install 'apache-airflow[gcp,statsd,sentry]'==1.10.10

Running Airflow on Mac os Locally

Before starting make sure you set your airflow home variable to the working directory. Validate this by typing echo $AIRFLOW_HOME .

We are going to initialize the airflow database using the following command. Make sure you are in the airflow-tutorial working directory.

airflow initdb

You can run the webserver by typing airflow webserver

Now create a new tab , Don’t forget to again put the Airflow Home variable while

export AIRFLOW_HOME =/Users/your_username/Desktop/airflow-tutorial 
cd Desktop/airflow-tutorial
conda activate airflow-tutorial
airflow scheduler

Now both our scheduler and webserver is running on localhost:8080