Join TheLearningDev RSS Feed

Setting Up Apache Airflow For Local Development in Mac M1

How to setup airflow for local development, errors you might face and how to work around them?

As I always say start with the documentation. This documentation is all over the place, and you have to make a few decisions. Like whether to use breeze or virtualenv etc.,

My goto setup is to use the plain old virtualenv

Fork and Clone the repo

git clone <forked-repo-url>
cd airflow
git remote add upstream git@github.com:apache/airflow.git

Install Mysql

brew install mysql
brew services start mysql

Create Virtualenv

virtualenv -p python3 venv

Activate Virtualenv

source venv/bin/activate

Install required packages

pip install --upgrade -e ".[devel,google,postgres]"

I installed those 3 because I work with them predominantly. Every time I setup the project, I get an some error or the other

Some Red stuff you might receive

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
ERROR: Command errored out with exit status 1:
   command: /Users/bhavaniravi/projects/airflow/venv/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/f6/d24p66nx1rj669t5dbk6gg780000gn/T/pip-install-lc77g8re/pygraphviz_80c4808c56e0490cb23cf6b35d596a18/setup.py'"'"'; __file__='"'"'/private/var/folders/f6/d24p66nx1rj669t5dbk6gg780000gn/T/pip-install-lc77g8re/pygraphviz_80c4808c56e0490cb23cf6b35d596a18/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/f6/d24p66nx1rj669t5dbk6gg780000gn/T/pip-wheel-j8ki0jyn
       cwd: /private/var/folders/f6/d24p66nx1rj669t5dbk6gg780000gn/T/pip-install-lc77g8re/pygraphviz_80c4808c56e0490cb23cf6b35d596a18/
  Complete output (73 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-10.14-arm64-3.8
  creating build/lib.macosx-10.14-arm64-3.8/pygraphviz
  copying pygraphviz/scraper.py -> build/lib.macosx-10.14-arm64-3.8/pygraphviz
  copying pygraphviz/graphviz.py -> build/lib.macosx-10.14-arm64-3.8/pygraphviz
  copying pygraphviz/__init__.py -> build/lib.macosx-10.14-arm64-3.8/pygraphviz
  copying pygraphviz/agraph.py -> build/lib.macosx-10.14-arm64-3.8/pygraphviz
  copying pygraphviz/testing.py -> build/lib.macosx-10.14-arm64-3.8/pygraphviz
  creating build/lib.macosx-10.14-arm64-3.8/pygraphviz/tests
  copying pygraphviz/tests/test_unicode.py -> build/lib.macosx-10.14-arm64-3.8/pygraphviz/tests
  copying pygraphviz/tests/test_scraper.py -> build/lib.macosx-10.14-arm64-3.8/pygraphviz/tests
  copying pygraphviz/tests/test_readwrite.py -> build/lib.macosx-10.14-arm64-3.8/pygraphviz/tests
  copying pygraphviz/tests/test_string.py -> build/lib.macosx-10.14-arm64-3.8/pygraphviz/tests
  copying pygraphviz/tests/__init__.py -> build/lib.macosx-10.14-arm64-3.8/pygraphviz/tests
  copying pygraphviz/tests/test_html.py -> build/lib.macosx-10.14-arm64-3.8/pygraphviz/tests
  copying pygraphviz/tests/test_node_attributes.py -> build/lib.macosx-10.14-arm64-3.8/pygraphviz/tests
  copying pygraphviz/tests/test_drawing.py -> build/lib.macosx-10.14-arm64-3.8/pygraphviz/tests
  copying pygraphviz/tests/test_repr_mimebundle.py -> build/lib.macosx-10.14-arm64-3.8/pygraphviz/tests
  copying pygraphviz/tests/test_subgraph.py -> build/lib.macosx-10.14-arm64-3.8/pygraphviz/tests
  copying pygraphviz/tests/test_close.py -> build/lib.macosx-10.14-arm64-3.8/pygraphviz/tests
  copying pygraphviz/tests/test_edge_attributes.py -> build/lib.macosx-10.14-arm64-3.8/pygraphviz/tests
  copying pygraphviz/tests/test_clear.py -> build/lib.macosx-10.14-arm64-3.8/pygraphviz/tests
  copying pygraphviz/tests/test_layout.py -> build/lib.macosx-10.14-arm64-3.8/pygraphviz/tests
  copying pygraphviz/tests/test_attribute_defaults.py -> build/lib.macosx-10.14-arm64-3.8/pygraphviz/tests
  copying pygraphviz/tests/test_graph.py -> build/lib.macosx-10.14-arm64-3.8/pygraphviz/tests
  running egg_info
  writing pygraphviz.egg-info/PKG-INFO
  writing dependency_links to pygraphviz.egg-info/dependency_links.txt
  writing top-level names to pygraphviz.egg-info/top_level.txt
  reading manifest file 'pygraphviz.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no files found matching '*.png' under directory 'doc'
  warning: no files found matching '*.txt' under directory 'doc'
  warning: no files found matching '*.css' under directory 'doc'
  warning: no previously-included files matching '*~' found anywhere in distribution
  warning: no previously-included files matching '*.pyc' found anywhere in distribution
  warning: no previously-included files matching '.svn' found anywhere in distribution
  no previously-included directories found matching 'doc/build'
  adding license file 'LICENSE'
  writing manifest file 'pygraphviz.egg-info/SOURCES.txt'
  copying pygraphviz/graphviz.i -> build/lib.macosx-10.14-arm64-3.8/pygraphviz
  copying pygraphviz/graphviz_wrap.c -> build/lib.macosx-10.14-arm64-3.8/pygraphviz
  running build_ext
  building 'pygraphviz._graphviz' extension
  creating build/temp.macosx-10.14-arm64-3.8
  creating build/temp.macosx-10.14-arm64-3.8/pygraphviz
  clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -iwithsysroot/System/Library/Frameworks/System.framework/PrivateHeaders -iwithsysroot/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/Headers -arch arm64 -arch x86_64 -Werror=implicit-function-declaration -DSWIG_PYTHON_STRICT_BYTE_CHAR -I/Users/bhavaniravi/projects/airflow/venv/include -I/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8 -c pygraphviz/graphviz_wrap.c -o build/temp.macosx-10.14-arm64-3.8/pygraphviz/graphviz_wrap.o
  pygraphviz/graphviz_wrap.c:1756:7: warning: 'tp_print' is deprecated [-Wdeprecated-declarations]
        0,                                    /* tp_print */
        ^
  /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8/cpython/object.h:260:5: note: 'tp_print' has been explicitly marked deprecated here
      Py_DEPRECATED(3.8) int (*tp_print)(PyObject *, FILE *, int);
      ^
  /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8/pyport.h:515:54: note: expanded from macro 'Py_DEPRECATED'
  #define Py_DEPRECATED(VERSION_UNUSED) __attribute__((__deprecated__))
                                                       ^
  pygraphviz/graphviz_wrap.c:1923:7: warning: 'tp_print' is deprecated [-Wdeprecated-declarations]
        0,                                    /* tp_print */
        ^
  /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8/cpython/object.h:260:5: note: 'tp_print' has been explicitly marked deprecated here
      Py_DEPRECATED(3.8) int (*tp_print)(PyObject *, FILE *, int);
      ^
  /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/include/python3.8/pyport.h:515:54: note: expanded from macro 'Py_DEPRECATED'
  #define Py_DEPRECATED(VERSION_UNUSED) __attribute__((__deprecated__))
                                                       ^
  pygraphviz/graphviz_wrap.c:2711:10: fatal error: 'graphviz/cgraph.h' file not found
  #include "graphviz/cgraph.h"
           ^~~~~~~~~~~~~~~~~~~
  2 warnings and 1 error generated.
  error: command 'clang' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for pygraphviz

To mitigate this error

brew install graphviz
brew info graphviz

You will get an installation path. Add that to this env variable

export GRPAHVIZ_DIR=/opt/homebrew/Cellar/graphviz/2.48.0

Pre-commit

pre-commit install

Running Mypy

mypy <filepath>

Running Pytest

pytest <path-to-file>::function_name
pytest <path-to-file>::className::func_name
PreviousHow to Send Email Alerts from Airflow?NextHow to Copy/Move S3 Files With Apache Airflow

Last updated