KubernetesPodOperator vs. KubernetesExecutor
KubernetesExecutor
KubernetesExecutor is where Airflow spins up a new pod to run an Airflow task. Unlike Celery executor the advantage is you don't have a bunch of workers always running. KubernetesExecutor is on-demand thereby reducing cost.\
One dowside of kubernetes executor can be the time it takes to spin up the pod but compared to the advantages it can be close to null
KubernetesPodOperator
When using KubernetesPodOperator
, all the business logic and it's associated code resides in a docker image. During execution, airflow spins up a worker pod, which pulls the mentioned docker image and executes the respective command.
Pros
Works well across cross-functional teams
Single airflow instance can be shared across teams without hassle
Supports different languages and frameworks
Decouples DAG and the business logic
Last updated