Using Airflow Providers With KubernetesPodOperator
When you're constrained to use KubernetesPodOperator you will be at a disadvantage of using Airflow providers. There is something you can do about it. Using Airflow provider like any other library.
- 1.Install airflow and associated providers you need in the Docker image, KubernetesPodOperator will be using.
- 2.Create a python script that will be consumed by KubernetesPodOperator
- 3.Inside the script, import the provider operator and call its execute method
from airflow.providers.amazon.aws.operators.s3 import S3CreateObjectOperator
create_object = S3CreateObjectOperator(
task_id="s3_create_object",
s3_bucket=BUCKET_NAME,
s3_key=KEY,
data=DATA,
replace=True,
)
create_object.execute(dict())
Newsletter embed