Can Next Scheduled Run Automatically Re-Run failed Task?
No and Yes.
No Airflow doesn't support it as a feature out of box
But you can use APIs and structure your dag in such a way to make this possible
Here is how I would go about it
- 1.Add a task at the start of the pipeline
- 2.In this task query the last-1th dagrun. The last dagrun is the current one
- 3.Get all the failed task from that dagrun
- 4.Clear their state
Idempotency
- Ensure all your tasks are idempodent ie., doesn't matter how many times a task is ran it takes the same set of input and arrives at the same set of output
- This means if a task fails inserting 100th row, it should be able to revert back and reinsert them when cleared
Newsletter embed