Can Next Scheduled Run Automatically Re-Run failed Task?
No and Yes.
No Airflow doesn't support it as a feature out of box
But you can use APIs and structure your dag in such a way to make this possible
Here is how I would go about it
  1. 1.
    Add a task at the start of the pipeline
  2. 2.
    In this task query the last-1th dagrun. The last dagrun is the current one
  3. 3.
    Get all the failed task from that dagrun
  4. 4.
    Clear their state
Idempotency
  • Ensure all your tasks are idempodent ie., doesn't matter how many times a task is ran it takes the same set of input and arrives at the same set of output
  • This means if a task fails inserting 100th row, it should be able to revert back and reinsert them when cleared
Copy link