Can Next Scheduled Run Automatically Re-Run failed Task?

No and Yes.

No Airflow doesn't support it as a feature out of box

But you can use APIs and structure your dag in such a way to make this possible

Here is how I would go about it

  1. Add a task at the start of the pipeline

  2. In this task query the last-1th dagrun. The last dagrun is the current one

  3. Get all the failed task from that dagrun

  4. Clear their state

Idempotency

  • Ensure all your tasks are idempodent ie., doesn't matter how many times a task is ran it takes the same set of input and arrives at the same set of output

  • This means if a task fails inserting 100th row, it should be able to revert back and reinsert them when cleared


Last updated