Metadata
- URL:: https://benn.substack.com/p/down-with-the-dag
- Author:: Benn Stancil
- Publisher:: benn.substack
- Published Date:: 2022-08-12
- Tags:: Workflow Orchestrators
Highlights
- Unfortunately, we’ve built most of our data pipelines using the latter model—we want the updated datasets or fresh dashboards at the end of our lineage graph, but define our schedules around what’s at the beginning of it.
Finally, Airflow released Data-aware scheduling — Airflow Documentation on 2022-09-01:
Transclude of Pasted-Image-20221209071022
and Dagster its Declarative Scheduling For Data Assets on 2022-12-08.