- Tags:: #🗣️Talks , [[Data Engineering]]
- Author:: [[Maxime Beauchemin]]
- Link:: [Functional Data Engineering - A Set of Best Practices | Lyft - YouTube](https://www.youtube.com/watch?v=4Spo2QRTz1k)
- Source date:: [[2018-05-28]]
- Read date:: [[2022-06-27]]
Comes with an article version: [Functional Data Engineering — a modern paradigm for batch data processing | by Maxime Beauchemin | Medium](https://maximebeauchemin.medium.com/functional-data-engineering-a-modern-paradigm-for-batch-data-processing-2327ec32c42a)
The most surprising thing of the talk is his radical approach towards [[Slowly Changing Dimensions]]: no type-2. Instead, snapshot all data! (Storage and compute is cheap. Engineering time is expensive and this is very easy to reason about)
![[Pasted image 20220627181250.png|500]]
So you end up with a partition lineage:
![[Pasted image 20220627181400.png|500]]
And, in the passing, a very (IMHO) controversial statement:
>you had small teams of highly specialized data professionals building the warehouse for the company. I think that's not true anymore (...) everyone is welcome to use and create and mute and change and shape the future of the data warehouse (33:15)
*Socorro*, Maxime, the [[Tragedy of the Commons]]!
## Past dependencies
![[Pasted image 20230706065033.png]]
>Given that backfills are common and that past dependencies lead to high-depth DAGs with limited parallelization, it’s a good practice to avoid modeling using past-dependencies whenever possible.