Here are some numbers to put our current scale into perspective: * The DWH has 120 (micro)service sources. * Some 1000 ETL/ELT jobs run per day to capture and aggregate data. * About 50% of employees at our HQ outside of tech roles have SQL skills. * A Business Analyst runs, on average, 300 SQL queries per day. * Our DWH contains 15,000 tables, 247,000 fields, and 121 billion rows. Just nine Data Engineers support all of this!
We use the DV only as a back-end DWH layer. Combining the best of both worlds has proven to be extremely valuable for Picnic. We use DV v2.0 with key hashes, enabling us to easily capture the state of entities (hubs) and relationships (links) over time in satellite tables.