Metadata
- Author: Iliana Iankoulova
- Full Title:: A Decade of Scaling (Real-Time) Analytics and Master Data at Picnic
- Category:: 🗞️Articles
- Document Tags:: Data Culture,
- URL:: https://blog.picnic.nl/a-decade-of-scaling-real-time-analytics-and-master-data-at-picnic-ae672f21ee0f?source=rss----12f9dce2953c---4
- Read date:: 2025-03-29
Highlights
At the time, a single team was responsible for everything — data modeling, warehousing, event streaming, master data management, and machine learning. That made sense early on, but it became a bottleneck. Urgent business demands would routinely derail long-term initiatives. To solve this, we split the team into specialized tech teams, each focused on a critical domain: Master Data, Event Streaming, Data Warehousing, and Machine Learning. Each team began small, with dedicated tech and product leadership, and scaled as the company grew. (View Highlight)
One of the most impactful shifts we made was democratizing the tools we had mastered as data engineers. We started by onboarding a small group of analysts — one per business area — with SQL. That group soon became experts in advanced analytics and helped train the next wave of analysts. Instead of creating a gap between analytics and engineering, we gave analysts access to the same toolset — Python, dbt, SQL, orchestration frameworks. They could now build backend models and frontend GUI using the same infrastructure engineers rely on. The impact was massive. Insights and operational automations that once took weeks were now delivered in days — or even hours. (View Highlight)