Metadata
- Author: Jonathan Parks
- Full Title:: Data Quality at Airbnb
- Category:: 🗞️Articles
- Document Tags:: Data Quality,
- URL:: https://medium.com/airbnb-engineering/data-quality-at-airbnb-e582465f3ef7
- Finished date:: 2023-12-12
Highlights
leadership has set high expectations for data timeliness and quality, and increased focus on cost and compliance (View Highlight)
To ensure that we continue to meet these expectations, it was apparent that we needed to make sizable investments in our data (View Highlight)
Airbnb leadership signed off on the Data Quality initiative — a project of massive scale to rebuild the data warehouse from the ground up using new processes and technology. (View Highlight)
We also committed to a decentralized organizational structure composed of data engineering pods reporting into product teams (as opposed to a single centralized Data Eng org). This model ensures data engineers are aligned with the needs of consumers and the direction of product, while ensuring a critical mass of engineers (3 or more). Team size is important for providing mentorship/leadership opportunities, managing data operations, and smoothing over staffing gaps. (View Highlight)
To complement the distributed pods of data engineers, we founded a central data engineering team that develops data engineering standards, tooling, and best practices. The team also manages global datasets that don’t align well with any of the product teams. (View Highlight)
Tables must be normalized (within reason) and rely on as few dependencies as possible. Minerva does the heavy lifting to join across data models. (View Highlight)