## Metadata - Author: [[matt-arderne|Matt Arderne]] - Full Title:: The future history of Data Engineering - Category:: #🗞️Articles - Document Tags:: [[Data engineering|Data engineering]], - URL:: https://groupby1.substack.com/p/data-engineering - Finished date:: [[2023-08-24]] ## Highlights > Most businesses' **data engineering** needs have been solved or will shortly be solved by managed services that 10 years ago would require endless and extensive self-built ETL pipelines, databases and tools. ([View Highlight](https://read.readwise.io/read/01h8j2d1rv9nrk4fqp0mpmpv5v)) > Most businesses' **data engineering** needs have been solved or will shortly be solved by managed services that 10 years ago would require endless and extensive self-built ETL pipelines, databases and tools. ([View Highlight](https://read.readwise.io/read/01h8j2d7s8whg207k9kqbr4w7h)) > Most businesses' **data engineering** needs have been solved or will shortly be solved by managed services that 10 years ago would require endless and extensive self-built ETL pipelines, databases and tools. ([View Highlight](https://read.readwise.io/read/01h8j2d95d6msnag3072n7brgk)) > Most businesses' **data engineering** needs have been solved or will shortly be solved by managed services that 10 years ago would require endless and extensive self-built ETL pipelines, databases and tools. ([View Highlight](https://read.readwise.io/read/01h8j2dap8r2rv5ka132e6d1cc)) > Most businesses' **data engineering** needs have been solved or will shortly be solved by managed services that 10 years ago would require endless and extensive self-built ETL pipelines, databases and tools. ([View Highlight](https://read.readwise.io/read/01h8j2dc4ytpg2hee65xpbq616)) But most people who will build the latter will still call themselves Data Engineers > Most businesses' **data engineering** needs have been solved or will shortly be solved by managed services that 10 years ago would require endless and extensive self-built ETL pipelines, databases and tools. > For the exceeding majority of businesses, this means they can and should focus on building capacity for business logic, analysis and predictions instead of data engineering. ([View Highlight](https://read.readwise.io/read/01h8j2dzpf60m9ahx80czpjdyt)) > Eventually, analytics engineering could face the same turn of the tide. When the tooling gets so good that the **team is composed entirely of analysts and product people, and no contriving engineers.** > In the same way that structural engineers are only required when building on quicksand, data engineers are only required when building upon a dataswamp. As the tooling gets better, so do the foundations stabilise. ([View Highlight](https://read.readwise.io/read/01h8j2pyfzbqgvbanzptzrkt1c)) > The implication for engineers whose work is now easier is the following: > **Either you move in the direction of the new business problem.** > **Or you move to a new business that still has the old problem.** > **Or you specialise further until you find another domain to play in, and wait for the tide to turn again.** ([View Highlight](https://read.readwise.io/read/01h8j2sph0mnenwcqkm4ybp1pq)) > DPE just means that you are the Tech Lead of the Analytics Engineering. ([View Highlight](https://read.readwise.io/read/01h8j2wkk6bw058265xpdkpn8m))