your job as a data engineer will likely be spent on more strategic tasks rather than programming the nth ELT. (View Highlight)
Zack Wilson, Staff Data Engineer at Airbnb, predicts that in 2023 streaming data engineering jobs will account for 15-20% of all data engineers jobs and pay the most. (View Highlight)
Much of this ties into Functional Data Engineering, which is vital for data engineering. We can reduce side effects by slicing a DAG into functional tasks, as each function has a defined input and output. Each task can be written, tested, reasoned about, and debugged in isolation without understanding the external context or history of events surrounding their execution (View Highlight)
We may also see the emergence of specialized roles at the opposite end of the spectrum, where software engineering and data engineering intersect. In the future, software engineers may need to be well-versed in data engineering with the advent of streaming and event-driven architectures. (View Highlight)
DataOps is a method of working that facilitates communication and collaboration between data engineers, data scientists, and other data professionals to avoid silos. (View Highlight)
DataOps combines the best parts of Lean, Product Thinking, Agile, and DevOps and applies them to data management.
• Product Thinking: Seeing data as a product provides the best value to customers.
• Lean: Identifying the value of a data product helps eliminate waste and be more efficient.
• Agile: Embracing iterative product development and Minimum Viable Products (MVP) enables quick feedback from stakeholders.
• DevOps: Applying software engineering practices focusing on CI/CD, monitoring, observability, etc. (View Highlight)
your job as a data engineer will likely be spent on more strategic tasks rather than programming the nth ELT. (View Highlight)
Zack Wilson, Staff Data Engineer at Airbnb, predicts that in 2023 streaming data engineering jobs will account for 15-20% of all data engineers jobs and pay the most. (View Highlight)
Much of this ties into Functional Data Engineering, which is vital for data engineering. We can reduce side effects by slicing a DAG into functional tasks, as each function has a defined input and output. Each task can be written, tested, reasoned about, and debugged in isolation without understanding the external context or history of events surrounding their execution (View Highlight)
We may also see the emergence of specialized roles at the opposite end of the spectrum, where software engineering and data engineering intersect. In the future, software engineers may need to be well-versed in data engineering with the advent of streaming and event-driven architectures. (View Highlight)
DataOps is a method of working that facilitates communication and collaboration between data engineers, data scientists, and other data professionals to avoid silos. (View Highlight)
DataOps combines the best parts of Lean, Product Thinking, Agile, and DevOps and applies them to data management.
• Product Thinking: Seeing data as a product provides the best value to customers.
• Lean: Identifying the value of a data product helps eliminate waste and be more efficient.
• Agile: Embracing iterative product development and Minimum Viable Products (MVP) enables quick feedback from stakeholders.
• DevOps: Applying software engineering practices focusing on CI/CD, monitoring, observability, etc. (View Highlight)
