Metadata

Highlights

  • What I encounter a lot is Redshift and Snowflake in the warehouse (more of the latter these days) supplemented by Databricks for Machine Learning (ML) and Spark support.
  • MLFlow to bring MLOps and model tracking into practice
  • but I still see most firms prefer a traditional Data Warehouse in their ecosystem as the center of data with Databricks as the big data compute and Machine Learning workhorse.
  • the DBT partnership
  • the Snowflake + Databricks partnership
  • the integration of installing UDFs into SnowFlake to embed custom routines in SQL seems a little clumsy compared to doing it in PySpark on Databricks — but I need to try it out a little before I can really be sure.
  • Snowpark like any new technology will take a few years to evolve to the point that it is as performant, stable, and robust as Spark on Databricks