Columbia & Databricks
Columbia faced challenges with their legacy ETL and analytics systems, which could not support both batch and real-time use cases efficiently. This limitation hindered their ability to process data effectively for business and data team needs.
The goal was to migrate their data processing and analytics capabilities to a more scalable and cost-efficient platform. This required the integration of a system capable of handling high-performance ETL pipelines while providing reliable and real-time data access.
The EIM team at Columbia switched to Microsoft Azure and utilized Azure Databricks and Delta Lake. This move enabled them to build efficient ETL pipelines supporting both batch and real-time workloads. The Delta Lake provided secure, consistent, and performant data access.
Columbia achieved a 70% reduction in ETL pipeline creation time and a 48x improvement in processing times, reducing ETL workloads from 4 hours to 5 minutes. This increased data processing speed allowed for quicker insights and smarter business decisions.
Create Profile | Sign In
