Re-engineering efforts at Fidelity, CNN, and other companies have enabled faster access to real-time data. Experts share their strategies for better management. Organizations need a secure data pipeline to extract real-time analytics from workloads and deliver trusted data. However, data pipelines are becoming increasingly complex to manage. This article explores how leading companies are transforming their data pipelines using Snowflake’s Apache Iceberg and new solutions like Iceberg Tables to enhance efficiency and business outcomes.

The Complexity of Modern Data Pipelines

Modern data pipelines are essential for organizations to manage and analyze vast amounts of data in real-time. However, they are also increasingly complex, involving numerous processes and technologies to ensure data is processed, stored, and analyzed efficiently. This complexity necessitates robust solutions that can simplify pipeline management while maintaining flexibility and scalability.

Data pipelines are the backbone of any data-driven organization. They facilitate the flow of data from various sources to a destination where it can be analyzed and used to generate insights. Effective data pipelines are crucial for real-time analytics, enabling businesses to make timely decisions based on current information. However, managing these pipelines can be challenging due to the sheer volume of data and the need for real-time processing.

The Role of Snowflake in Transforming Data Pipelines

Companies such as Booking.com, Capital One, Fidelity, and CNN are re-engineering their data pipelines using Snowflake’s Apache Iceberg and a new solution, Iceberg Tables. These technologies include data lakehouses, data lakes, and data meshes, allowing IT leaders to simplify pipeline development and work with open data flexibly.

Snowflake’s Apache Iceberg and Iceberg Tables offer several benefits for data pipeline management. They enable organizations to handle large datasets efficiently, support various data formats, and ensure data consistency. These solutions also provide the flexibility to scale according to business needs, making them ideal for organizations looking to enhance their data management capabilities.

“With Iceberg, we can broaden our use cases for Snowflake as our open data lakehouse for machine learning, AI, business intelligence, and geospatial analysis — even for data stored externally,” said Thomas Davey, chief data officer for Booking.com.

Polaris Catalog: Enhancing Interoperability

Iceberg Tables, announced June 4 at the Snowflake Summit in San Francisco, comes on the heels of the recently announced Polaris Catalog, a vendor-neutral and fully open catalog implementation for Apache Iceberg. Polaris Catalog enables cross-engine interoperability, giving organizations more choice, flexibility, and control over their data.

Organizations can get started running Polaris Catalog hosted in Snowflake’s AI Data Cloud or using containers within their own infrastructure. This flexibility allows businesses to choose the deployment model that best fits their needs, ensuring they can leverage Polaris Catalog’s capabilities to enhance their data management strategies.

Why Companies Are Replacing Existing Batch Pipelines

Fidelity has reimagined its data pipelines using Snowflake Marketplace, saving the company time and resources in data engineering. Its supported business units, including fixed income and data science, can now analyze data faster, spending “more time on research and less on pipeline management,” said Balaram Keshri, vice president of architecture at Fidelity.

With Snowflake managing its data, Fidelity has significantly improved performance, enabling faster data loading, querying, and analysis. The Snowflake Performance Index reports that it has “reduced organizations’ query duration by 27% since it started tracking this metric, and by 12% over the past 12 months,” according to a press release.

Capital One’s Success with Data Sharing

Capital One, reportedly the first U.S. bank to migrate its entire on-premises data center to the cloud, has also found success with its new data pipelines, thanks to Snowflake’s data sharing capabilities. This feature allows multiple analysts to access related data without affecting one another’s performance. Users can also categorize data according to workload type.

“Snowflake is so flexible and efficient that you can quickly go from ‘data starved’ to ‘data drunk.’ To avoid that data avalanche and associated costs, we worked to put some controls in place,” wrote Salim Syed, head of engineering for Capital One Software, in a blog post.

CNN’s Real-Time Data Transformation

CNN’s dramatic pipeline transformation has provided accelerated access to analytics. Over the past year, the multinational news channel and website, owned by Warner Bros. Discovery, has shifted to using real-time data pipelines for workloads that support critical parts of its content delivery strategy. The goal is to move the horizon of actionable data down “from hours to seconds” by replacing existing batch pipelines.

“We will move around 100 terabytes of data a day across about 600,000 queries from our various partners,” said Zach Lancaster, engineering manager of Warner Bros. Discovery. Now, with its scalable and newly managed pipeline, CNN can scrape the data for core use cases and prioritize workloads that drive the most business value.

Steps to Transform Your Data Pipeline

As user-friendly as the Snowflake platform is, IT leaders still need a clear strategy in mind as they improve their data pipelines. Here are three steps to help transform your data pipeline effectively.

Step 1: Engage Stakeholders

For starters, “think about how you can bring your stakeholders on board. You want them to become the ultimate stewards of the process,” Lancaster said. Engaging stakeholders ensures that the pipeline transformation aligns with business goals and receives the necessary support for successful implementation.

Step 2: Revisit Use Cases

Second, revisit your use cases. “Platforms develop over the years, as does your business, so try to re-evaluate your use cases and dial back your system,” Torrance advised. This approach can help with cost optimization and ensure that the pipeline meets current business needs.

Step 3: Understand Requests

Third, “make sure you understand the ask of each request and how you expect to use it over time in your data pipeline,” Lancaster said. Clear understanding of requests ensures that the pipeline is designed to handle future demands and remains flexible enough to accommodate changes.

Cross-Functional and Centralized Pipelines

If a company is redesigning its data pipeline, it needs to be cross-functional and serve the most central parts of the business. Consider “machine-to-machine use cases,” as these are important for interoperability within your entire tech stack.

Finally, remember that more intricate systems aren’t always better. “Think carefully. Just because I have a request, do I need to accomplish it? And does the added complexity add value to the business, or does it do a disservice to the stakeholder?” Lancaster said. This consideration ensures that the pipeline remains efficient and aligned with business objectives.

Conclusion

Transforming data pipelines is essential for organizations looking to leverage real-time analytics and improve data management. By adopting solutions like Snowflake’s Apache Iceberg and Iceberg Tables, companies can simplify pipeline development, enhance performance, and ensure scalability. Engaging stakeholders, revisiting use cases, and understanding requests are crucial steps in this transformation process. As organizations continue to navigate the complexities of data management, adopting flexible and scalable solutions like those offered by Snowflake will be key to maintaining competitive advantage. By focusing on efficient pipeline management and leveraging advanced technologies, businesses can unlock the full potential of their data and drive innovation.