Netflix recently published how its Machine Learning Platform (MLP) team provides an ecosystem around Metaflow, an open-source machine learning infrastructure framework. By creating various integrations for Metaflow, Netflix is able to support hundreds of Metaflow projects maintained by multiple engineering teams.
Metaflow's integrations with Netflix's production systems enable projects to move from prototype to production without incurring unsustainable operational overhead. The engineering team explains their key to success:
Given the very diverse set of ML and AI use cases we support [...], we don't expect all projects to follow the same path from prototype to production. Instead, we provide a robust foundational layer with integrations to our company-wide data, compute, and orchestration platform, as well as various paths to deploy applications to production smoothly. On top of this, teams have built their own domain-specific libraries to support their specific use cases and needs.
One integration example provided is the "Fast Data" library for Metaflow. Netflix hosts its main data lake on S3 as Apache Iceberg tables and uses Apache Spark for ETL. The Fast Data library enables fast, scalable, and robust access to the Netflix data warehouse by leveraging high-performance components from the Python data ecosystem. This library allows Netflix to process terabytes of data collectively and encode complex relationships between titles, actors, and other film attributes, supporting the company's broad business applications.
The Fast Data Library for Metaflow (Source)
Netflix's production workflow orchestrator, Maestro, plays a critical role in managing Metaflow projects in production. It supports scalability and high availability and enables seamless integration of Metaflow flows with other systems through event-triggering. Using this integration, Netflix engineers can support content decision-making and answer "what content Netflix should bring to the service".
Finally, for deployments that require an API and real-time evaluation, Netflix provides an integrated model hosting service, Metaflow Hosting. Metaflow Hosting "provides an easy to use interface on top of Netflix's existing microservice infrastructure, allowing data scientists to quickly move their work from experimentation to a production grade web service that can be consumed over a HTTP REST API with minimal overhead."
Using this integration, Netflix hosts and scales a model to compute various media asset features. Once it finds the features, a consuming service can store them in a store for future use. An old talk provides an overview and further details about this service.
Hosting and Consuming the Media Feature Computation Model (Source)
Netflix implemented the integrations using Metaflow's extension mechanism, "which is publicly available but subject to change and hence not part of Metaflow's stable API yet." They invite engineers to contact them on the Metaflow community Slack to discuss building additional extensions.