Metis: Constructing Airbnb’s Subsequent Technology Knowledge Administration Platform | by Xiaobin Zheng | The Airbnb Tech Weblog | Jun, 2023

Xiaobin Zheng
The Airbnb Tech Blog

How Airbnb developed our knowledge catalog right into a platform for managing and governing our knowledge warehouse at scale.

By: Erik Ritter, Jiaxin Ye, Sylvia Tomiyama, Woody Zhou, Xiaobin Zheng, Zuzana Vejrazkova

At Airbnb, tens of millions of knowledge belongings exist in a posh ecosystem to tell our enterprise and enhance our merchandise. The Knowledge Administration workforce’s mission is to empower the corporate to handle its knowledge ecosystem at scale.

To do that, we want an correct understanding of all the belongings in our ecosystem and the way they relate to one another. In different phrases, it requires correct metadata. Our knowledge administration platform Metis, named for the Greek goddess of excellent counsel, is our resolution to make sure that reliable metadata could be captured, managed, and consumed at scale.

Metis is an evolution of our current basis of metadata merchandise inside Airbnb.

Dataportal was our first effort in direction of democratizing knowledge: efficiently enabling knowledge customers to seek out trusted knowledge. It was an enormous boon to productiveness and fairly forward of its time.

As knowledge reliability and compliance rules turned vital, we wanted a extra complete and detailed understanding of how knowledge was remodeled. This led to our adoption of Apache Atlas as our knowledge lineage resolution. Apache Atlas powers merchandise like SLA Tracker (see Visualizing Knowledge timeliness at Airbnb), which mixes touchdown time metadata and lineage to allow debugging upstream knowledge delays.

As our necessities for metadata elevated, increasing to extra areas like value administration, knowledge high quality, and many others, our wants for a knowledge catalog have expanded:

  • Skill to control each the info and metadata describing it
  • Guardrails and proposals to enhance knowledge high quality
  • Auditability of a dataset’s historical past, each for debugging & governance functions

We quickly discovered that knowledge administration needed to be pursued as a self-discipline, thus constructing Metis because the one-stop-shop for accessing all knowledge metadata.

Metis is made up of three core merchandise: Dataportal, Unified Metadata Service (UMS), and Lineage Service. Collectively, this platform permits Airbnb to handle tens of millions of knowledge belongings throughout many domains. A brief record of belongings we assist embrace:

  • Apache Hive and Trino datasets
  • Metrics and Dimensions, powered by Airbnb’s Metric Platform: Minerva
  • Charts and Dashboards from Apache Superset and Tableau®
  • Knowledge Fashions, together with these licensed by Midas
  • Machine Studying options and fashions
  • Groups and staff of Airbnb (not technically a knowledge asset, however key to assist prime quality possession and guarantee metadata stays updated for all of the above knowledge belongings)
Metis Structure

On a excessive stage, Metis consists of following elements:

Dataportal — serves as a catalog and administration UI for human customers.

Viaduct — Airbnb’s in-house GraphQL API layer modeling offline knowledge ecosystem.

UMS Core service — a backend service holding system schema and enterprise logic wanted for metadata administration.

Metadata storage:

  • MySQL — primarily storing crucial metadata that must be centrally managed
  • Lineage Graph — a centralized service amassing and serving knowledge lineage
  • Elasticsearch — serving search & discovery use circumstances

Offline Element — exterior to UMS Core service to carry out offline duties: e.g. offline metadata consistency test, coverage enforcement.

Offline Dataset — offline export of metadata for analytics use circumstances.

Dataportal serves because the UI for Airbnb’s knowledge catalog and is a spot for individuals to seek out and handle all of the belongings supported by Metis. It’s constructed as a Single Web page Utility utilizing React and TypeScript and is subsequently versatile sufficient to serve the big number of workflows required for knowledge administration and governance. The frontend communicates with UMS and different companies by way of a GraphQL API; that is particularly vital as we wish to stop each sequential fetches of lineage data and over-fetching giant quantities of metadata to make sure a performant person expertise.

The Dataportal expertise begins with search, in order that each knowledge customers and knowledge house owners can discover the belongings they want. We’ve designed our search and discovery expertise with a couple of rules in thoughts:

  • Show related metadata instantly within the search outcomes to assist individuals discover the precise asset they’re on the lookout for
  • Uprank prime quality and generally used knowledge belongings, within the case that the person is unaware of the precise asset they want

Consequently, search outcomes are inclined to return prime quality, licensed datasets, together with the outline, latest person rely, and final time it was modified to assist the person discover which asset they wish to choose:

Dataportal search end result web page

As soon as the specified asset is situated, the person can go to the Entity Web page to carry out a big number of consumption, administration, and governance actions. We construction all of the content material on the entity web page into tabs grouped by class of knowledge or motion:

The numerous tabs out there for metadata on Hive Tables

Consumption and documentation associated tabs make it simple for individuals to learn to use this desk, with column and desk descriptions within the Configuration tab, proprietor and client knowledge on the Factors of contact tab, and additional particulars on the right way to use the desk on the Documentation tab. Past that, these pages additionally enable customers to tackle administration actions, as seen within the beneath screenshots:

Anybody can tag columns that include private knowledge. Altering and eradicating tags require a evaluate to make sure that private knowledge is accurately recognized in our warehouse.

The above screenshot highlights solely a subset of the way we leveled up the Dataportal from a searchable knowledge catalog into the one centralized place to handle and govern all of your knowledge belongings.

Unified Metadata Service, or UMS, is the backend core of our centralized knowledge administration platform. It supplies:

  • A centralized schema and Graphql API layer on prime of it to entry metadata
  • A centralized relationship graph to attach siloed metadata
  • Centralized metadata administration capabilities to allow methods to fulfill compliance and governance necessities with out reinventing the wheel

The centralization of metadata into UMS prevents all metadata suppliers and customers from needing to combine with one another; as a substitute all suppliers and customers solely should combine with UMS:

Decreasing integration factors for metadata

UMS performs numerous roles throughout metadata integrations and use circumstances. In a decentralized knowledge ecosystem, we’re very opinionated about what metadata needs to be saved, replicated to, or served by UMS.

UMS helps proxying learn requests to many knowledge methods. This contains proxying learn requests to:

  • Hive Metastore for desk schema and desk properties.
  • Lineage service for uncooked Hive desk knowledge lineage.
  • Knowledge Governance service for knowledge governance standing for datasets.

UMS centrally manages a couple of crucial enterprise metadata and shops in its personal metadata database with administration capabilities:

  • Validation and authorization for updates
  • Audit historical past
  • Approval workflow for delicate operations on crucial metadata

As a part of Airbnb’s Knowledge High quality Initiative, we carried out knowledge high quality scores which might be instantly tied to every knowledge asset within the knowledge warehouse. Knowledge high quality scores for datasets are generated in an offline method and ingested into UMS metadata database for on-line consumption.

Just like conventional knowledge catalog, UMS centrally manages indexes in an Elasticsearch cluster for various entities to energy knowledge discovery.

There are circumstances the place metadata must be saved or replicated into Metis storage layer. UMS integrates with metadata suppliers in a wide range of paved mechanisms to ingest metadata leveraging Airbnb’s tech stack. These embrace:

  • Stream processing (Flink) jobs ingesting metadata change occasions.
  • ETL(Airflow) jobs that run each day to tug from metadata suppliers and push to UMS.
  • Direct calls to UMS API.

After we onboard a brand new metadata supplier, the important thing work concerned is figuring out product necessities and aligning on the scope of metadata integration, adopted by finalizing the precise integration mechanism.

The ultimate main piece of Metis is our Lineage Service. We adopted Apache Atlas as Airbnb’s knowledge lineage resolution for Knowledge Warehouse again in 2020.

At Airbnb, Apache Atlas holds a big lineage graph containing over 100 million nodes and 300 million edges. The first quantity of lineage knowledge comes from manufacturing Hive tables and a big quantity of intermediate Hive tables in our Knowledge Warehouse.

Now we have extensively personalized and tuned Apache Atlas to deal with the big scale lineage occasions in our Knowledge Warehouse:

  • Apply sharding technique on lineage occasions to extend parallelism.
  • Bettering Atlas server code effectivity on prime of a graph database.
  • Wonderful tuning underlying storage methods backing the graph database for scalability and latency.
  • Learn path optimization and filtering assist for accessing lineage knowledge extra effectively.

Atlas’s lineage-related elements, together with its Graph Engine (JanusGraph), Kind System, Ingest (with Hook integrations), and lineage API, have allowed us to effectively accumulate and serve lineage knowledge, offering priceless insights into the relationships between numerous knowledge belongings and pipelines. It’s powering many crucial knowledge compliance, knowledge reliability and knowledge high quality merchandise. See Visualizing Knowledge Timeliness at Airbnb.

As proven above, Airbnb’s strategy to knowledge administration has considerably developed over the previous 6 years. We began constructing Dataportal with a purpose to “democratize knowledge” at Airbnb, and we now have Metis: a platform that permits anybody at Airbnb to go looking, uncover, devour, and handle all the info and metadata in our offline warehouse. Metis has been serving crucial roles throughout knowledge compliance, knowledge reliability, knowledge high quality initiatives and helps 1000+ knowledge customers each week.

Our future work will contain two key priorities: firstly, we are going to give attention to evolving our system structure and underlying know-how to be able to preserve tempo with the speedy evolution of our knowledge ecosystem. Secondly, we plan to develop our protection to extra methods and allow extra superior knowledge administration capabilities, reflecting our ongoing dedication to investing in knowledge right here at Airbnb.

Metis wouldn’t have been doable with out the members of the info administration workforce in addition to our cross practical and cross org collaborators. They embrace, however usually are not restricted to: Adam Kocoloski, Adam Wong, Cindy Yu, Dave Nagle, Erik Ritter, Jerry Wang, Jiaxin Ye, John Bodley, Jyoti Wadhwani, Liyin Tang, Michelle Thomas, Nathan Towery, Paul Ellwood, Sylvia Tomiyama, Vyl Chiang, Woody Zhou, Xiaobin Zheng, and Zuzana Vejrazkova.

Apache Airflow, Apache Atlas, Apache Hive, Apache Superset, Atlas, and Hive are both registered logos or logos of The Apache Software program Basis in the US and different international locations.

All logos, service marks, firm names and product names are the property of their respective house owners. Any use of those are for identification functions solely and don’t suggest sponsorship and endorsement.