How Pinterest Leverages Realtime Consumer Actions in Advice to Increase Homefeed Engagement Quantity | by Pinterest Engineering | Pinterest Engineering Weblog |

How Pinterest Leverages Realtime Consumer Actions in Advice to Increase Homefeed Engagement Quantity | by Pinterest Engineering | Pinterest Engineering Weblog |
How Pinterest Leverages Realtime Consumer Actions in Advice to Increase Homefeed Engagement Quantity | by Pinterest Engineering | Pinterest Engineering Weblog |

Xue Xia, Software program Engineer, Homefeed Rating; Neng Gu, Software program Engineer, Content material & Consumer Understanding; Dhruvil Deven Badani, Engineering Supervisor, Homefeed Rating; Andrew Zhai, Software program Engineer, Superior Applied sciences Group

Image from — black background with turquoise grid points

On this weblog put up, we’ll reveal how we improved Pinterest Homefeed engagement quantity from a machine studying mannequin design perspective — by leveraging realtime consumer motion options in Homefeed recommender system.

The Homepage of Pinterest is the considered one of most necessary surfaces for pinners to find inspirational concepts and contributes to a big fraction of total consumer engagement. The pins proven within the prime positions on the Homefeed must be personalised to create an enticing pinner expertise. We retrieve a small fraction of the massive quantity of pins created on Pinterest as Homefeed candidate pins, in keeping with consumer curiosity, adopted boards, and so forth. To current probably the most related content material to pinners, we then use a Homefeed rating mannequin (aka Pinnability mannequin) to rank the retrieved candidates by precisely predicting their personalised relevance to given customers. Due to this fact, the Homefeed rating mannequin performs an necessary position in bettering pinner expertise. Pinnability is a state-of-the-art neural community mannequin that consumes pin alerts, consumer alerts, context alerts, and so forth. and predicts consumer motion given a pin. The excessive stage structure is proven in Determine 3.

Flow map of candidate pins going through pinnability models, becoming relevance ordered, then to Homefeed

The Pinnability mannequin has been utilizing some pretrained consumer embedding to mannequin consumer’s curiosity and desire. For instance, we use PinnerFormer (PinnerSAGE V3), a static, offline-learned consumer illustration that captures a consumer’s long run curiosity by leveraging their previous interplay historical past on Pinterest.

Nevertheless, there are nonetheless some elements that pretrained embeddings like PinnerSAGE doesn’t cowl, and we will fill within the hole through the use of a realtime consumer motion sequence function:

  • Mannequin pinners’ short-term curiosity: PinnerSAGE is educated utilizing 1000’s of consumer actions over a long run, so it principally captures long-term curiosity. Then again, realtime consumer motion sequence fashions short-term consumer curiosity and is complementary to PinnerSAGE embedding.
  • Extra responsive: As an alternative of different static options, realtime alerts are capable of reply quicker. That is useful, particularly for brand spanking new, informal, and resurrected customers who should not have a lot previous engagement.
  • Finish-to-end optimization for suggestion mannequin goal: We use a consumer motion sequence function as a direct enter function to the advice mannequin and optimize straight for mannequin aims. Not like PinnerSAGE, we will attend the pin candidate options with every particular person sequence motion for extra flexibility.

To be able to give pinners real-time suggestions to their current actions and enhance the consumer expertise on Homefeed, we suggest to include the realtime consumer motion sequence sign into the advice mannequin.

A secure, low latency, realtime function pipeline helps a strong on-line suggestion system. We serve the most recent 100 consumer actions as a sequence, populated with pin embeddings and different metadata. The general structure might be segmented to occasion time and request, as proven in Determine 2.

at event time, rockstore stores information from Kafka log via NRT/Flink App Materializer. At request time, HF logging/serving request go through Unity HF, USSv2 Aggregator, USSv2 view, then stored in rockstore and transform into merged UFr

To reduce the applying downtime and sign failure, efforts are made in:

ML facet

  • Options/schema power validation
  • Delayed supply occasion dealing with to stop knowledge leakage
  • Itemized actions monitoring over time knowledge shifting

Ops facet

  • Stats monitoring on core job well being, latency/throughput and so forth.
  • Complete on-calls for minimal software downtime
  • Occasion restoration technique

We generated the next options for the Homefeed recommender mannequin:

Headers: Feature Name & Description.  pin EngagementActionTypeSequence — Users’ past 100 engagement actions (e.g. repin, click, hide, etc) pinEngagementEmbeddingSequence — Users’ past 100 engagement pins’s pinSAGE embedding pinEngagementTimestampSequence — The timestamp of users’ past 100 engagement

Determine 3 is an outline of our Homefeed rating mannequin. The mannequin consumes a <consumer, pin> pair and predicts the motion that the consumer takes on the candidate pin. Our enter to the Pinnability mannequin contains alerts of varied sorts, together with pinner alerts, consumer alerts, pin alerts, and context alerts. We now add a singular, realtime consumer sequence alerts enter and use a sequence processing module to course of the sequence options. With all of the options remodeled, we feed them to an MLP layer with a number of motion heads to foretell the consumer motion on the candidate pin.

Diagram of Pinterest Homefeed Ranking (Pinnabilty) Model Architecture

Latest literature has been utilizing transformers for suggestion duties. Some mannequin the advice drawback as a sequence prediction job, the place the mannequin’s enter is (S1,S2, … , SL-1) and its anticipated output as a ‘shifted’ model of the identical sequence: (S2,S3, … , SL). To maintain the present Pinnability structure, we solely undertake the encoder a part of these fashions.

To assemble the transformer enter, we utilized three necessary realtime consumer sequence options:

  1. Engaged pin embedding: pin embeddings (discovered GraphSage embedding) for the previous 100 engaged pins in consumer historical past
  2. Motion kind: kind of engagement in consumer motion sequence (e.g., repin, click on, disguise)
  3. Timestamp: timestamp of a consumer’s engagement in consumer historical past

We additionally use candidate pin embedding to carry out early fusion with the above realtime consumer sequence options.

initial architecture of user sequence transformer module

As illustrated in Determine 3, to assemble the enter of the sequence transformer module, we stack the [candidate_pin_emb, action_emb, engaged_pin_emb] to a matrix. The early fusion of candidate pin and consumer sequence is proved to be essential in keeping with on-line and offline experiments. We additionally apply a random time window masks on entries within the sequence the place the actions had been taken inside sooner or later of request time. The random time window masks is used to make the mannequin much less responsive and to keep away from range drop. Then we feed it right into a transformer encoder. For the preliminary experiment, we solely use one transformer encoder layer. The output of the transformer encoder is a matrix of form [seq_len, hidden_dim]. We then flatten the output to a vector and feed it together with all different options to MLP layers to foretell multi-head consumer actions.

In our second iteration of the consumer sequence module (v1.1), we made some tuning on prime of the v1.0 structure. We elevated the variety of transformer encoder layers and compressed the transformer output. As an alternative of flattening the complete output matrix, we solely took the primary 10 output tokens, concatenated them with the max pooling token, and flattened it to a vector of size (10 + 1) * hidden_dim. The primary 10 output tokens seize the consumer’s most up-to-date pursuits and the max pooling token can characterize the consumer’s long term desire. As a result of the output dimension turned a lot smaller, it’s reasonably priced to use an express function crossing layer with DCN v2 structure on the complete function set as beforehand illustrated in Fig.2.

Improved architecture of user sequence transformer module (v1.1)

Problem 1: Engagement Charge Decay

By means of on-line experiments, we noticed the consumer engagement metrics steadily decayed within the group with realtime motion sequence therapy. Determine 6 demonstrates that for a similar mannequin structure, if we don’t retrain it, the engagement achieve is way smaller than if we retrain the mannequin on contemporary knowledge.

Chart of Homefeed Repin Volume Increase change by time. Blue line represents retrained model. Red line represents fixed model.

Our speculation is that our mannequin with realtime options is sort of time delicate and requires frequent retraining. To confirm this speculation, we retrain each the management group (with out realtime consumer motion function) and the therapy group (with realtime consumer motion function) on the similar time, and we examine the impact of retraining for each fashions. As proven in Determine 6, we discovered the retraining advantages within the therapy mannequin far more than within the management mannequin.

Chart of Overall repin gain of sequence model retrain and control model retrain across day 0 to day 11

Due to this fact, to deal with the engagement decay problem, we retrain the realtime sequence mannequin twice per week. In doing this, the engagement fee has grow to be far more secure.

Problem 2: Serving Massive Mannequin at Natural Scale

With the transformer module launched to the recommender mannequin, the complexity has elevated considerably. Earlier than this work, Pinterest has been serving the Homefeed rating mannequin on CPU clusters. Our mannequin will increase CPU latency by greater than 20x. We then migrated to GPU serving for the rating mannequin and are capable of maintain impartial latency on the similar value.

On Pinterest, one of the crucial necessary consumer actions is repin, or save. Repin is among the key indicators of consumer engagement on the platform. Due to this fact, we approximate the consumer engagement stage with repin quantity and use repin quantity to judge mannequin efficiency.

Offline Analysis

We carry out offline analysis on totally different fashions that course of realtime consumer sequence options. Particularly, we tried the next architectures:

  • Common Pooling: the only structure the place we use the common of pin embedding in consumer sequence to current consumer’s quick time period curiosity
  • (Convolutional Neural Community (CNN): makes use of CNN to encoder a sequence of pin embedding. CNN is appropriate to seize the dependent relationship throughout native data
  • Recurrent Neural Community (RNN): makes use of RNN to encoder a sequence of pin embedding. In comparison with CNN, RNN higher captures long term dependencies.
  • Misplaced Brief-Time period Reminiscence (LSTM): makes use of LSTM, a extra subtle model of RNN that captures longer-term dependencies even higher than RNN through the use of reminiscence cells and gating.
  • Vanilla Transformer: encodes solely the pin embedding sequence straight utilizing the Transformer module.
  • Improved Transformer v1.0: Improved transformer structure as illustrated in Determine 4.

For Homefeed floor particularly, two of crucial metrics are HIT@3 for repin and conceal prediction. For repin, we attempt to enhance the HIT@3. For disguise, the objective is to lower HIT@3.

Headings: Model, hide, repin. Average Pooling -1.61% 0.21% CNN -1.29% 0.08% RNN -2.46% -1.05% LSTM -2.98% -0.75% Vanilla Transformer -8.45% 1.56% Improved Transformer v1.0 -13.49% 8.87%

The offline outcome reveals us that even with the vanilla transformer and solely pin embeddings, the efficiency is already higher than different architectures. The improved transformer structure confirmed very robust offline outcomes: +8.87% offline repin and a -13.49% disguise drop. The achieve of improved transformer 1.0 from vanilla transformer got here from a number of elements:

  1. Utilizing motion embedding: this helps mannequin distinguish optimistic and adverse engagement
  2. Early fusion of candidate pin and consumer sequence: this contributes to nearly all of engagement achieve, in keeping with on-line and offline experiment,
  3. Random time window masks: helps with range

On-line Analysis

Then we performed a web based A/B experiment on 1.5% of the overall site visitors with the improved transformer mannequin v1.0. Through the on-line experiment, we noticed that the repin quantity for total customers elevated by 6%. We outline the set of recent, informal, and resurrected customers as non-core customers. And we noticed that the repin quantity achieve on non-core customers can attain 11%. Aligning with offline analysis, the disguise quantity was decreased by 10%.

Lately, we tried transformer mannequin v1.1 as illustrated in Determine 4, and we achieved a further 5% repin achieve on prime of the v1.0 mannequin. Cover quantity stays impartial for v1.0.

Headings: Model Variation, Cumulative Homefeed Repin Volume (all users & non-core users) Cumulative Homefeed Hide Volume (all users).  Sequence Model V1.0 6% 10% -10% Sequence Model V1.1 + Feature Crossing 11% 17% -10%

Manufacturing Metrics (Full Site visitors)

We wish to name out an attention-grabbing statement: the net experiment underestimates the ability of realtime consumer motion sequence. We noticed larger achieve after we rolled out the mannequin because the manufacturing Homefeed rating mannequin to full site visitors. It’s because the training impact of optimistic suggestions loop:

  1. As customers see a extra responsive Homefeed, they have an inclination to have interaction with extra related content material, and their habits modified (for instance, extra clicks or repins)
  2. With this habits change, the realtime consumer sequence that logs their habits in realtime additionally shifted. For instance, there are extra repin actions within the sequence. Then we generate the coaching knowledge with this shifted consumer sequence function.
  3. As we retrain the Homefeed rating mannequin with this shifted dataset, there’s a optimistic compounding impact that makes the retrained mannequin extra highly effective, thus, a better engagement fee. This then loops us again to 1.
Diagram of feedback loop of Realtime Sequence Model: 1. User behavior change: User’s behavior changed as they see more responsive recommendations Leads to 2. Training data change -User action sequence feature itself changed — More repin actions in training data then leads to 3. Ranking model improved — model is retrained on latest dataset — predicts user action more accurately — higher engagement Then loop back to 1

The precise Homefeed repin quantity improve that we noticed after transport this mannequin to manufacturing is larger than on-line experiment outcomes. Nevertheless, we won’t disclose the precise quantity on this weblog.

Our work to make use of realtime consumer motion alerts in Pinterest’s Homefeed recommender system has significantly improved the Homefeed relevance. Transformer structure seems to work greatest amongst different conventional sequence modeling approaches. There have been varied challenges alongside the best way and are non-trivial to deal with. We found that retraining the mannequin with realtime sequence is necessary to maintain up the consumer engagement. And that GPU serving is indispensable for big scale, advanced fashions.

It’s thrilling to see the large achieve from this work, however what’s extra thrilling is that we all know there’s nonetheless far more room to enhance. To proceed bettering Pinner expertise, we’ll work on the next elements:

  1. Function Enchancment: We plan to develop a extra fine-grained realtime sequence sign that features extra motion sorts and motion metadata.
  2. GPU Serving Optimization: That is the primary use case to make use of GPU clusters to serve giant fashions at natural scale. We plan to enhance GPU serving usability and efficiency.
  3. Mannequin Iteration: We’ll proceed engaged on the mannequin iteration in order that we absolutely make the most of the realtime sign.
  4. Adoption on Different Surfaces: We’ll strive comparable concepts in different surfaces: associated pins, notifications, search, and so forth.

This work is a results of collaboration throughout a number of groups at Pinterest. Many due to the next those that contributed to this mission:

  • GPU serving optimization: Po-Wei Wang, Pong Eksombatchai, Nazanin Farahpour, Zhiyuan Zhang, Saurabh Joshi, Li Tang
  • Technical help on ML: Nikil Pancha
  • Sign era and serving: Yitong Zhou
  • Quick controllability distribution convergence: Ludek Cigler

To be taught extra about engineering at Pinterest, try the remainder of our Engineering Weblog and go to our Pinterest Labs website. To discover life at Pinterest, go to our Careers web page.