twitter / the-algorithm

# X's Recommendation Algorithm
X's Recommendation Algorithm is a set of services and jobs that are responsible for serving feeds of posts and other content across all X product surfaces (e.g. For You Timeline, Search, Explore, Notifications). For an introduction to how the algorithm works, please refer to our engineering blog.
## Architecture
Product surfaces at X are built on a shared set of data, models, and software frameworks. The shared components included in this repository are listed below:
| Type | Component | Description |
|------------|------------|------------|
| Data | tweetypie | Core service that handles the reading and writing of post data. |
| | unified-user-actions | Real-time stream of user actions on X. |
| | user-signal-service | Centralized platform to retrieve explicit (e.g. likes, replies) and implicit (e.g. profile visits, tweet clicks) user signals. |
| Model | SimClusters | Community detection and sparse embeddings into those communities. |
| | TwHIN | Dense knowledge graph embeddings for Users and Posts. |
| | trust-and-safety-models | Models for detecting NSFW or abusive content. |
| | real-graph | Model to predict the likelihood of an X User interacting with another User. |
| | tweepcred | Page-Rank algorithm for calculating X User reputation. |
| | recos-injector | Streaming event processor for building input streams for GraphJet based services. |
| | graph-feature-service | Serves graph features for a directed pair of users (e.g. how many of User A's following liked posts from User B). |
| | topic-social-proof | Identifies topics related to individual posts. |
| | representation-scorer | Compute scores between pairs of entities (Users, Posts, etc.) using embedding similarity. |
| Software framework | navi | High performance, machine learning model serving written in Rust. |
| | product-mixer | Software framework for building feeds of content. |
| | timelines-aggregation-framework | Framework for generating aggregate features in batch or real time. |
| | representation-manager | Service to retrieve embeddings (i.e. SimClusers and TwHIN). |
| | twml | Legacy machine learning framework built on TensorFlow v1. |
The product surfaces currently included in this repository are the For You Timeline and Recommended Notifications.
### For You Timeline
The diagram below illustrates how major services and jobs interconnect to construct a For You Timeline.
The core components of the For You Timeline included in this repository are listed below:
| Type | Component | Description |
|------------|------------|------------|
| Candidate Source | search-index | Find and rank In-Network posts. ~50% of posts come from this candidate source. |
| | tweet-mixer | Coordination layer for fetching Out-of-Network tweet candidates from underlying compute services. |
| | user-tweet-entity-graph (UTEG)| Maintains an in memory User to Post interaction graph, and finds candidates based on traversals of this graph. This is built on the GraphJet frame