Recommender-system misconceptions, v1: recent notes

In recent work, I have gradually tried recommender-system ideas in production. Beyond exploring model architectures, a series of posts by POSO author Zhao Zhichen gave me a much stronger understanding of the business side of recommendation. As of August 2, 2025, I am recording a few current thoughts for future study.

Ranking

As the final stage of a recommender system, ranking is the most direct ranking model. Precisely because its training data is generated by the system itself, it can easily suffer from sample-distribution shift, much like credit-risk modeling. In credit risk, only approved applicants produce subsequent repayment behavior and therefore default labels; we never know whether rejected applicants would have defaulted. The common remedy is reject inference: infer a possible risk label for rejected applicants to expand the training set and reduce bias. In essence, this is a form of data augmentation.

Ranking also determines the preference of the whole system. An improvement made in retrieval will not show up in results at all if the ranking model does not recognize its value.

Two issues with ranking:

1. Positive examples are not fully reliable. A positive example is an explicit user choice, but a negative example is not necessarily truly negative. Exposure determines, to a degree, how likely an example was to become positive. Some attractive items are never exposed and are gradually eliminated by the system.
2. Self-reinforcement. This is the distribution shift mentioned above: when a system learns only from itself, it can drift further and further in the wrong overall direction.

Pre-ranking and retrieval

Pre-ranking supports the final ranker. To reduce the ranker’s workload, it learns to approximate the ranker’s output, so NDCG is used as a ranking metric. Retrieval not only supplies candidates for downstream models; it also fills gaps. If the ranker cannot surface a certain type of content for some reason, a retrieval channel must provide it. Retrieval can be rule-based, model-based, or based on two-tower similarity.

Outputs from multiple retrieval channels need deduplication during merging, and allocation order matters. A channel deployed first is more likely to shape the system and gradually dominate it. For example, if channel A is served first and users who prefer B leave, deploying a B channel later may never produce positive returns because those users have already gone.

The same pattern appears in precision marketing. If a user is first contacted through an outbound channel, they may develop resistance; later marketing no longer reaches a “fresh” customer and conversion declines. When different products market to the same user, the first one to reach them gains the initiative. Operations therefore matter enormously.

Rules and models

Writing models is not inherently more advanced, and writing rules is not inherently ordinary. Rules are necessary at every layer—not only to correct models, but also for commercial considerations such as advertisers’ placements.

Tree models in ranking

Tree models are common in Kaggle, yet are rarely used for final ranking in production. The core reason is that they struggle with the continually arriving ID features of online learning. A tree can assign an unknown feature a default value, but new IDs grow quickly in real systems. Facebook once addressed this by retraining daily, but that conflicts with online learning; as the business grows, even a tree model may not converge within a day.

Embeddings solve this more naturally: new IDs can be assigned new embedding slots with almost no change to the deployment process.

The following is adapted from Zhao Zhichen’s original post:

So far, I have not seen a strong example of combining tree models and embeddings. More precisely, trees have little need or reason to add embeddings. Embeddings map low-dimensional IDs into a higher-dimensional space and store finer-grained information. In my own experience, doubling embedding dimensions for all features usually improves metrics to varying degrees and never hurts them, which suggests that larger embeddings provide more information capacity. Decision trees, by contrast, split directly on information in the features themselves.

Embeddings are powerful but storage-intensive. This is where tree models shine: because they do not need embeddings, they use little memory. They are worth considering for peripheral modeling problems. Those problems can be important. For example, predicting an item’s lifecycle stage—growth, stable expansion, traffic decline, or death—does not benefit from an item ID; useful features are total exposure, time since launch, exposure at each stage, and conversion. Another example is predicting whether a user will return the next day from behavioral features such as category-specific watch counts, tenure, and total watch time today.

In summary, prioritize tree models when:

1. Features do not include continually emerging ID-like categories, and categorical features are enumerable, such as age or city.
2. Inputs mix many kinds of categorical and numerical features, especially counter-style features such as time spent in an app.

The DNN paradigm

Deep neural networks follow a standardized production pattern in recommendation:

1. Map every feature to an ID through hashing.
2. Map each ID to a fixed-length embedding through an embedding table.
3. Concatenate embeddings and feed them into the network.

Facebook’s 2019 DLRM architecture is shown below:

Sparse features are ID-like features, while dense features are numerical. Sparse features are looked up in embedding tables; together with dense features, they are combined through pairwise inner products as feature interactions before entering the network.

August 2, 2025, in Suzhou