Logistic regression is usually encountered in the first few lessons of machine learning. As a probabilistic mapping of linear regression, it is highly interpretable and remains widely used in production recommender systems.

In recommendation, logistic regression turns the task into binary classification: predicting whether a user will click. This aligns naturally with click-through rate (CTR) prediction.

The familiar logistic-regression formula is:

$P(y=1 \mid \mathbf{x}) = \frac{1}{1 + e^{-\mathbf{w}^T \mathbf{x} - b}}$

Here, $\mathbf{w}$ is the weight vector, $\mathbf{x}$ is the feature vector, and $b$ is the bias term. The loss function is cross-entropy loss.

Logistic regression takes a linear combination of input features and maps it to a real number in $[0, 1]$ through the sigmoid function. Because that range has a natural probabilistic interpretation, the model was adopted early in CTR prediction for advertising.

The sigmoid function is popular not only because its output lies between 0 and 1, but also because its derivative is easy to compute: $f(x)(1-f(x))$.

Why logistic regression

Compared with mathematically elegant support vector machines and tree-based models, logistic regression has held an important place throughout the history of recommender systems.

There are three main reasons to use it:

• Sound statistical foundation: logistic regression is a generalized linear model that assumes a Bernoulli-distributed dependent variable, which resembles a CTR task.
• Strong interpretability: every feature has a corresponding weight. Whether it is a feature cross or a standalone feature, its contribution to the final prediction is visible through the magnitude of that weight.
• Low training cost: training can be parallelized across machines.

There are already many explanations online of logistic regression, its derivation, and gradient descent, so I will not repeat them here. In early recommender systems, features were often engineered manually, which was inconvenient—especially when feature interactions had to be considered. GBDT+LR emerged to automate feature combination. I will save that for a future post.

August 4, 2024, in Suzhou