CTR Predition DIN

1 Paper Overview

1.1 Summary

Deep Interest Network (DIN) was proposed in June 2017 by Alibaba’s Precision Targeting Retrieval and Fundamental Algorithms Team. It targets CTR prediction in e-commerce, emphasizing the full utilization and mining of user historical behavior data. DIN introduces an attention mechanism to construct different user representations for different ads, thereby more accurately capturing the user’s current interest under fixed data dimensions. Core idea: User interests are diverse, and for a specific ad, different user interests have varying impacts (local activation).

1.2 Paper Information

Paper Title: Deep Interest Network for Click-Through Rate Prediction
Paper Link: https://arxiv.org/abs/1706.06978
Code Repository: https://github.com/zhougr1993/DeepInterestNetwork, with author recommendation to also check https://github.com/mouna99/dien

1.3 Core Insights

Existing CTR prediction models generally follow a common pattern:

Map high-dimensional sparse categorical features into low-dimensional space via Embedding.
Combine and transform these low-dimensional representations group-wise (e.g., sum pooling or mean pooling) to form fixed-length vectors.
Concatenate these vectors and feed them into an MLP to learn non-linear relationships.

This approach has a critical flaw: In e-commerce, user interests are represented by historical behaviors (e.g., visited items, shops, categories). However, under this paradigm, the same fixed-length vector represents user interest regardless of the candidate ad — severely limiting model expressiveness, since user interests are inherently diverse.

The bottleneck of Embedding & MLP models lies in representing diverse user interests — fixed-dimension user vectors become a constraint.

To solve this, DIN introduces a network that adaptively learns user interest representations based on the relevance between the candidate ad and user historical behaviors. Specifically, it introduces a local activation unit module based on Attention, which weights historical behaviors to represent user interest, where weights are learned via interaction between the candidate ad and historical behaviors.

Additionally, the paper introduces two techniques to aid training large networks: Mini-batch Aware Regularization and the Dice activation function.

1.4 Terminology

Diversity: Users exhibit broad interests when browsing e-commerce sites. For example, a young mother may show interest in wool sweaters, handbags, earrings, children’s clothing, and sportswear — indicating wide-ranging interests.
Local Activation: Whether a user clicks on a recommended item depends only on a subset of their historical behavior, not the entire history. For example, a swimmer who previously bought a travel book, ice cream, potato chips, and a swimming cap — when shown goggles, their click decision is primarily influenced by the swimming cap, not the other items. In this CTR prediction, only part of the historical data (swimming cap) is decisive.

2 Interpretation Framework

2.1 Memorization and Generalization

One major challenge in recommendation systems is balancing Memorization and Generalization.

Memorization recommends items directly related to the user’s past behavior.
Generalization learns new feature combinations to improve diversity.

In DeepFM, Wide & Deep correspond to Memorization & Generalization, respectively.

2.1.1 Memorization

For CTR prediction with large-scale sparse features, non-linear transformation followed by linear models (e.g., LR + feature cross-products) is common. Memorization constructs non-linear features via manual feature cross-products to capture high-order correlations — essentially “remembering” feature pairs that co-occurred historically.

Example:

Feature 1 — Major: {CS, Humanities, Other}
Feature 2 — Downloaded Music "Xiao Chou": {Yes, No}

One-hot dimensions: 3 and 2, respectively. Cross-product result:

Feature 3 — Major × Downloaded Music: {CS ∧ Yes, CS ∧ No, Humanities ∧ Yes, Humanities ∧ No, Other ∧ Yes, Other ∧ No}

LR is a typical representative, using raw sparse features and cross-product features. Dense features are often bucketed into sparse features. Advantages: high interpretability, fast implementation, easy feature importance analysis — proven effective in industry.

Disadvantages:

Requires heavy manual feature engineering.
Risk of overfitting — if all features are cross-multiplied, the model may memorize training samples.
Cannot capture unseen feature pairs — e.g., if no one downloaded “Xiao Chou,” the weight for that pair will be zero.

2.1.2 Generalization

Generalization learns low-dimensional dense embeddings for sparse features to capture correlations — embeddings carry semantic information, similar to word vectors in NLP. Generalization relies on transitivity of correlations. Representative models: DNN and FM.

Advantages: less manual intervention, better generalization to unseen feature combinations.

In sparse user-item matrices (e.g., niche users/items), NNs struggle to learn effective embeddings. Dense embeddings may assign non-zero predictions to unrelated pairs, causing over-generalization. Here, Memorization excels — it can “remember” special feature combinations.

2.2 Evolutionary Trajectory

Though NN and FM models appear complex, understanding their evolution — “how to balance memorization and generalization,” “how to handle high-dimensional sparse categorical features,” and “how to implement feature crossing” — reveals that new algorithms are often incremental improvements along this trajectory.

Sparse categorical/ID features are the “first-class citizens” in recommendation/search. Even numerical features (e.g., exposure count, CTR) are often bucketed into categorical features before feeding into models.

However, sparse categorical/ID features suffer from weak individual expressiveness, explosive feature combinations, and uneven distribution. To address this, new techniques have emerged.

Since single categorical/ID features are weak, feature crossing is essential. DNN maps categorical/ID features to dense vectors via embedding, then feeds them into DNN to automatically learn deep feature interactions, enhancing generalization.

3 DNN

3.1 Deep Model Rationale

Accurate CTR prediction requires balancing user, advertiser, and platform interests. CTR technology evolved from LR/FM → ensemble models (RF/GBDT/XGBoost) → deep CTR models (FNN/PNN/WDL/DeepFM/DIN). The core question: How to enable automatic feature combination?

Examples:

Wide & Deep, DeepFM: Combine high-order and low-order features to enhance expressiveness.
PNN: Introduce a product layer (inner/outer product) before MLP to emphasize embedding interactions.

Alibaba’s perspective:

We first considered dimensionality reduction, then feature combination. DNN naturally became a candidate. We also wanted to model user behavior sequences — e.g., a user clicks an item in “Good Goods,” then in “Guess You Like,” then searches — influenced by prior behavior. While indirect methods exist, direct modeling with LR is difficult — hence RNN came to mind.

3.2 DNN Model

Most DNN models follow the Embedding + MLP architecture: map high-dimensional sparse features to fixed-length low-dimensional embeddings, feed them into MLP to fit high-order non-linear relationships, and use Sigmoid to output click probability (0–1). Compared to LR/GBDT/FM, DNN reduces manual feature engineering and learns non-linear relationships.

Typical flow:

Sparse Features → Embedding Vector → Pooling Layer → MLPs → Sigmoid → Output

3.3 Mechanism

The Base Model (below) represents most existing CTR models.

Base Model

Red, blue, and pink nodes represent Goods ID, Shop ID, and Category ID — sparse features. White nodes represent other inputs (e.g., user features like User ID, context features like ad slot). Goods 1–N represent user historical behaviors. Candidate Ad is also a product with Goods/Shop/Category ID.

Observing the Base Model bottom-up:

Module 1: Feature Representation
- Features divided into four categories: user profile, user behavior, ad, context.
- Ad is the target.
- Each category contains multiple fields: user info (gender, age), behavior (visited items), ad (ad ID, shop ID), context (design type, time).
- Features encoded as one-hot (e.g., female → [0,1]) or multi-hot (multiple 1s in a vector).
- In CTR sequence models, each field contains a behavior list, each mapped to a one-hot vector.
Module 2: Embedding Layer
- Learns low-dimensional vector representations, converting sparse matrices to dense.
- Each field has its own embedding matrix.
- Since user behavior lengths vary, embedding matrix columns are variable — cannot directly concatenate with other fields for MLP input.
Module 3: Pooling Layer
- Different users have different behavior counts → variable embedding vector sizes. MLP requires fixed dimensions → pooling creates fixed-length vectors.
- Sum pooling: aggregates embeddings per category into a fixed-length vector.
Module 4: Concatenation Layer
- After embedding and pooling, sparse features become multiple fixed-length abstract user interest vectors.
- Concatenate these vectors into a single abstract user interest vector → MLP input.
Module 5: MLP Layer
- Learns high-order cross features from concatenated vector.
Loss Function: Negative log-likelihood (LogLoss), supervised by labels.

3.4 Model Characteristics

Advantages:

Fits high-order non-linear relationships, reduces manual feature engineering.

Disadvantages:

Limited representation of user interest diversity (major bottleneck). User behavior counts vary — pooling (sum/average) loses information:
- K-dimensional vector can represent at most K independent interests — users may have more.
- Larger K improves performance but increases parameters and overfitting risk.
Ignores user-ad relationship. User behavior data contains rich interest info, but prior models didn’t model its structure (Diversity + Local Activation). For the same user, interest vector is fixed regardless of candidate ad — limiting expressiveness.
Ignores implicit feature mining. DNN treats behavior as interest — but behavior is a carrier of interest. Direct representation ignores behavioral dependencies and temporal dynamics.
Ignores interest evolution. User interests change over time (seasons, trends, taste). In Taobao, interests are diverse and evolve independently — only target-related interests matter.
No need to compress all historical purchases into a vector — only relevant interests affect current behavior. E.g., a female swimmer clicks goggles due to swimwear purchase, not last week’s shoes.

4 DIN

To address DNN’s limitations, Alibaba proposed DIN. Core idea: User interests are diverse, and for a specific ad, different interests have varying impacts (local activation). DIN models both Diversity and Local Activation.

DIN does not use a single vector for all interests — instead, it adaptively computes user interest representations based on historical behavior relevance (for a given ad). The representation vector varies per ad. DIN computes user interest representation by considering the relevance between the candidate ad and user history — via a local activation unit that soft-searches relevant historical behaviors and uses weighted sum to obtain ad-specific interest representation. Behaviors highly relevant to the ad get higher activation weights, dominating user interest. This variable representation greatly enhances model expressiveness.

4.1 Innovations

Deep Interest Network introduces:

Diversity Modeling: Represents user interests via an interest distribution — weighted sum pooling models diverse interests.
Local Activation:
- DNN’s sum/average pooling loses information. DIN improves this via attention — dynamically learns user interest embeddings from history, constructing different user representations per ad → more precise current interest capture.
- Weights assigned to historical behaviors vary per ad — determined by current behavior ID and candidate ad → Attention mechanism. For current candidate Ad, locally activate relevant historical interest info.
- Higher relevance → higher attention score → dominates prediction.
Regularization for Sparse Features: Traditional L1/L2/Dropout insufficient for billion-parameter CTR models. DIN proposes adaptive regularization — different regularization weights per feature frequency in each mini-batch.
Dice Activation Function: Traditional ReLU outputs 0 for negative inputs, slowing convergence. PReLU’s fixed 0 threshold is suboptimal — Dice adapts threshold dynamically based on data.
Adaptive Regularization for Large Sparse Data: Deep DNNs with sparse inputs easily overfit. DIN’s adaptive regularization significantly improves performance.

4.2 Architecture

DIN architecture models both Diversity and Local Activation — see diagram below.

DIN Model

Diversity and Local Activation

We’ll examine each component.

5 Features

5.1 Feature Classification

Alibaba’s display ad system features divided into four categories:

User profile features.
User behavior features (clicked items — variable length per user).
Candidate ad (also a product).
Context features.

Each category includes multiple feature fields — e.g., user profile: gender, age; behavior: clicked goods, categories, shops; context: time.

5.2 Input Characteristics

CTR inputs typically:

High-dimensional.
Very sparse.

Some fields are single-value (mutually exclusive) — e.g., gender → one-hot.

Some are multi-value discrete — e.g., user behavior: clicked multiple items → multi-hot encoding (multiple 1s in vector). Examples:

YouTube: watched/searched videos — multiple, but sparse relative to total videos.
E-commerce: multiple purchased good/shop IDs — variable-length history per user.

5.3 Feature Processing

DNN does not perform explicit feature crossing — instead, learns interactions via DNN.

Single-value features are straightforward. Multi-value features are trickier — variable sample lengths. Solution: Embedding → Pooling + Attention.

6 Embedding

Deep learning in recommendation/search revolves around sparse ID features — primarily via Embedding. Converts “exact match” to “fuzzy lookup” to enhance generalization — maps high-dimensional sparse categorical/ID features to low-dimensional dense vectors.

6.1 Characteristics

Embedding is foundational for NN/FM models in recommendation.
High-dimensional sparse categorical/ID features are “first-class citizens.”
Each feature field has its own embedding matrix.
Embedding converts “exact match” to “fuzzy lookup” → improves scalability.
Differs from NLP Embedding:
- NLP: One word per position → extract row vector from embedding matrix.
- Recommendation: One field → multiple features → merge embeddings via pooling. E.g., App Field: WeChat:0.9, Weibo:0.5, Taobao:0.3 → Embedding = 0.9 × WeChat_vec + 0.5 × Weibo_vec + 0.3 × Taobao_vec.

6.2 Variable-Length Features

MLP requires fixed-length input — but user click sequences vary in length. Solution: Pooling layer.

7 Pooling Layer

Pooling converts embedding vectors to fixed-length vectors — solves variable dimensionality.

7.1 Purpose

Users have multiple interests → two problems:

Multiple categorical/ID features per interest (e.g., clicked goods, watched videos, searched terms → multiple good/shop IDs).
Variable historical behavior counts → multi-hot features → variable-length embedding lists → MLP needs fixed input.

To reduce dimensionality and enable arithmetic, we first embed ID features.

How to model diverse interests? Merge multiple low-dimensional embedding vectors (from IDs) into one vector → user interest representation.

Since MLP needs fixed input, we “merge” into a fixed-length vector.

This “merge” is pooling.

7.2 Implementation

Various approaches:

YouTube DNN: Simple average of watched/searched embedding vectors.
Neural Factorization Machine: Compresses $n$ ($n=$num of features) k-dimensional vectors into one k-dimensional vector — bi-interaction pooling (pooling + 2nd-order interaction).
DIN: Weighted average of embeddings — weights from attention mechanism.
Text Classification: Feed multiple word vectors into RNN — final RNN output represents the “merged” vector. DIEN adapts this with attention-controlled GRU gates.

7.3 DNN

Base DNN uses sum pooling (element-wise sum) or average pooling (element-wise mean), then concatenates all vectors for overall representation.

Sum: For 10 goods with 16-dim embeddings → sum each of 16 dimensions.
Average: Regardless of click count, final vector has same dimension as each good’s embedding.

Base model uses the same user representation vector for any candidate — no differentiation.

7.4 DIN

In Alibaba’s system, each good has 3 feature fields (good, category, shop). Embeddings concatenated → good representation.

Pool good sequence via sum → user behavior sequence representation. Concatenate with other features → MLP input.

MLP input’s embedding vector (excluding candidate) is the user representation.

Base Model’s pooling loses information — DIN uses weighted sum pooling for Diversity (sum alone doesn’t capture diversity — weights help).

DIN uses weighted average of embeddings — weights from attention mechanism.

8 Attention Mechanism

Attention mechanism: For different ads, user historical behaviors have different weights. E.g., user has behaviors A,B,C — for ad D, weights 0.8,0.1,0.1; for ad E, weights 0.3,0.6,0.1. These weights are learned by the Activation Unit.

DIN adds attention to DNN — uses attention for pooling → user interest vector varies per candidate → “thousand faces of interest.”

Goal: Based on user history, mine user interest-candidate ad relationship. Clicks often depend on partial interests — foundation for attention. Both user behavior and candidate ad map to embedding space — their relationship is learned there.

8.1 Problem

DIN’s attention aims to represent user interest for different goods via a fixed-length vector — simple in concept, hard in practice.

Traditional DNN: Embedding → Pooling (sum/average) → fixed-length User Representation → loses information.
User Embedding dimension k → represents at most k independent interests — users have more.
Traditional DNN ignores user-ad relationship — same weights for all ads → same user vector per ad. To express diversity, increase vector dimension — causes overfitting and computational burden.

DIN uses attention-like mechanism to solve this.

8.2 Attention Mechanism

Attention: Model pays different “attention” to user behaviors — “relevant” histories weighted higher, “irrelevant” ignored. Different features have different weights → some dominate prediction.

In video recommendation, DIN adds user history features (e.g., last 20 show/video IDs) → attention network → merges with non-history features in MLP.

DIN uses attention to model local activation — assigns different weights to historical behaviors during user interest representation → Embedding → Pooling + attention.

From backpropagation: Candidate ad activates relevant historical interests → assigns different weights.

DIN’s solution: Represent user interest not as a point, but as a time-varying distribution — multi-modal (peaks = interests, peak height = intensity). For different candidates, interest intensity varies → even in low dimensions, near-infinite expressiveness.

In short: User interest embedding $V_u$ is a function of candidate ad embedding $V_a$ — same user, different ads → different $V_u$.

Where:

$V_i$: embedding of behavior ID i (e.g., good_id, shop_id).
$V_u$: weighted sum of all behavior IDs → user interest.
Candidate ad affects each behavior ID’s weight → Local Activation.
Weight: contribution of behavior ID $i$ to total user interest embedding for candidate ad $V_a$. In practice, weight = Dice activation output, input = $V_i$ and $V_a$.

8.3 Implementation

DIN cannot directly use standard attention — user interest representation must vary per candidate.

Local Activation Unit adapts NMT’s attention — learns candidate ad-user history relationship → relevance scores (weights) → weighted sum of history → user interest representation. User shows different interest representations per ad — same history, different weights.

During pooling, candidate-related goods get higher weights, unrelated get lower — attention idea. Compute attention scores via interaction between candidate and each good in sequence — input: embeddings and outer product. Different candidates → different user vectors → more flexible.

DIN’s user interest vector for candidate ad:

\[V_u = \sum_{i=1}^{H} w_i \cdot e_i\]

Where:

$e_i$: user U’s historical behavior embedding (e.g., good_id, shop_id), length H.
$V_u$: weighted sum of all behavior embeddings → user interest.
$V_a$: candidate ad A’s embedding.
$w_j$: weight of $e_j$.
Weight: contribution of behavior ID j to total user interest embedding for candidate ad $V_a$.
In implementation, weight $w_j$ = function output (Activation Unit) → Dice activation → $g(V_i, V_a)$, input = $V_i$ and $V_a$.
Candidate ad affects each behavior ID’s weight → Local Activation.
$a(\cdot)$: feed-forward network → outputs local activation weights → multiplies user vector.

Result: User interest vector $V_u$ varies per ad → “thousand faces of interest.” E.g., user bought milk and swimwear — shown goggles → swimwear memory activated; shown diapers → milk memory activated.

In DIN attention, user interest vector $V_u$ is weighted average of historical item embeddings — weight $W_i$ of item i determined by $V_i$ and $V_a$ (function $g$). Same user, different candidates → different $V_u$ → “thousand faces.”

Main difference from base model: Activation Unit computes weights via similarity between ad and user embedding → weighted sum of behavior sequence → superior performance.

8.4 Normalization

Typically, attention scores are normalized via softmax — ensures non-negative weights summing to 1.

But DIN paper emphasizes: do not normalize attention scores for click sequences — directly weight-sum with item embeddings → preserves interest intensity. E.g., 90% clothes, 10% electronics in history — predicting T-shirt vs. phone → T-shirt activates most behaviors → larger user behavior vector numerically.

9 Evaluation Metrics

Evaluation uses Alibaba’s GAUC — proven more stable and reliable than AUC.

AUC: Probability that positive sample score > negative sample score. In CTR, used to rank candidate ads per user. But users differ — some inherently high-click. Traditional AUC ignores user-level differences.

GAUC: Computes AUC per user → weighted average by clicks/impressions → eliminates user bias → more accurate model evaluation.

10 Adaptive Regularization

Deep models with sparse inputs → many parameters → overfitting risk.

Traditional L1/L2/Dropout ineffective for CTR. User data follows long-tail law — many features appear rarely, few appear often → adds noise, worsens overfitting.

Simple fix: manually remove low-frequency features — but loses information, threshold arbitrary.

DIN’s solution:

Adaptively adjust regularization strength per feature frequency.
High-frequency features → weaker regularization.
Low-frequency features → stronger regularization.

Improved L2: During SGD, each mini-batch trains only non-zero feature parameters. Adding L2 requires training all parameters — computationally prohibitive. DIN proposes: apply L2 only to features in current mini-batch.

11 Summary

Paper summary:

Users have multiple interests (visited goods/shops). Embed to reduce dimensionality → use pooling (sum/average) to model diverse interests → solves variable-length input → fixed-length user representation vector.
Sum/average loses information → assign different weights per behavior ID → weights determined by behavior ID and candidate ad → Attention → Local Activation.
DIN uses activation unit for Local Activation, weighted sum pooling for Diversity.
Model optimization: Dice activation, adaptive regularization → improved performance and convergence.

References

Building Wide & Deep with NumPy