Personalization Infrastructure at Scale

The systems and data architecture required to deliver meaningful personalization across millions of SKUs and users.

The recommendation system I worked on at AliExpress US served over 50 million monthly users and lifted GMV by about 40 percent. When people ask how, they usually expect an answer about the model. The model mattered least.

What mattered was that we could answer a narrow question reliably: for this user, in this session, on this surface, which twenty items out of millions are worth ranking? Everything upstream of that question is infrastructure, and everything downstream is a tuning exercise. Teams that lead with the model and treat the infrastructure as plumbing consistently underperform teams that do the reverse, and I have watched that play out enough times to state it as a rule rather than an opinion.

Personalization is a retrieval problem wearing a ranking costume

At small catalog sizes you can score everything. Five hundred products, a few thousand users, and a reasonably fast machine, and the whole thing is a ranking exercise. Most personalization advice on the internet is written for this situation.

Past roughly a hundred thousand SKUs, scoring everything is off the table, and the architecture inverts. Now the dominant question is candidate generation: how do you get from millions of items to a few hundred plausible ones, fast, without discarding the item the user would actually have bought? Ranking quality is bounded by retrieval quality. A perfect ranker over a bad candidate set produces a confidently wrong answer.

This is where most of the real engineering lives, and it is the least discussed part of the stack. In practice you end up running several retrieval strategies in parallel and merging them:

• Behavioral co-occurrence: items frequently viewed or bought together, which is cheap, robust, and carries most of the early lift.
• Embedding nearest-neighbor: semantic similarity over item and user vectors, which handles the long tail that co-occurrence cannot reach.
• Attribute and constraint filters: category, price band, availability, shipping eligibility, which are boring and non-negotiable.
• Freshness and exploration slots: deliberately reserved capacity for items with insufficient data, without which your catalog slowly collapses into a few thousand winners.
• Business rule overlays: margin, inventory position, strategic promotion, applied as a separate layer so you can audit what the business asked for versus what the model wanted.

Keeping that last category separate is one of the most valuable architectural decisions you can make. The moment business rules get baked into the model, you lose the ability to explain why an item appeared, and you lose it permanently.

The cold start problem is a catalog problem, not a user problem

Every discussion of cold start focuses on new users. In a large catalog the harder version is new items, and it is harder because it is structural rather than temporary.

A new user resolves quickly. Two or three interactions and you have signal. A new SKU in a catalog of millions may never accumulate enough behavioral data to compete with an established item, because the system that would give it exposure requires exposure to have happened. That feedback loop concentrates GMV in a shrinking set of items and slowly makes your catalog worse, and it does so in a way that looks like success on every metric you are probably tracking.

The fixes are unglamorous. Content-based embeddings that let a new item inherit signal from similar items. Mandatory exploration budget, meaning a fixed fraction of impressions goes to under-observed inventory even though it costs measured conversion. Hierarchical priors so a new item starts with its category's baseline rather than with nothing.

The exploration budget is the one that gets cut first in every organization I have seen, because it is a visible cost with a delayed and diffuse benefit. Defending it is a political problem more than a technical one. I would rather run with a mediocre model and a protected exploration budget than an excellent model without one.

Feature freshness is where the real leverage is

There is a persistent belief that personalization quality is driven by model sophistication. In my experience the single largest lever is how fresh your features are, and the gap is not close.

A simple model reading features that are thirty seconds old will beat a sophisticated model reading features that are six hours old, in almost every commerce context. The reason is that shopping intent is extremely time-local. What someone looked at four minutes ago tells you more about what they will buy in the next ten minutes than their entire purchase history from last year.

This means the interesting engineering problem is the feature pipeline, not the training pipeline. You need session-level state available at request time with single-digit millisecond reads, you need the online and offline feature computation to produce identical values, and you need to detect when they diverge. Training and serving skew is the most common cause of a personalization system that tests beautifully offline and does nothing in production. It is also the hardest failure to notice, because nothing errors. The numbers are just quietly wrong.

If you have to choose between a better model and fresher features, take the fresher features. Then check that your offline and online feature computations actually agree, because the one you assume is fine is usually the one that is broken.

The metrics most teams optimize are the wrong ones

Click-through rate is the default personalization metric because it is dense, fast, and easy to instrument. It is also actively misleading at scale.

Optimizing CTR trains your system toward items that are interesting to look at rather than items people buy, and the gap between those compounds. You will see engagement improve while GMV stays flat, and the system will look like it is working. I have seen personalization surfaces with strong click metrics that were provably neutral on revenue when tested properly.

What I would track instead, roughly in order of how much I trust it:

• Incremental GMV measured against a holdout, not against a pre-launch baseline. Holdouts are expensive and teams hate them, and they are the only measurement that survives scrutiny.
• Catalog coverage, meaning what fraction of your SKUs received meaningful impressions this week. Falling coverage predicts future revenue problems months before revenue shows it.
• Conversion rate by user tenure segment, since aggregate conversion rate hides the fact that your system may be doing great for repeat buyers and nothing for new ones.
• Latency at the ninety-ninth percentile, because slow personalization is worse than no personalization and the tail is what users feel.

Holdouts deserve a specific note. The pressure to shrink or eliminate the holdout is constant, because it is visible foregone revenue. Resisting that pressure is one of the higher-value things a growth leader does, since without a holdout every subsequent claim about the system's impact is unfalsifiable.

Build the boring parts first

If I were standing up personalization infrastructure from scratch today, the order would be: instrumentation, then identity resolution, then a real-time feature store, then candidate retrieval, then ranking. Ranking last, deliberately.

Identity resolution in particular is the step teams skip and regret. If you cannot reliably stitch a user's behavior across devices and between logged-out and logged-in states, every downstream model is learning from fragments. In consumer electronics at Next2Market, where we drove 350 percent GMV growth and an 18 percent conversion rate improvement, a meaningful share of the conversion gain came from finally connecting sessions that had previously looked like separate anonymous visitors. No model change involved. Just correct plumbing.

The reason to sequence it this way is that each layer bounds the one above it. Bad instrumentation caps identity resolution. Broken identity caps feature quality. Stale features cap retrieval. Weak retrieval caps ranking. Investing in the top of that stack while the bottom is broken produces a system that is sophisticated and ineffective, which is the most expensive outcome available.

Personalization at scale is not really a machine learning achievement. It is a data engineering achievement with a model attached at the end. Teams that internalize that ship faster and get more lift, and teams that do not spend two years building something impressive that moves nothing.