* == work done while at Pinterest

Introduction

Large ID embedding tables underpin some of the largest modeling advances at Pinterest, including teacher-model scaling and cross-domain pretraining for ranking and retrieval. These tables are first pre-trained to learn broadly useful Pin representations and then fine-tuned as part of the downstream surface-specific ranking model. Here, we’ll focus on the Homefeed ranking model.

The strength of the embedding tables comes from their enormous capacity to absorb information about billions of Pins and learn rich representations. That power, however, is coupled with fragile training dynamics that have historically constrained fine-tuning to one epoch.

That compromise has become increasingly dissatisfactory; single-epoch fine-tuning prevents us from effectively optimizing our models over small, fresh datasets in retraining, and disproportionately harms effective representation learning of certain components of our ranking model which benefit from longer training, like Vision Language Models (VLMs). A one-epoch ceiling forces every component of our model to learn on the schedule of its most training-sensitive component.

When trained longer, two problems appear:

1. Model quality degrades sharply. In Homefeed ranking, we saw nearly a 6% decline in our offline save metric when extending training from one epoch to two epochs.
2. GPU waste from communication-bound training accumulates. Distributed embedding lookup and synchronization overhead created large idle periods in the training loop, leaving GPUs waiting on communication rather than performing useful computation.

Background

The “one-epoch” phenomenon of large ID embedding tables is well documented in recommendation systems literature. Their enormous capacity, combined with sparse and long-tailed updates, makes it easy for the model to “memorize” individual IDs associated with repeated examples rather than learn representations that generalize.

Prior work at Pinterest had already shown that multi-epoch ID embedding training does not fail in every setting. Taming the One-Epoch Phenomenon in Online Recommendation System by Two-stage Contrastive ID Pre-training demonstrated that embeddings themselves could be pretrained for multiple epochs using a contrastive-only pretraining objective before being transferred to downstream models for finetuning. We were interested in multi-epoch post-training: once the pretrained embeddings are part of the Homefeed ranking model, could we fine-tune them for multiple epochs without catastrophic overfitting?

Related work from Pinterest Ads explored strategies to train longer in post-training, including frequency-adaptive updates and larger learning rates; however, the ID embedding tables in this setting did not exhibit the same catastrophic multi-epoch degradation we observed in Homefeed.

Making multi-epoch training safe

Freezing as a method to identify the problem
To answer why we saw a nearly 6% decline in the offline save metric when extending training to two epochs, we isolated each component to determine whether the whole model was overfitting or whether the regression was concentrated in a particular module. The embedding table emerged as the dominant source of the regression: when we froze the table, the multi-epoch regression disappeared and the result beat the single-epoch trained model.

Over three epochs, we saw a +1.25% gain over the one-epoch model. Although freezing the embedding table immediately unlocked multi-epoch training, it prevented task-specific adaptation in the embedding table. Was task-specific adaptation possible given the fragility of embedding tables? Or were we constrained to only allowing tables to train during pretraining?

Isolating different gradient paths
We attempted to control gradient paths to prevent overfitting. The model uses three types of training objectives:

1. Head losses, which optimize per-action predictions such as save and click.
2. Pairwise losses, which optimize relative ranking between positive and negative items.
3. Contrastive loss, which trains next-item prediction as an auxiliary sequence-modeling objective. Next-item prediction is used only during training and is not an output of the production ranking model.

All three objectives can update the shared embedding table, but their gradients reach it through different paths:

The contrastive objective provides one gradient path to the table through its sequence-oriented next-prediction task. Separately, for the candidate Pin being scored, the head and pairwise objectives can send gradients to the table through two candidate-side paths:

1. Transformer path: The candidate embedding passes through cross-attention and subsequent transformer layers, creating a deep gradient path from the head and pairwise objectives back to the embedding table.
2. Feature cross path: The candidate embedding feeds directly into the MLP or DCN feature tower, creating a shorter path from the head and pairwise objectives back to the table.

We tested a series of gradient isolation strategies:

• Contrastive-only updates: We blocked gradients from the head and pairwise objectives from reaching the embedding table, while keeping the contrastive path active. The table learned only from its auxiliary next-item objective.
• Selective candidate gradients: We selectively re-enabled candidate-side gradients through the transformer path, the feature cross path, or both.

Below, we compare full co-training, frozen-embedding training, and the gradient isolation strategies vs. a one-epoch full trained model.

To summarize the results

• Full co-training allows the embedding table to train alongside the rest of the ranking model and leads to catastrophic overfitting. Contrastive + both is equivalent to full co-training.
• Frozen embedding table freezes the embedding table and allows the rest of the model to train throughout the training cycle; this resulted in the best gains.
• Contrastive-only training only allows gradients from contrastive loss to reach the embedding table and allows co-training; this also solved the overfitting issue but the results were worse than fully frozen embedding tables.
• Contrastive + transformer allows gradients from the contrastive loss and transformer route gradients from pairwise and head losses. This caused mild overfitting.
• Contrastive + feature cross allows gradients from contrastive loss and feature cross route gradients from pairwise and head losses. This resulted in catastrophic overfitting. One possible explanation is that the transformer’s additional structure and depth act as a form of implicit regularization.

Although gradient isolation prevented overfitting, we could not beat the model trained with a fully frozen embedding table through this technique.

Decoupling embedding optimization

We further explored decoupling embedding optimization in two settings with different practical goals:

1. The first asked whether multiple passes over a smaller training-data window could replace a single pass over a longer window.
2. The second asked whether we could give a newly added vision-language model more optimization time without allowing the embedding tables to overfit.

Because these investigations used different modeling and data configurations, we report their results separately. However, both pointed to the same pattern: freezing the embedding table early and adapting it later made multi-epoch training effective.

Reusing a smaller training-data window
The current Homefeed ranking model runs on a 3-week single-epoch training setup, fine tuning a pretrained foundation model backbone that is trained on data across 5 product surfaces including Homefeed. We explored adopting a shorter fine-tune window with multi-epoch training to trade in fresher fine-tune data for the ability to use a fresher pretrained backbone. Reducing the fine-tune window would also enable us to experiment faster on other workstreams that require feature logging.

The first investigation treated a 20-day, one-epoch full co-training run on the Homefeed ranking model as its control. Below, we compare the save offline metric lift of different training data windows, training horizons, and update strategies.

Initial experiments on doing full co-training saw regressions despite the fresher pretrained backbone, because of the embedding table overfitting. Freezing the embedding table prevented that sharp degradation. With a 10-day training-data window, the fully frozen configuration remained near the baseline after two epochs and slightly improved after three.

We then tested out freezing the embedding table for different training windows, and found the timing of embedding adaptation mattered. Freezing the table up to the last epoch produced a 0.49% improvement; on the other hand, co-training the table during the first epoch and freezing it for the rest of training produced a 0.80% regression. This delayed update strategy remained positive as we expanded the training window, but gains saturated around three epochs. The same pattern held with a 20-day training window, leading to a 1.16% lift. On three-epoch runs, we observed that freezing the embedding tables up to the last epoch performed much better than freezing them before.

We hypothesized that update timing mattered because the pretrained embedding table could adapt too quickly to repeated training examples. If the embeddings became trainable from the beginning, they could absorb the learning signal before the rest of the network had learned how to use the pretrained representations effectively. Keeping the table frozen initially might allow the surrounding network to learn more generalizable patterns, after which embeddings could safely adapt.

To test this hypothesis, we held the training-data window, training horizon, and duration of embedding updates fixed, while progressively shifting the one-epoch update window later within a two-epoch run:

The results showed a roughly linear, monotonic relationship between update timing and model performance that is consistent with our hypothesis. Based on this finding, we adopted the strategy of keeping the table frozen for one epoch and allowing co-training in the second epoch over a small training data window. The smoothness of this trend suggests that update timing may be a tunable optimization dimension.

Giving the VLM more optimization time

A parallel investigation was motivated by giving a newly added VLM more time to optimize. The VLM adapted more slowly than the established ranking components, but as with the one-epoch model, extending unrestricted full co-training caused overfitting. Similar to the above results, the best update strategy for the VLM-based model was the delayed scheme. The below results are compared against a one-epoch VLM model where the model and embedding table co-trained.

Hence, we discovered a recipe to allow task-specific adaptation in post-training for embedding tables that not only beat the one-epoch model, but also the fully frozen multi-epoch model.

Making longer training efficient

To make extended training practical, we had to reduce the amount of time spent waiting on sparse communication, synchronization, compilation overhead, and data delivery. The largest initial bottleneck was distributed embedding lookup.

Semi-synchronous training

Our embedding tables are sharded across GPUs. Looking up sparse IDs requires the system to redistribute IDs to the devices that own the corresponding table shards, perform the lookup, and return the resulting embeddings. This introduces cross-GPU all-to-all communication, whose cost grows as our embedding tables, batches, and context lengths scale.

For the production Homefeed ranking model, distributed embedding lookup alone could consume more than 15% of total training time. More importantly, much of that time appeared as a bubble in the step loop: GPUs waited while data moved between devices instead of executing model kernels.

We addressed this with a semi-synchronous training pipeline that overlaps embedding communication for the next batch with computation for the current batch. The pipeline has three main components:

• a lookahead dataloader that prepares the next batch’s sparse IDs early
• a prefetching forward wrapper that launches the next lookup immediately after the current forward pass; and
• an embedding prefetcher that coordinates work across CUDA streams and events.

Conceptually, the schedule is:

1. Run the forward pass for batch B − 1.
2. Launch ID redistribution and embedding lookup for batch B on a separate prefetch stream.
3. Continue the current step’s useful computation while that communication runs in the background.
4. Synchronize only when batch B needs its embeddings. In steady state, the lookup has already completed by that point.

TorchRec’s pipelining utilities did not integrate cleanly with our custom trainers and modules; a custom implementation gave us the flexibility we needed without forcing a larger rewrite of the training stack.

Bounded embedding staleness

The pipeline is semi-synchronous because the embeddings prefetched for batch B are read before the backward pass for batch B − 1 finishes. As a result, batch B uses embeddings that were last updated by batch B − 2: a bounded delay of one embedding update.

Empirically, we observed no model-quality degradation. Two factors likely explain why:

• Consecutive batches have little overlap in their sparse IDs, so most prefetched rows would not have been modified by the immediately preceding batch anyway.
• The update schedule is approximately analogous to gradient accumulation with a step size of two for the embedding table.

To preserve correctness and performance, we had to ensure that:

• Prefetch reads did not race with backward writes on another CUDA stream
• TorchRec autograd continued to update the intended sparse rows
• Embedding collectives did not collide with backward all-reduce operations, because NCCL calls could not be overlapped safely in our execution path
• The pipelined loop remained compatible with torch.compile rather than introducing new graph breaks

Embedding lookup optimizations

Overlapping communication hides latency, but reducing the amount of communication and kernel work is even better. We made several additional changes to the embedding path.

• Fused training-ID lookup: The prefetch path previously issued multiple sharded lookups per step, such as candidate signatures and deduplicated user-sequence signatures. Each lookup paid the fixed cost of collective setup, including split exchange and payload all-to-all operations. We now concatenate these IDs into one flat request, perform a single sharded lookup, and partition the returned embeddings afterward, reducing lookups by up to a factor of three.
• Specialized singleton-row lookup: Many embedding-bag operations were degenerate bags containing exactly one ID per row. The generic ragged-bag path still performed offset processing and reduction work that these inputs did not need. We added a specialized cuEmbed operation that uses a direct gather with a custom backward implementation. Registering a fake implementation also allows torch.compile to trace the operation instead of breaking the graph. Unit tests confirmed exact gradient parity with the generic path.
• Optimized small embedding table lookups: In addition to our large ID embedding tables, we also leverage smaller lookup tables in the model, which contain only tens of rows but can receive millions of indexed lookups per step. The default backward kernel used an atomic scatter-add, creating heavy contention on a small number of rows. We replaced it with an equivalent dense formulation: one_hot(indices)ᵀ @ output_grad. This turns the weight-gradient computation into a GEMM, which the GPU executes much more efficiently. The forward pass remains a standard gather, and tests verified bit-equivalent results.

Together, these changes reduced both collective latency and the number of kernels required for sparse training.

Amortizing optimizer communication

After the lookup and normalization bottlenecks moved, optimizer communication became a larger share of the step. Sharded optimizer state still required expensive broadcasts of updated parameters, leaving the GPUs idle while parameters were exchanged.

We amortized that cost with gradient accumulation, performing optimizer updates less frequently across multiple microbatches. TorchRec’s backward-fused embedding optimizer does not natively follow the same accumulation schedule, so we scaled the embedding learning rate to approximate the corresponding accumulated update.

Conclusion

The modeling and system insights allowed us to enable multi-epoch training in production, leading to +1.09% online saves. The best schedule we’ve discovered thus far allows the rest of the network to train first before enabling embedding adaptation. Future work could explore table-specific learning-rate schedules or treat embedding adaptation as a limited optimization budget, alternating between trainable and frozen phases to capture task-specific gains while controlling overfitting.

More broadly, these findings suggest that a single optimization schedule may not be appropriate for every model component, particularly when components differ in capacity, initialization, and convergence rate. The embedding table provided the clearest evidence of this. Whether a broader decoupling could produce additional benefits remains an open question.

Acknowledgements

Thanks to the teammates and reviewers who helped with the profiling, gradient-path investigations, experiment reviews, and repeated iterations that made both the multi-epoch and systems work possible.

Safe and Efficient Multi-Epoch Embedding Training was originally published in Pinterest Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

In our multi-part blog series, we have discussed why we chose Ray as a last mile data processing framework, how we integrated Ray into our existing ML Training Infrastructure, and how we enabled efficient Batch Inference at scale. In this blog post, we will discuss how we were able to apply Ray to another critical part of our ML Infrastructure stack: Model Evaluation.

In the Ray Batch Inference Blog Post, we ended on a section on accumulators (counters that would allow us to calculate metrics like ROC-AUC and cross entropy). This post is where those accumulators pay off: a model evaluation platform that combines the strengths of Ray and Apache Spark to compute metrics efficiently at scale.

Model evaluation may appear to be a relatively straightforward step after inference or training. But as ML workflows scale, evaluation quickly becomes an infrastructure challenge. Systems that run many experiments in parallel, such as feature importance, must efficiently compute and aggregate metrics over large volumes of inference data, as shown in Diagram 1.

Later in this post, we’ll show how we amortize the metrics compute step, overlapping it with inference for minimal added runtime cost, cutting end-to-end evaluation time by up to 32% while scaling linearly as we run more experiments.

Terminology

• Feature Importance (FI): A technique for quantifying how much each input feature contributes to a model’s predictions. In this post, we will reference a perturbation-based method where we perturb one or more columns, run inference, and compare the results against an unperturbed dataset.
• Experiment / baseline: One perturbed run vs. the unperturbed run.
• Model head: One output of a multi-task model.
• Batch: an arbitrary, compute-sized group of rows. This is the unit of data that one Ray task processes.
• Bucket: a batch that is group-preserving. It contains one or more complete segments, and no segment may span buckets.
• Segment: The semantic unit over which a group-wise metric is defined, such as a request.
• Serde [serialization/ deserialization]: The process and cost of writing data out and reading it back.

Background

Traditionally, evaluation and inference are two separate stacks stitched together. Diagram 2 below contrasts that unfused pipeline against the fused pipeline we’ll build up over the rest of this post.

Concretely, this evaluation step is typically implemented as a Apache Spark job that computes metrics across one or more inference experiments. As shown in the diagram above, we previously used Ray inference to write predictions to S3, then have a follow-up compute job to reread all this data and compute the metrics.

For a FI run, this approach introduces two sources of overhead:

• Cluster startup: Acquiring and warming up a cluster for the metric job.
• Data movement: Writing predictions to S3 and reading them back to compute metrics.

Because inference is done in batches, and metrics typically depend on all of the inference results, the unfused approach can be used to calculate any metric. To save the overheads, we need to fuse the stages, but this also implies that we need to be able to calculate metrics on batches of rows. Therefore, we need a metric platform that treats decomposability as a first principle.

Metrics fall into three groups:

Metric Contract

We define a common metric contract that captures each metric’s compute layout and provides a consistent interface across execution frameworks. We express this interface concretely as an Abstract Base Class.

from abc import ABC, abstractmethod

class ReducibleMetric(ABC):
    @abstractmethod
    def dataset_preprocessor(self, ds):
        """Prepare the dataset to satisfy the metric's data layout requirements."""
        pass

    @abstractmethod
    def compute_partials(self, batch):
        """Compute a partial result from one batch"""
        pass

    @abstractmethod
    def reduce_partials(self, partials):
        """Combine partial results into the final result."""
        pass

Decomposable and holistic metrics can calculate partials from arbitrary batches because they do not require related rows to be colocated.

Group-wise metrics introduce an additional constraint. Before handing data to Ray, we organize complete segments into buckets. A bucket is therefore a group-preserving batch: it contains one or more complete segments, and no segment crosses a bucket boundary. This lets us independently tune bucket size for efficient execution without changing the semantic unit of the metric.

Common Metrics found in Recommender Systems

We will walk through three metrics, MAE, ROC-AUC, and Precision@K; one from each group above. We will use the following dataset to compute each metric.

For the examples below:

• model_output contains the predictions, and label_continuous contains the target values.
• label_binarized is derived as 1 if label_continuous ≥ 0.5 and 0 otherwise.
• Batch A contains ReqA, and Batch B contains ReqB. This split lets us demonstrate batch-level computation and aggregation.

MAE (Decomposable)
Mean Absolute Error measures the average absolute difference between predictions and their actual values.

MAE_b = (1/n_b) · Σ_{i ∈ batch b} |pred_i − label_continuous_i|

Unlike MSE/RMSE, MAE does not penalize large errors as heavily.

Per-row absolute errors:

Global: Σ = 1.45, N = 8 → MAE = 0.18125.

Batch decomposition:
Batch A partial (Σ|err| = 0.65, n = 4)
Batch B partial (Σ|err| = 0.80, n = 4)
Merge (0.65 + 0.80) / (4 + 4) = 0.18125

class MAE(ReducibleMetric):
    def dataset_preprocessor(self, ds): 
  # No dataset preparation needed. Rows are independent.
 return ds

    def compute_partials(self, batch):
  # Emit summed absolute error and row counts from each batch.
        error = np.abs(batch["model_output"] - batch["label_continuous"])
        return error.sum(), len(error)

    def reduce_partials(self, partials):
  # Sum partials and divide once globally. 
        error, count = np.sum(partials, axis=0)
        return error / count

ROC-AUC (Holistic)
ROC-AUC is a binary classification metric which measures the ability of a model to distinguish between classes across different decision thresholds.

Method 1 — Global Sort (what sklearn does)

Global Sort

We sort the rows by model_output in descending order, keeping each prediction paired with its corresponding binary label.

Cumulative Sum

After sorting, we calculate cumulative true-positive (TP) and false-positive (FP) counts and prepend 0 to both vectors.

Calculate the TPR and FPR

We calculate the TPR at each threshold by dividing the TP vector by the number of positive labels.

We calculate the FPR at each threshold by dividing the FP vector by the number of negative labels.

Calculate the Trapezoidal area under TPR/FPR

We calculate the AUC of TPR/FPR to get the final ROC_AUC value.

Sorting our eight rows by score:

With 4 positives and 4 negatives, the area works out to AUC = 0.9375 (or 15/16 positive/negative pairs are ranked correctly).

We observe that ROC_AUC using this method is calculated over the entire dataset due to a global sort. We could break the dataset into batches, calculate ROC-AUC for each batch, and average the results. However, this does not produce the same result as a global ranking.

Consider the following calculation:

• Batch A (ReqA) = 1.00
• Batch B (ReqB) = 0.75
• Average = 0.875, not 0.9375

Averaging a ranking number across batches is wrong, because AUC depends on how rows rank against all other rows, not just those in their batch.

We could force a real global sort in Apache Spark, but doing that across all metrics, model heads, and experiments is exactly the cross-stack cost we’re trying to avoid.

Method 2 — Confusion Matrix

Consider the standard method to approximate ROC_AUC.

Define thresholds

Pick a fixed set of thresholds in [0, 1].

Calculate Confusion Matrix

Calculate the confusion matrix (TP, FP, TN, FN) for each threshold

Calculate the final metric

Calculate the TPR/FPR and finally derive the AUC.

This method can be calculated batch-wise because the global confusion matrix can be calculated from the batch confusion matrices.

We can create the following confusion matrices per batch and then sum:

From this approximate example, when we derive the area under the curve, we get 0.875. If we use more thresholds, for example [0.00, 1.00] with step size of 0.01, we would get 0.9375. Note: This does not mean that threshold-based aggregation generally produces the exact global AUC. In this toy example, a step size of 0.01 recovers the exact value because the prediction scores align with the threshold grid.

class ROCAUC(ReducibleMetric):
    def dataset_preprocessor(self, ds): 
 # No dataset preparation needed. Rows are independent.
return ds

    def compute_partials(self, batch):
  # Emit confusion matrix across thresholds. 
        pred = batch["model_output"]
 truth = batch["label_binarized"].astype(bool)

        positive = pred[None, :] >= self.thresholds[:, None]
        return np.stack([(positive & truth).sum(1), (positive & ~truth).sum(1),
                       (~positive & ~truth).sum(1), (~positive & truth).sum(1)],1)

    def reduce_partials(self, partials):
  # Sum confusion matrices and add ROC endpoints.
 # Integrate in ascending FPR order. 
        tp, fp, tn, fn = np.sum(partials, 0).T
 tpr, fpr = tp / (tp + fn), fp / (fp + tn)
        return np.trapz(np.r_[0, tpr[::-1], 1], np.r_[0, fpr[::-1], 1])

Precision@K (Group-wise)
Precision@K is a common metric used for ranking systems. It measures the fraction of the top K ranked items that are relevant.

This metric is calculated within an individual segment of data, such as a session or request. A segment may be identified by a single key or a composite key.

The procedure for calculating Precision@K is as follows.

Group/Rank Dataset
Group the dataset by segment and sort by predicted score.

Window and Count
Select the top K items and count the relevant items within that window.

Calculate Metric
Divide the number of relevant items by K to calculate Precision@K for each segment. Average the per-segment scores to obtain the final result.

On our data with K = 2:

Average = (1.00 + 0.50) / 2 = 0.75.

We observe this metric has a similar decomposition pattern. We can break it down into subproblems where we look at each segment, calculate the metric, then average final scores of all subproblems.

We have two problems when trying to scale this approach:

1. The subproblem (batch_size) is tied to the size of each segment. This is inefficient as each executor may receive a small and variably sized amount of data.
2. If we combine segments together, we must ensure that no segments span a bucket boundary.

Our goal is to keep every row for a given segment in the same bucket while ensuring that each bucket contains enough segments to saturate the compute resources.

More succinctly, we want all rows within each segment to be colocated and each bucket to be sufficiently large. Since we are operating at the global level, we can define this as an Apache Spark procedure. It keeps all rows for each segment together, then packs complete segments into buckets large enough to use each Ray worker efficiently without splitting a segment across buckets.

class PrecisionAtK(ReducibleMetric):
    def dataset_preprocessor(self, ds):
  # Colocate each segment and pack into buckets.
        return bucket_complete_segments(
            ds,
            by="request_id",
        )

    def compute_partials(self, bucket):
  # Rank within each segment and calculate top-K score.
        scores = []
        for segment in group_by(bucket, "request_id"):
            ranking = np.argsort(
                -segment["model_output"]
            )
      # Assume each segment contains at least K items.
            top_k = ranking[:self.k]
            relevant = segment["label_binarized"][top_k].sum()
            scores.append(relevant / self.k)

        return sum(scores), len(scores)

    def reduce_partials(self, partials):
  # Sum segment scores and divide by segment count.
        score, count = np.sum(
            partials,
            axis=0,
        )
        return score / count

Fusing Metrics with Inference

In the upper figure in diagram 3, we show a typical Sequential Inference-Metrics multi-experiment pipeline. We run model inference over all the experiments, serialize, and persist those results along with the carryover labels. Note this inference dataset scales with the input data and the number of experiments. Next, we deserialize this data and compute metrics. GPU inference is typically more resource intensive than metrics computation. This design causes the CPU workers to sit idle during inference time.

The lower figure in diagram 3 illustrates a fused approach where we pipeline the metric computation once inference completes for each bucket. The shared object store minimizes serde overhead. This enables us to effectively overlap metric computation with inference, reducing total runtime.

Results

We benchmarked two setups against a Pin Ranking Model, sweeping across three dataset sizes (Small ≈ 30K, Medium ≈ 300K, Large ≈ 3M rows) and several experiment counts:

• Unfused: Ray inference, then a separate Ray metric job.
• Fused: Ray inference with metrics in the same pass.

When we gather metrics during inference, we do not see any meaningful runtime increases. The time measured here includes startup time (acquiring the cluster) and runtime (actual computation).

Furthermore, we can see that runtime grows linearly with experiment count while startup stays flat.

As the number of experiments increases from 1 to 300 on the Large dataset, fused runtime grows approximately linearly (~43 min fixed + ~41s per experiment), while startup time remains roughly constant. The separate metric stage (~11s per experiment) is work that fusion absorbs through overlap with inference, keeping the marginal cost per experiment small and predictable.

What’s next

Other upcoming topics include:

• Ray Tune at Pinterest
• Improving the data loader performance in Ray
• Fault-tolerant Ray Batch Inference and Training
• Feature Importance on Ray

Acknowledgments

Related Pins: Shivin Thukral, Zilin Ren
AI Platform: Rubin Fergerson IV, Qingxian Lai
Data Platform Sciences: Darren Reger, Adam Obeng
Agent Platform: Tan Wang

Scaling Model Evaluation at Pinterest with Ray (Part 4) was originally published in Pinterest Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

Modern recommendation systems, including those used for ads ranking and content discovery, have grown substantially in scale and complexity. This growth stems from both increasingly large model architectures. The resulting memory footprint often exceeds the capacity of a single GPU, making it necessary to shard production models across multiple GPUs using distributed training techniques such as Distributed Model Parallelism (DMP).

This evolution brings challenges to the everyday engineer work on testing and debugging the model code. The bottleneck includes:

• Local Machine Hardware Limits: Because production models require multi-GPU setups, initializing a full model locally on a single-gpu workstation is virtually impossible.
• Input Data Dependencies: When an engineer modifies code inside a specific downstream module, getting realistic input data is a major hurdle. Traditionally, obtaining that input requires initializing the entire multi-GPU pipeline from scratch and executing every preceding module in the network — wasting immense amounts of time and computation resources.
• Slow Iteration Feedback Loops: Engineers face prolonged iteration loops when attempting to verify sub-module changes, as every code edit requires re-executing or re-initializing heavy distributed trainer setups.

To eliminate these bottlenecks, we build ModuleLens, which pivots from a legacy “Model-First” paradigm to an isolated “Module-First” engineering framework. The following table contrasts these two operational paradigms across key engineering metrics:

Under the module-first paradigm, engineers can isolate any given module, avoid multi-GPU model initialization entirely, and reconstruct and profile it locally within seconds. In addition to accelerating developer workflows, this modular foundation offers a fast, structured execution environment that empowers autonomous AI agents to analyze, refactor, and optimize PyTorch code independently.

Architecture

The architecture of ModuleLens is organized into three primary components: ModuleTracer for capturing runtime data, Module Repo for central storage, and a Consumption Layer serving human interfaces and AI-based tooling. The following diagram depicts this structure.

ModuleTracer

ModuleTracer is designed to capture and record all essential data — including input and output tensors, model parameter weights, and initialization arguments. This comprehensive tracking allows any target PyTorch module to be reconstructed seamlessly within the consumption layer, enabling offline execution with captured inputs and validation against expected outputs.

To carry out trace capture, ModuleTracer relies on several key mechanisms:

• Non-Intrusive PyTorch Forward Hooks: Attaches listeners directly to PyTorch modules to seamlessly log parameter weights (state_dict), input and output tensors, and execution metadata during runtime.
• In-Memory Module Init Tracking: Ensures target modules can be reconstructed by dynamically tracking initialization arguments (__init__ args and kwargs). It achieves this in memory by inspecting imported classes and patching Python’s __import__ function.
• Resource-Efficient Tracing Loops: Reduces GPU compute overhead by registering early_stop_func and post_train_step_func callbacks. These wait for model warmup, capture data across a single training step, and terminate early.
• Optimized Handling of Large Tensors: To efficiently profile large tensors, ModuleTracer automatically enables shape-only tracing (log_shapes), logging solely tensor dimensions so that synthetic inputs can be reconstructed during restoration. Additionally, if an input tensor is shared across multiple modules, ModuleTracer retains only a single copy and uses reference links for all other instances.

ModuleRepo

Extracted module state is persisted within a queryable backend built on top of the Weights & Biases (WandB) Registry. All captured modules are stored into one registry and organized around Collection and Versioned Artifacts:

Collection Indexing: Target collections are indexed using an MD5 hash of the model and module names: hash(model_name, module_name). Additionally, top-level trace executions log overall run history to special collections structured as registered_model_xxx, indexed directly by model name. The registry supports both attribute-based and tag-based search filtering.

Versioned Artifact Payload: Every captured Module produces a distinct version entry containing two standardized artifacts:

• JSON Metadata File: Contains runtime execution metadata including the model and module metainfo, and the autocast info.
• Binary State Payload: Serialized via torch.save, containing the precise input tensors, baseline output tensors, and module parameter state_dict.

Consume Layer

ModuleLens exposes its capabilities through two tailored surfaces to consume the traced data:

Human Engineers: Interactive Jupyter Extension and IPython CLI for hands-on debugging, exploration, and manual profiling.

AI Agents: Agent Skills that allow LLM agents (e.g., Claude Code, Cursor) to autonomously discover, rebuild, benchmark, and profile modules without human intervention.

The consume layer provides the following capabilities:

The screenshot below illustrates the command-line interface available to users.

Use Cases

Ever since its deployment across our Ads and Core models, ModuleLens has also been adopted for various internal tools built by our engineering teams.

Training Analyzer

Evaluates module-level metrics across different architectures. Through hierarchical visual representations of latency, FLOPs, and memory usage across the module tree, engineers can rapidly identify training bottlenecks.

KernelAgent

An LLM-driven agent designed to generate high-performance GPU kernels. It leverages captured ModuleLens data to benchmark, profile, test, and validate the accuracy and speed of generated kernels.

Scaling Law ROI Tools

Evaluates the return on investment (ROI) for model scaling. By utilizing ModuleLens to track training and serving FLOPS, activation and parameter memory usage, and execution throughput, it helps assess infrastructure costs against model scaling gains.

Looking Ahead

Our immediate priority is stabilizing the core system and fostering adoption across machine learning teams. Concurrently, we are enriching our Agent Skills repository to furnish AI coding agents with advanced optimization capabilities, including automated Triton/CUDA kernel synthesis as well as autonomous training and serving optimization workflows. By building reliable, execution-friendly interfaces for both developers and autonomous agents, ModuleLens optimizes human engineering velocity while laying the groundwork for scalable, fully automated performance optimization.

Beyond optimization, ModuleLens can serve as an execution substrate for model autoresearch and RSI (recursive self-improvement). By giving agents reproducible access to isolated modules, captured inputs and outputs, and trustworthy performance signals, it enables them to formulate experiments, modify model components, benchmark results, and retain successful changes in a tight feedback loop. Over time, this foundation can support systems that continuously improve both model quality and the tools and strategies used to optimize the model.

Acknowledgement

This project benefited from the valuable contributions of many individuals. We express our gratitude to Archer Liu, leader of Product ML Infra (PMLI), for his leadership and support, Chen Yang (ATG) for insightful design discussions and feedback; Abhinav Naikawadi (HomeFeed) and Paulo Soares (Ads ranking) for leading testing, adoption, and advocacy efforts; and Jiasong Sun, Xinyi Zhang, Diane Li and Hongze Zhao (Product ML Infra) for their contributions to code development and review.

Introducing ModuleLens: Zooming into PyTorch Modules for Human and Agentic Engineering was originally published in Pinterest Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Pinterest serves 640 million monthly active users worldwide, and our backend services span hundreds of thousands of instances running code written in Java, Python, C++, Go, Node.js.

Developing observability on bottlenecks aka Profiling is a key step to driving data driving infrastructure efficiency optimization efforts for Pinterest. At our scale, even a 1% efficiency gain translates to a meaningful impact to our profit margins.

This post describes how Pinterest partnered with Intel to drive an open source first approach to developing profiling i.e. gProfiler and adopt, harden, extend it by contributing — (1) Dynamic Profiling, a heartbeat-based remote profiling layer that lets teams trigger profiling on-demand; (2) on-demand Intel PerfSpect collection for hardware telemetry like cache behavior, memory bandwidth, and CPU stalls; (3) wall time profiling to capture latency from threads blocked on I/O, locks, or RPCs that CPU flame graphs miss; (4) PMC flame graphs that tie specific functions to hardware events such as L3 misses and branch mispredictions; and (5) production-hardening fixes that keep the agent efficient at fleet scale.

The Profiling Challenge at Scale

Profiling at scale requires many features like continuous profiling, aggregation, multi-host profiling, Kubernetes integration, and industry-level support. Meanwhile, engineers were using disparate tools — perf, async-profiler, py-spy_ — each requiring manual setup and producing siloed results.

We needed a unified profiler that could:

• Support all our language runtimes under a single user interface
• Scale from a single instance to a fleet of hundreds of thousands of instances
• Provide both continuous and on-demand profiling
• Integrate hardware performance counters for deep bottleneck analysis
• Remain fully open source to avoid vendor lock-in and limit collaboration

Why gProfiler

gProfiler is a continuous profiler originally developed by Granulate (a former Intel subsidiary) and open sourced under Apache 2.0 as an outcome of collaboration between Pinterest and Intel. Unlike single-runtime profilers, gProfiler orchestrates multiple language-specific profilers and analyzer — perf, async-profiler, pyPerf, py-spy, rbspy, phpspy, and others — to produce unified Flamegraphs that capture CPU activity across all processes on a host, regardless of language or runtime. The table below shows the profilers gProfiler orchestrates for each supported runtime:

By merging runtime-specific stacks (e.g., Java method names from async-profiler) with system-wide native stacks from perf, gProfiler produces flame graphs that show the complete picture — from application code down through native libraries and into kernel routines.

Critically, gProfiler is fully open source, aligned with Pinterest’s strategy of building on and contributing to open source infrastructure. The entire pipeline — from agent to storage to visualization — is free of proprietary dependencies, and Pinterest can fork, patch, or extend any component.

The story of gProfiler open source

Pinterest originally tried to develop an internal profiling tool called ProfilerX to consolidate CPU profiling across our fleet. It quickly became clear that the engineering effort needed to build all of the capabilities we would want from scratch would be considerable. The first and foremost capability we were looking for was continuous / always on profiling. This capability was critical to our journey of multi-year infrastructure efficiency.

We had a clear decision to make Build or Buy or Partner. Trying to solve problems that are shared across multiple companies made it clearer that building one more closed source profiler project was not the answer. We should partner rather than pursue just a Build or Buy strategy. The level of profiling insights that a leading hardware vendor like Intel could provide was significantly ahead of what we could build on our own as was evident from the decades of capabilities Intel developed in Intel vTune™. Understanding and interpreting architecture specific counters anyways requires talking to the creators.

Beyond hardware Intel has demonstrated decades of clear leadership in software optimizations and open source contributions, therefore we decided to partner with Intel on this. This first and obvious choice was vTune™ but there were several gaps, first vTune was never developed for a cloud first environment and it had a significant customer base that Intel needed to continue to support therefore opensourcing / rebuilding it would been very disruptive to existing users and didn’t make sense for us to pursue.

Intel’s acquisition of Granulate presented a cleaner opportunity to start something more purpose built for the cloud. We really liked the ability for Granulate to continuously profile workloads. After months of discussion, in late 2023 Pinterest requested Intel to release gProfiler agent and gProfiler Performance Studio backend under a permissive Apache 2.0 license — enabling any organization, including Pinterest, to self-host and extend the full profiling stack. Intel generously agreed, recognizing the broader benefit to the customer base. The vision was to make this profiler tooling easily accessible so we get down to solving optimizations together. Intel with its ability to optimize vertically from silicon to application runtime allowed us to get one step closer to our optimization and efficiency goals.

Once Intel open sourced gProfiler and gProfiler Performance Studio on March 11, 2024, Pinterest began deploying gProfiler agents to our fleet and standing up the backend infrastructure internally.

To deploy gProfiler at scale we addressed critical reliability issues: Pinterest and Intel collaborated closely to harden gProfiler for production use at fleet scale. Together, we addressed critical reliability issues spanning agent resource consumption, disk usage, multi-terabyte ingestion, and build pipeline efficiency — and contributed the resulting fixes back upstream, improving gProfiler’s reliability for the broader community. We cover this reliability journey in detail later in this post.

Once the agent was running reliably across our fleet, Pinterest contributed the Dynamic Profiling feature to upstream gProfiler — system that allows developers to request on-demand profiles at fleet scale.

This capability also unlocked additional features — wall-clock time profiling, PMU hardware event-based flame graphs (intel/gprofiler#1014), and Intel PerfSpect integration (intel/gprofiler#959) — all of which can be dynamically configured to provide deeper profiling insights when needed.

gProfiler Open Source Architecture

To deep dive into Pinterest’s dynamic profiling contributions, let’s first review gProfiler’s architecture. The system consists of two components: the gProfiler agent, which runs on each node where profiling data is collected, and gProfiler Performance Studio, a backend that aggregates, stores, and visualizes profiling data across the fleet.

gProfiler Agent

The gProfiler agent is an orchestrator that coordinates multiple language-specific profilers on a single host. When it starts, it follows a well-defined profiling cycle:

1. Initiation: The gProfiler orchestrator initiates all configured profilers — perf for system-wide native profiling, plus runtime-specific profilers like async-profiler (Java), PyPerf (Python eBPF profiler), py-spy (Python fallback profiler), rbspy (Ruby), dotnet-trace (.NET), and phpspy (PHP).
2. Process discovery: Each profiler discovers eligible processes by scanning /proc/*/maps for language-specific library signatures (e.g., libjvm.so for Java, libpython*.so for Python) or by matching executable names (e.g., node for Node.js). This detection runs every profiling cycle and takes approximately 50ms to scan thousands of processes.
3. Profiling session: Each profiler attaches to its target processes using the appropriate mechanism — JVMTI for Java, ptrace for Python/Ruby/PHP, eBPF for PyPerf, EventPipe for .NET — and samples stack traces at the configured frequency (default: 11 Hz).
4. Stack trace output: Each profiler writes its collected stack traces to output files, with language-specific frame suffixes (_[j] for Java JIT, _[p] for Python, _[rb] for Ruby, _[k] for kernel, etc.).
5. Merge: The orchestrator parses all stack trace outputs and merges them. Runtime-specific stacks are scaled to match the perf sample counts for each process, then substituted in place of the native stacks that perf collected for those same processes. Processes without a runtime profiler (Go, C++, Rust) retain their perf stacks as-is.
6. Upload: The merged, collapsed stack trace file is uploaded to the backend for indexing and visualization.

This design means that a single gProfiler agent produces a unified view of CPU activity across all languages running on a host, without requiring any code changes to the profiled applications.

gProfiler Performance Studio (Backend)

The gProfiler Performance Studio is the backend infrastructure that receives, stores, and serves profiling data. It is structured as a set of microservices:

Legend : Solid arrows are the continuous data plane (profile upload → AWS S3 → AWS SQS → indexer → ClickHouse → flame graphs). Dashed arrows are the dynamic control plane (UI creates a profiling request; agent fetches start / stop profiling requests via heartbeat).

• Backend (Webapp): The central web application that exposes all APIs to the frontend and API users. It receives profiling data uploads from agents, stores raw profile files in S3, enqueues indexing tasks in Amazon Simple Queue Service (SQS), and reads/writes service metadata in PostgreSQL.
• Frontend UI: A React-based interface for exploring flame graphs, filtering by service/host/container/time range, and comparing profiles across time periods.
• Indexer: A Go service that consumes events from SQS, downloads raw profiling data from S3, and bulk-inserts it into ClickHouse. It handles the transformation from collapsed stack format into the columnar schema optimized for flame graph queries.
• FlameDB REST: A Go query service that sits between the frontend and ClickHouse. It constructs flame graph data by querying the appropriate resolution tier (raw, 1-minute, 1-hour, or 1-day aggregates) based on the requested time range, and performs frame-level language identification using pattern matching on stack frame names.
• Logging Service: Collects and centralizes logs from agents for debugging and operational monitoring.

The backend uses S3 for durable profile storage, SQS for decoupling ingestion from indexing, PostgreSQL for service metadata and configuration, and ClickHouse as the columnar analytical store for fast flame graph queries across billions of stack trace samples.

This open source architecture gave Pinterest a strong foundation to build upon. Our contributions focused on extending it with dynamic profiling capabilities, hardening it for production scale, and integrating hardware-level insights — all contributed back upstream.

Pinterest’s Open Source Contributions with Intel

Dynamic Profiling: Single-Click Managed Profiling at Fleet Scale

The original gProfiler agent only supported a statically configured profiling mode, without the ability to dynamically reconfigure the agent in a given session.

Pinterest operates a large fleet of nodes across AWS EC2 and Kubernetes (Pincompute). At that scale, a static agent posed real challenges. There was no way to target profiling to specific services, no way to adjust the configuration on the fly, and no way to control profiling remotely without SSH access to each node.

The Dynamic Profiling feature we contributed to the open source gProfiler project solves exactly this problem. It offers zero-setup, single-click managed RBAC controlled profiling designed for scalability and ease of management:

• Service owners gain full control to start and stop profiling on demand, enabling targeted detection of service-specific bottlenecks, regressions, and optimization opportunities — all without shell access.
• Performance experts can fine-tune arguments to run various profilers, investigate common bottlenecks across services or processes, and dive deep into hardware-level insights using Intel PerfSpect — also without shell access.

This streamlined, accessible profiling capability has made efficiency and cost optimization a core part of Pinterest’s engineering culture.

Under the hood, we needed the ability to:

• Install the agent on host in a dormant state
• Activate profiling remotely on specific hosts, services, or clusters
• Change profiling parameters (frequency, duration, profiler selection) without redeploying
• Stop profiling instantly if resource pressure is detected

How it works:

1. gProfiler agents are installed on all hosts by default in dormant mode (less than 1% CPU, less than 100MB memory when idle). They send periodic heartbeats to the gProfiler Performance Studio every 60 seconds.
2. When an engineer submits a profiling request through the UI or API — specifying a service, host list, duration, frequency, and profiler configuration — the Profiling Service converts it into requests and queues them into the database
3. On the next heartbeat from a targeted host, the Heartbeat Handler responds with the pending profiling request. The agent validates, enqueues, and executes it.
4. Request execution is idempotent — agents track received and executed request IDs to prevent duplicate execution, and report completion status back through subsequent heartbeats.

he full dynamic profiling system — agent-side profiling request management, heartbeat client, and backend profiling-orchestration APIs — has been merged upstream into the Intel gProfiler repository and the Intel gProfiler Performance Studio repository, available for any organization to use.

PerfSpect: Deeper Insights for Infrastructure Decisions

Flamegraphs show where software spends time, but not how hardware is stalled or saturated. Intel PerfSpect adds CPU stalls, memory bandwidth, CPI, cache, and pipeline telemetry backed by hardware Performance Monitoring Counters (PMCs) — and Pinterest extended its gProfiler integration for on-demand collection of PerfSpect via the Dynamic Profiling UI, with metrics stored in the same backend for correlation with software hotspots. A standalone build pipeline bundles PerfSpect independently of Intel’s release cycle and runs on both x86 and ARM, covering the majority of Pinterest’s fleet including Graviton. The result: engineers pick optimal AWS instance types based on real workload behavior, not vendor specs.

Wall Time Profiling

Default CPU flamegraphs miss a huge class of microservice latency — threads blocked on I/O, locks, DB queries, or RPC calls never appear. Pinterest contributed wall time profiling (intel/gprofiler#1012), which enables sampling stacks by elapsed wall-clock time to capture both running and waiting states. Engineers toggle CPU and wall time modes on the fly through the same UI — no agent redeploys — and compare them side-by-side in Performance Studio.

PMC Counter Flamegraphs

CPU profiling shows where time goes and PerfSpect shows system-wide hardware behavior, but neither pinpoints which functions cause cache misses or branch mispredictions. Intel and Pinterest contributed custom PMC flamegraphs (intel/gprofiler#1013) that sample on hardware events (L3 misses, TLB misses, branch mispredicts, memory stalls) so each frame’s width reflects event frequency, not CPU time. Agents advertise supported events, the backend validates requests, and results render alongside CPU and wall time profiles — letting engineers tie specific Java or C++ functions to hardware bottlenecks.

Reliability Journey

Taking gProfiler from prototype to production-grade infrastructure at Pinterest scale required solving several reliability challenges, many of which resulted in fixes contributed back to the open source project(PRs #987, #988, #993, #995, #997, #998, #1002) :

Memory optimization: The original agent consumed 2.8 GB at steady state on typical hosts and spiked to 4–5 GB+ on high-process-count hosts (500+ processes) — frequently triggering OOM kills. The root cause was 3,000+ leaked pipe file descriptors from subprocesses, compounded by leaks in the grpcio library and the profile collection path. Through comprehensive subprocess cleanup (98% FD reduction), grpcio leak fixes, and collection-path optimizations, we brought the agent to under 400 MB stable across both scenarios — critical for running as a DaemonSet alongside production workloads.

Core dump elimination: The async-profiler component occasionally triggered core dumps consuming over 100GB/day of disk on storage-optimized instances. We disabled core dumps at the child process level and added zombie process checks, eliminating this entirely.

Staticx temp file accumulation: Each agent restart left behind extracted binary directories in /tmp, accumulating hundreds of gigabytes over time. We addressed this through proper cleanup lifecycle management.

ClickHouse “too many parts”: During aggressive backfill after an outage, high indexer concurrency with small batch sizes created more parts than the merge scheduler could compact, blocking inserts. We tuned batch sizes, added parts-limit monitoring, and implemented the Orion shard manager for automated cluster operations.

Build time reduction: gProfiler builds originally took over 30 minutes. Through build pipeline optimization — migrating from Jenkins to Buildkite, parallelizing architecture-specific builds, and caching intermediate artifacts — we achieved a 70% reduction in build time.

Surfacing recurring efficiency opportunities

Pinterest has started using gProfiler data to drive optimization across top CPU-bound services. The effort is still a prototype — engineers eyeball flamegraphs and track findings in a spreadsheet — but it has already paid off: we were able to identify common patterns across multiple services that lead to meaningful improvements e.g

1. Frequent Pattern.compile calls delivered 2–3% CPU savings in the affected sub-service.
2. compression tuning across Zstd/Zlibng/LZ4/Snappy
3. Serialization optimization with SIMD-accelerated alternatives like SimdJSON alongside Kryo, Jackson, Protobuf, and MessagePack.

We’re working to formalize this into a structured recommendations framework in alignment with our original vision of what we hope the service would deliver.

Results and Impact

Since deploying gProfiler across Pinterest’s infrastructure:

• Coverage: gProfiler agents are installed provide profiling observability across our fleet
• Productivity: Self-service dynamic profiling UI enables any engineer to start profiling their service in minutes, without needing to call the performance engineers or lookup documentation for profiling
• Engagement: Since its launch we see at least ~140 users across (engineers, efficiency leads, experts) using gProfiler on a monthly basis
• Efficiency: The profiling platform has detected several bottlenecks, regressions and optimizations with vetted recommendations translating to measurable savings
• Lightweight footprint: Agents stay nimble even at fleet scale — 0.15% CPU and ~200 MB memory in silent/dormant mode, and under 2% CPU and 2 GB memory while actively profiling
• Reliability: The infrastructure operates with 99.99% heartbeat success rate service level objective (SLO), sub-5-minute P99 indexing latency, and sub-second P95 flamegraph query latency
• Open source: Dynamic profiling, PerfSpect integration, memory optimizations, core dump fixes, and build improvements have all been contributed upstream

What’s Next

We’re continuing to invest in several areas:

• Latency profiling: Knowing what is blocking a service from running at higher utilizations
  Spark profiling: Extending gProfiler to support Spark job-level profiling, with optimization opportunities through JIT analysis and thread-name-aware flame graphs
• Workload-level profiling: Granular profiling by Kubernetes namespace, pod, and individual process — moving beyond host-level to workload-level visibility
• Structured optimization framework: Evolving today’s prototype recommendations effort into a formalized system with precision and accuracy KPIs, automated vetting, and hardware-informed instance selection using PerfSpect data (cache miss patterns, memory bandwidth saturation) to guide right-sizing decisions
• Regression detection: Automated detection of performance regressions across deployments using profiling data differentials
• GPU profiling: Extending the platform to cover GPU-bound workloads, integrating tools like NVIDIA Nsight Systems and Nsight Compute for kernel-level insights, alongside dynolog and DCGM for fleet-wide GPU telemetry
• LLM-powered analysis: Augmenting the optimization framework with large language models speed up RCA and analysis to provide fix suggestions with source pointers

Conclusion

Maintaining software performance at fleet scale is a fundamentally different challenge than occasionally profiling a single service manually. It requires a platform that spans languages, operates across deployment models, and empowers every engineer — not just performance specialists to make things efficient. By partnering with Intel on gProfiler, Pinterest helped build that platform on a fully open source foundation: dynamic profiling for remote, on-demand analysis across hundreds of thousands of hosts; PerfSpect integration for hardware-level insights; wall time profiling and PMU counter flame graphs to answer questions that CPU-only profiling can’t; and a reliability-hardened pipeline that ingests hundreds of terabytes of profiling data monthly. Every major capability we built has been contributed upstream, so any organization can deploy the same infrastructure. If you’re looking for a unified, vendor-free profiling platform that scales from a single host to an entire fleet, gProfiler is ready.

Resources to get started with gProfiler

gProfiler is open source and ready to deploy:

• gProfiler Agent: github.com/intel/gprofiler
• gProfiler Performance Studio: github.com/intel/gprofiler-performance-studio
• Intel PerfSpect: github.com/intel/PerfSpect

Whether you’re running a handful of services or a fleet spanning hundreds of thousands of hosts, gProfiler’s dynamic profiling infrastructure — contributed by Pinterest — gives you the control to profile what you need, when you need it, without the overhead of profiling everything all the time.

All performance metrics and infrastructure statistics cited in this post are based on Pinterest internal data, Q1 2026.

Acknowledgements

This work would not have been possible without the collaboration between Pinterest and Intel’s open source teams.

We’d like to thank the current Intel team for their continued partnership on gProfiler and PerfSpect Min Yeol Lim, Keith Sundberg, Rogerio Gelamo, and Kim McLaughlin through ongoing support, development and maintenance of gProfiler, and Jason Harper for perfspect. This wouldn’t have been possible without Intel business and engineering leadership by Kelly Hammond and Bill Pearson.

We also want to recognize leaders who were part of laying the original foundation of this open source project: Asaf Ezra, Brendan Greg, Dave Burgess, Greg Lavender, Greg Ernst, Harshad Sane, Peggy Irelan and Steve Beckis. Additionally all members of the original Granulate team supported the original project.

From Pinterest, we’d like to thank the following current and past members who contributed to and helped maintain gProfiler: Ambud Sharma for Pinterest Software Performance Vision, Prashant Patel for Technical design and Infrastructure scale out. We are especially grateful to engineering contractors who helped bring this work to life : Artur Sarlo, Ashok Chatharajupalli, Marc Antonio Vieira De Queiroz and Renato Gagliardi Sabo. We’d also like to recognize Eric Lopez, Vibhav Garg, and Vaibhav Shankar for their partnership, and Nasser Manesh, David Westbrook, and Molly Junck for their leadership support.

To learn more about engineering at Pinterest, check out the rest of our Engineering Blog and visit our Pinterest Labs site. To view and contribute to open source projects at Pinterest, visit our GitHub.

Pinterest is built to get inspired and then turn the inspiration into realization — a dinner, a renovation, a wedding, a new skill. That only works if we understand more than the Pin a user just clicked. The useful unit of personalization isn’t the thing someone just clicked, it’s what they’re trying to accomplish. Such goals unfold over weeks and months, across dozens of sessions, through hundreds of saves.

We call those goals journeys: the intersection of a user’s interests, intent, and context at a point in time. A journey is a sequence of interactions, often spanning sessions, centered on one interest and a clear intent. For example, “Summer dresses” are an interest. “Figuring out what’s in style for a summer wedding I’m attending in August” is a journey. Someone might be interested in five at once: a kitchen remodel, a vegetable garden, learning sourdough and the mix changes as life does.

If we can name those journeys, recommendations and notifications can help someone finish a project instead of bouncing between unrelated interests. This post is about how we infer them in production: a fine-tuned 4-billion-parameter large language model (LLM) that reads a chronological activity log and emits a ranked list of journey names, in the user’s locale.

We didn’t start there. The initial user journey system clustered keywords and attached attributes with separate machine learning models. This post is the follow-up which covers how we formulate journey inference as generation, how we distill frontier-model quality into a 4B student we can actually serve, how the serving stack holds up at notification scale, and what we’re building next. The recipe is domain-agnostic: any platform with sequential user activity and structured intent targets (e-commerce, streaming, search) can adopt the same activity-to-intent formulation, collapsing classical multi-stage recommendation pipelines into a single generative model, making generative journey inference a general recipe for personalization beyond Pinterest.

Why interests aren’t enough

Most recommenders optimize the next item. However, journeys on Pinterest are longer-horizon. Planning a kitchen is not the same as a one-off click on a cabinet Pin, and it isn’t well described by a durable interest like “Home Decor”. The cabinet Pin, the backsplash search, and the garden-bed save might be one project, two projects, or noise. Getting that distinction right changes what we retrieve, what we write in a notification, and when we should stay quiet.

So the output we want is not a taxonomy label. It’s a short, specific, ranked list of a user’s active journeys, with specific names they’d recognize.

The legacy clustering system

We had almost no labeled journeys when we started, so we stayed lean. We reused pretrained text embeddings, started from a few hundred human examples, and built a multi-stage pipeline:

1. Extract keywords from search queries, annotations and interests on engaged Pins, and Pins on owned boards.
2. Embed and cluster those keywords hierarchically. Each cluster is a journey candidate.
3. Name the cluster by ranking its keywords and taking the top one.
4. Rank and diversify the clusters with a pointwise model, then a similarity penalty so the top of the list isn’t five near-duplicates.
5. Classify stage with heuristics: evergreen vs. situational (roughly, does engagement persist past a few weeks?), and for situational journeys, ongoing vs. ended (time since last engagement).

That pipeline powered the first journey-aware notifications. Against interest-based notifications, it lifted email click rate by 88%, push open rate by 32%, and positive survey feedback by 23%.

It also compounded its own errors. Every stage was locally reasonable; together they fragmented coherent goals.

• Fragmentation: Cabinets, countertops, and backsplash are one kitchen, not three journeys fighting for the same notification slot.
• Generic buckets: When a cluster was messy, the naming model fell back to the most frequent keywords — “Art”, “Recetas”, “Shopping”. Taxonomically true, useless for personalization.
• Incidental noise: A handful of meme Pins or celebrity clicks could occupy a top slot. Clustering doesn’t know which activity is a goal and which is a passing click.

Journey inference as generation

To solve these problems, we stopped treating journey inference as a chain of transformations and started treating it as one act of generation: read the activity, write the intent. The model gets a single input prompt containing a semi-static profile plus a chronological activity log, and returns strict JSON — a ranked list of journey names:

User Profile:
gender: female; language: en-US; country: US; age: 30-35
User Owned Boards: Kitchen Reno 2026; Weeknight Dinners; Garden Plans
User Activities:
2026-06-14 search shaker cabinets.
2026-06-14 save subway tile backsplash, kitchen backsplash; quartz countertop, white kitchen.
2026-06-12 click raised garden bed, vegetable garden; tomato trellis.
2026-06-09 search sourdough starter discard.
2026-05-30 save cast iron skillet cornbread, dutch oven recipes.
...

{"journeys": ["Modern Farmhouse Kitchen", "Raised Bed Vegetable Garden",
              "Sourdough Bread Making", "Cast Iron Recipes"]}

Notice what’s gone: no keyword extraction step, no embeddings, no clusters, no separate naming model. One model reads the log and writes the answer.

Some deliberate choices in the prompt:

• Actions collapse into a tiny vocabulary: search, save, like, click, comment, dislike. The model doesn’t need to distinguish a repin from a save-to-device; it needs to know both are strong positives. The prompt states the ordering explicitly: search/save > comment > like > click > dislike.
• Events are grouped by (date, action), with a semi-colon (;) separating Pins and a comma (,) separating tags within a Pin. That’s token economy, and it also gives the model a cleaner view of same-session bursts.
• Dislike is deliberately included in the input, as hides are evidence. The prompt says to suppress a journey whose support is predominantly dislikes unless there is evidence of significant related positive engagements.
• Temperature 0, reasoning tokens off, max 300 output tokens. We want a deterministic, replayable read of the same log, and with a fixed output schema for decoding purposes. Reasoning is turned off to reduce online inference cost.
• A 360-day lookback, up to 150 events. Journeys are long-horizon by definition, we experimented with different lookback windows and # of events to balance cost and relevance.

The output contract is deliberately simple: drop non-strings, dedupe case-insensitively, and the first occurrence wins. The output is parsed defensively because a generative system that occasionally emits prose where you expected JSON is a system that occasionally drops a user’s entire personalization payload.

The other thing worth saying out loud is that once you collapse the pipeline, the prompt becomes the ranking policy and the safety policy. Everything the staged system encoded in model weights and thresholds now has to be clearly stated. In production it asks for up to about twelve distinct journeys, each 1–5 words, in the user’s locale, ranked by:

1. Engagement strength (search/save over click)
2. Repeated support
3. Recency
4. Coherence
5. Specificity (“Mid-Century Coffee Table”, not “Furniture”)

It also encodes the failure modes we observed with the legacy journey clustering system: don’t merge unrelated topics; drop one-off outliers; treat auto-generated tags as noisy; lightly diversify the top of the list only among journeys that are already well supported. Empty output is allowed when the activity doesn’t support anything but inventing unsupported journeys is not.

These instructions aren’t just decorative. The diversity pass replaced a similarity-penalty diversifier. The dislike rule replaced a hand-tuned negative-signal filter. Few tag-noise instructions exist because auto-generated annotations are genuinely noisy, and the model will happily build a journey out of a classification artifact if you let it. Prompts like this deserve the same review discipline as model code, so we treat changes to them as launches.

Clustering was the right cold-start system, and it still exists as a fallback. Production inference is now LLM-first: one model, one decode, structured output.

Distilling a frontier teacher into a 4B student

Frontier models can infer coherent journeys without task-specific training, but they are too expensive to run for hundreds of millions of people on Pinterest regularly. Instead, we use them as teachers to generate synthetic training examples: each activity history is paired with a high-quality target journey. We also found that teacher outputs improved when the model was allowed to use additional reasoning tokens before producing its final structured response.

We fine-tuned open-weight Qwen3 students with 0.6B, 1.7B, 4B and 8B parameters. Supervised fine-tuning (SFT) teaches the model from the teacher-generated examples, while low-rank adaptation (LoRA) lets us update small adapters rather than every model parameter.

Quality improved rapidly during the first few thousand examples, then largely stopped increasing. Adding more similar examples provided little benefit; selecting diverse, high-quality examples — especially difficult consolidation and multilingual cases — was more valuable.

Model size presented a separate trade-off. The 0.6B and 1.7B students were faster, but produced less relevant and consistent journeys on the harder cases. Among the sizes we tested, the 4B model offered the best balance between output quality and serving throughput, so it became our production student.

Table 1: Same-user comparison of the staged clustering pipeline against the fine-tuned LLM. Fragments consolidated (Cabinets + Countertops → Modern Farmhouse Kitchen). Generic buckets got specific (Art → Penguin Art Drawing). Dropped intents came back (Cast Iron Recipes). Noise disappeared (Celebrity News, Memes). And user B got journeys in Spanish — something the keyword pipeline structurally couldn’t do, because it named journeys out of an English-annotation vocabulary.

Serving at notification scale

Scoring hundreds of millions of people on Pinterest is a batch problem with online-shaped latency. We run a full refresh when the algorithm changes, and a daily incremental pass over recently active users, so journeys move when behavior does.

We benchmarked the offline workflow against serving the same 4B model behind an online inference stack. Online serving won on throughput. The reason is a set of now-mature decode optimizations that a naive offline loop typically doesn’t get for free: PagedAttention, so the key-value cache isn’t wasted on padding; continuous batching, so a finished sequence immediately yields its GPU slot to the next user instead of waiting for the slowest prompt in a fixed batch. The serving stack is NVIDIA Dynamo in front of vLLM decode workers on L40S GPUs. Frontend and decode workers scale independently, and the frontend routes with key-value (KV) cache awareness so requests sharing prompt structure land on workers that already hold the relevant prefix, which matters because every request shares a long, stable system prompt. We enable prefix caching and chunked prefill and cap concurrent sequences per worker.

In production, a cluster on the order of 100 GPUs sustains about 775 requests per second end-to-end, with a median around 1.2 seconds and p95 around 3.0 seconds. The batch side is where the cheaper wins live. Inference runs as a daily fan-out over journey-eligible users, sharded into small per-job batches with bounded concurrency inside each batch, so a slow model response back-pressures one shard instead of the pipeline. The single most effective throughput lever we added wasn’t a GPU at all: we track when each user was last inferred and skip anyone processed within the last couple of days. Journeys don’t change hour to hour, and paying to re-derive an unchanged journey is the most expensive kind of waste — the kind that looks like progress on a dashboard.

If a decode fails, we retry that user rather than storing a partial list. An empty but valid JSON list is a successful inference; some users don’t have a supportable journey that day, and we’d rather send nothing journey-specific than a generic “Art”. We suppress inference of sensitive traits at multiple levels before anything is stored:

• Input keywords pass our notification content filter before reaching the prompt, so the model can’t build a journey on a keyword we wouldn’t have sent anyway.
• Output journey names pass that filter and our generative content safety filters, so names that are unsafe in a journey-inference sense are dropped even if they would pass a generic keyword list.
• The model is instructed never to infer or target topics related to health conditions, politics, religion, sexual orientation, gender identity, race, etc.
• Profile inputs are minimized, with ages bucketed at the low end and grouped, because the model needs a rough life stage rather than a birthday. Under-18 users in the EEA, Switzerland, and the UK aren’t eligible yet due to local laws.

Inference writes a ranked list of journeys per user into a store that notification serving reads at send time, and each journey name then becomes a retrieval query. We built embedding-based retrieval models to pull Pins for each user journey which are further ranked by our Pin ranking models. Journeys the user was recently notified about are fatigued, and journeys they’ve given explicit negative feedback on are suppressed outright. Only then does the notification framework decide whether to send at all, and on which channel.

How we know it’s better

There is no ground-truth label for a user’s latent goal. We evaluate at three levels, from inspection to a causal test.

• Same-user qualitative comparison: For journey-eligible users, we put clustering output and LLM output side by side on the same user activity. We keep both the strong cases and the honest misses: over-generalization (“Home Cooking” when the log is clearly “Spam Kimchi Fried Rice”) and under-generation (dropping a well-supported side interest). This is slow, but it’s the only way to see fragmentation and language quality that metrics flatten.
• Evidence-grounded diagnosis: A separate language model scores a sample of emitted journeys on axes we care about in production: relevance to the visible context, support from the profile and activity log (is this name faithful to what we showed the model?), diversity / non-redundancy across the list, and granularity. The support axis is the important one. A fluent, on-brand name that isn’t in the log is a failure, even if a user might have clicked it.
• Online experiment: Initial experiment ramped to the US and Canada users, with the clustering production system as control, showed lifts of 1.1% on email click-through and 1.3% on push opens, plus multiple statistically significant sitewide engagement and retention gains. A subsequent multilingual international rollout replicated the direction, and shipping both experiments increased the daily, weekly, and monthly active users on the platform.

What we’re building now

The fine-tuned Qwen-4B model is the production default. Everything in this section is under experiment — trained or rolling out, not what every user gets today.

Semantic IDs, so Pins aren’t a bag of tags

Tags mean every Pin costs a handful of tokens and drags along a dependency on Pin-to-text mappings. Semantic IDs (SIDs) are the alternative: each Pin is compressed by a residual-quantized variational autoencoder (RQ-VAE) into five levels of 11 bits each, which we pack into a single 64-bit integer and unpack at prompt-build time into special tokens:

<sid_begin><sid1_1263><sid2_973><sid3_725><sid4_793><sid5_1727><sid_end>

A general-purpose model has no idea what <sid3_725> means, so we bridge the gap with continued pre-training (CPT): extend the tokenizer with the SID vocabulary, then run next-token prediction over interleaved SID sequences, Pin titles, and tags until the model has mapped its world knowledge onto our catalog. SFT goes on top, so that given a SID sequence the model writes journeys.

SIDs only work better for users with dense engagement, so production is a fallback ladder: fewer than ten SID-carrying events and we fall back to tags for the whole prompt; a single event missing a SID falls back to tags for that event; and search queries always keep their literal text, because a typed query is the highest-fidelity signal we have. The bet is fewer input tokens, no dependency on annotations, and a shared item language between recommenders and LLMs. The cost is interpretability — a tag-based prompt can be read by a human mid-debugging session. A SID prompt cannot.

Offsite conversions as a complementary signal

Onsite search and saves remain the strongest intent signals, but they can be sparse for dormant or low-activity users. We therefore use recent offsite actions — such as page visits, add-to-carts, sign-ups, and checkouts — as a complementary signal, adding a compact, safety-filtered summary to capture timely shopping or planning intent beyond Pinterest.

Offsite evidence remains secondary: onsite outweighs offsite, and checkout is stronger than add-to-cart or a page visit. We only add it when onsite activity is sparse, cap its prompt footprint, and avoid ranking offsite-only journeys at the top. We are testing whether better prompting is enough or whether a fine-tuned offsite-aware model is needed.

One model, several products

Once inference is generative, adding an output is a prompt-and-fine-tune change rather than a new service.

• Multi-granularity journeys: Rather than a flat list of names, have one call emit each journey together with deeper cuts of it and adjacent topics that broaden discovery, so “Sourdough Bread Making” comes back with a specific next step into the interest and a sibling topic a user might jump to. That gives downstream retrieval a ladder of intent per user instead of a single rung.
• Copy titles: A good journey name isn’t always good notification copy, and we had engagement data showing which names underperform as titles (“Photography”, “Hairstyles”, “Quotes” — the generic ones again). One model now produces both, with a hard ordering constraint: finalize the journey names first, then write copy, and never reshape a name to make its copy sound better.

Closing the loop with User feedback

Distillation got us strong journeys, but it has a ceiling. The student is optimized to match the teacher, so its quality is bounded by the teacher’s sense of what users want. To go further, we need to optimize directly toward user preferences, using real feedback from users themselves.

Much of that signal already exists. Journey emails include a module asking “Are you interested in <journey>?” with Yes and No buttons. Each answer is logged, and a No immediately suppresses that journey for that user. Beyond these explicit answers, engagement inside journey-aware notification sessions gives us a much larger implicit signal: saves, clicks, and hides on the Pins we retrieved for a journey.

This feedback comes with a caveat: it is pointwise. Each response is a verdict on a single journey, while the model generates a ranked list. To bridge that gap, we split training into two stages. First, a mid-training stage applies preference optimization to single journeys across a large volume of feedback. Given a user’s activity, the model learns to make a journey that users confirm or engage with more likely, and a journey they reject less likely. Then, we formulated feedback-as-edit: the chosen response is the model’s list with the Yes journeys kept or the No journeys removed, and the rejected response applies the reverse edit. Starting from the mid-training checkpoint, fine-tuning restores list-level behavior: ranking, diversity, and the output format described earlier.

Improving the LLM with preference optimization and real user feedback is still an active area of experimentation. We are exploring how to weight explicit versus implicit feedback, and how to formulate a loss that is amenable to scaling, one whose gains follow a scaling law as feedback volume grows. Offline results are promising, and online A/B tests are now running. We hope to share more details in a follow-up blog post or publication.

Acknowledgements

We’d like to thank Justin Tran, Darien Boyd, Sarah Schwager, Shipeng Yu, Randy Tumalle, James Hernandez, Mengtian Hu, Dimitra Tsiaousi, Yaxin Li, Ying Wei, Sam Owens, Gloria Lee, Justin Pangilinan, Ye Tian, Rui Liu, Anya Trivedi, Binghui Gong, Tianqi Wang, Fangzheng Tian, Eric Tam, Mengying Yang, Lauren Fratamico, Andrew von Rosenbach, Ignacy Debicki, Zhenyu Tan, Jiajing Xu, and other contributors from Notifications, ATG Applied Science, ML Platform, GenAI Serving, and Content Safety teams for their insights and support!

From Activity to Intent: Generating User Journeys with LLMs was originally published in Pinterest Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

At Pinterest, reliable and trustworthy metrics are behind every decision: from measuring company-wide business performance to evaluating each feature experiment results. Hundreds of data producers — analysts, data scientists, and engineers across Pinterest — create thousands of metrics from our petabyte scale data lake. A metrics ecosystem this size can’t be ad hoc; it has to be organized, scalable, and resilient. So we built Metrics Board: an end-to-end platform that automates metric creation, enforces quality and enables agent-driven analytics.

In this post we walk through the product experience, architecture, integrations and explain some of the design choices we made to make metrics work at planet scale.

Why Metrics Matter at Pinterest

Most organizations rely on a familiar set of metrics — revenue, growth, engagement — to steer the business.

Metrics are how we understand whether Pinterest is delivering value to users and advertisers. Measures related to areas such as revenue, growth, and engagement turn the complexity of user behavior and Pinterest performance into a shared view of what is working, where we are falling short, and what matters most. These metrics drive the decisions our product, finance, and executive leaders make every day, from what to build to how we forecast and steer company strategy.

Metrics also power Pinterest’s product experimentation practice, which runs at massive scale — thousands of experiments are active at any given time. Those experiments are used as gates for each new feature launch. For results to be fair, comparable, and statistically significant, metrics must be clearly defined, reliable, and reusable. Beyond common core metrics, individual experiments often need bespoke metrics scoped to a single module, feature, or device. For example, a feature to add a new form of interaction with a Pin might need its own measure to evaluate engagement with the feature — even if core product engagement metrics are flat, the experiment might be successful if it creates a new form of user-engagement that a new metric would need to measure.

As much as key decision-makers need metrics, the way they get answers from data has changed too. Increasingly, the question isn’t typed as a SQL query or a dashboard filter — it’s asked in natural language to an agent. But agents are only as good as the trusted data they can access. For agentic analytics to work, metrics have to be discoverable, quality-checked and machine-readable.

Before Metrics Board

In the past, users were defining their metrics in bespoke data pipelines and wiring them by hand into our reporting and experimentation products. That approach had several problems:

• Time spent building similar data pipelines by hand was slow
• Inconsistent monitoring and availability were inconsistent, so breakages often only surfaced downstream
• Low awareness of existing metrics, leading to redundancy and trust erosion
• Metric metadata drifted from one place to the next with no source of truth

Put together, these problems slowed teams down, held experiments back, and left us with more metrics than we could trust. We needed a solution that would:

• Unify metrics across the organization
• Give us transparent definitions and code versioning
• Automate pipeline creation, orchestration and operations as much as possible
• Require and auto-generate documentation and metadata
• Automatically validate and monitor metrics for accuracy, reliability, and anomalies
• Give every metric and dashboard a clear, shared signal of quality and trust

Our goal was to make metrics easy for everyone who touches them — the people who produce them, the people who consume them, the applications that serve them, and, increasingly, the agents that reason over them.

The Solution

Define a metric once, Metrics Board handles the rest.

We built Metrics Board as a thin semantic layer for metrics on top of the data tools Pinterest already runs: a metric is defined once, and that definition drives everything downstream. It’s built around five pillars: inventory, compute, quality, publishing, and serving & discovery.

1. Inventory

The inventory pillar is our single source of truth for metric definitions. Every metric lives as a YAML spec in a dedicated Git repo, and is treated like any other production code, with version control, code review, and change history built in.

version: 1
metric_id: mb__metrics
display_name: Metrics Board metrics count
description: |-
  Count of metrics defined in MB. Test metrics aren't included.
domain: tech_data
metric_sql: |-
  with mb as (
    SELECT urn
    FROM query_analytics.pincat_db_latest_state_v2
    WHERE dt = '{end_date}'
    AND aspect = 'structuredProperties'
    AND is_active
  )
  SELECT 
    COUNT(mb.urn) as mb_metrics,
    '{end_date}' as dt,
  FROM mb
temporal_columns:
  - column_name: dt
    format: YYYY-MM-DD
    granularity: DAY
integrations:
  superset:
    enabled: true
  starrocks:
    enabled: false
ownership:
  nimbus_project: ii
compute:
  schedule: 0 9 * * *

One important consideration at the time of design was how much should Metrics Board be a true semantic layer. Do we enforce a model on top of existing datasets and have metrics defined as expressions leveraging the model? Or do we accept any SQL query and ask users to provide modelling information (temporal columns, metric or dimensions definitions) for the resulting data? We decided on the latter, driven by two main considerations: user feedback (“Put Pinners First” is one of our company values) and the sheer variety of tables which are used for metrics definitions — enforcing a semantic layer for each was simply not realistic.

We don’t ask teams to craft definitions by hand. The Metrics Board UI is a dedicated interface: users define metrics through guided forms, and the system generates the corresponding YAML and opens a pull request on their behalf. The result is one place to manage ownership, governance and metadata, that still integrates cleanly with the rest of the data stack.

2. Compute

To compute metrics, Metrics Board turns each definition into its own scheduled Airflow DAG. A definition carries all the metadata we need — schedule, SQL, validations, integrations — to generate a dedicated workflow. Isolating each metric in its own DAG lets us tune schedules, retries, and resources individually, and prevents a failure in one cascading to others.

At execution time, Metrics Board compiles each metric’s SQL into SparkSQL or PrestoSQL depending on the engine named in the definition. It then analyzes the SQL code to automatically create dependency tasks which wait on upstream data to be ready (we rely on sqlglot for parsing). It also automatically generates standard data quality checks and lets users define additional ones too (more on that below). Backfills are managed through Airflow, so owners can recompute history easily. Metrics Board also provides a layer for automated operations. For example if upstream data is late, Metrics Board automatically detects when it lands and resumes the metric computation DAG.

Taken together, the above listed features make metrics creation and management easy even for non-data teams. What used to take days or weeks to build can now be achieved in 1–2 hours — a significant boost for users’ development and experiment velocity.

3. Quality

A metric nobody trusts is worse than no metric. Every definition can declare validations that run as part of the metric’s own pipeline, backed by two engines:

• Rule-based checks for explicit expectations: day-over-day, week-over-week, month-over-month, and year-over-year thresholds, custom threshold rules, and comparisons against a parent metric — each applied to either a single point in time or a trend across time.
• ML-based anomaly detection that models a metric’s historical patterns — seasonality, trend, and normal variation — and surfaces deviations which a static rule would miss (you can read more about Warden here).

When a check fails, the right people hear about it in real time. Alerts fire as soon as validation runs, route to the owning team, and can be snoozed or acknowledged so the signal stays meaningful and doesn’t turn into noise.

All of this rolls up into a quality report per metric — per-check pass rates, a time series of results, gaps where validation didn’t run, and annotations (comments or linked tickets) — available via API and as a PDF export. Because the validations live in the same spec as the metric, quality is versioned and reviewed alongside the definition itself.

4. Publishing

A metric that lives only in its own pipeline isn’t useful yet — it has to reach the products people actually work in. Publishing does that automatically, driven entirely by the definition. Each metric declares its integrations, and turning a surface on is as simple as a flag: enable it, and Metrics Board fans the metric out to that surface with no per-surface wiring by hand. Today a single definition can publish to:

• PinCat (based on DataHub), our data catalog and metadata hub, so the metric is discoverable and its lineage is tracked
• Helium, our experimentation platform, so the metric is available to experiments the moment it’s defined
• Superset, our BI layer, where Metrics Board creates or updates the dataset and can stand up a monitoring dashboard automatically
• StarRocks, our analytics warehouse, for fast interactive querying

Publishing happens in two phases:

1. Metadata-level integrations (DataHub, Helium) publish at commit time as soon as the definition merges. The metric is catalogued and available to experiments before it has even run.
2. Data-level integrations (Superset, StarRocks) publish at compute time after each run when there’s fresh data to export.

Either way, the outputs (dataset IDs, the dashboard link, the output table, the pipeline URL) are written back to DataHub, so the catalog stays the single place to find a metric and everything attached to it.

5. Serving & Discovery

Discovery & governance. Metrics Board is where the org finds and evaluates metrics — but consumers have to be able to find a metric before they can evaluate it. Metrics Board centralizes definitions into a single source of truth, capturing rich metadata alongside them: ownership, description, certification grade, and quality signals.

That metadata is what turns centralization into discovery. Every metric automatically flows into PinCat, so users can discover them in the same place they already go for datasets. With metrics easy to find, the next question is what to trust. Two signals do that work, and they operate at two levels.

At the metric level, every metric carries a grade — a certification level that climbs from Unknown through C, B, and A up to A+ (fully certified and manually verified by a Data Quality Steward), with each rung demanding more quality work. It tells consumers what to trust and shows owners exactly how to level up. Furthermore, a metric’s key metadata — grade, certification state, and owner — is surfaced directly within its Superset chart.

At the reporting level, dashboards in Superset are classified into their own three tiers that reflect how critical and durable they are, and a dashboard’s tier drives three concrete things: how long it’s retained when unused, which Presto cluster its queries run on (and therefore its performance), and what documentation it must carry.

• Tier 1 is a small, curated set of company-critical dashboards. They carry the fullest documentation and are formally certified, and in return get the strongest guarantees — priority query performance and protection from automatic cleanup — with certification renewed periodically to keep them trustworthy.
• Tier 2 is for team-level dashboards others rely on. They require basic ownership and documentation and are retained as long as they see regular use.
• Tier 3 is the default for exploratory or test dashboards. They need no extra metadata and are cleaned up automatically once they go unused.

Together, metric grade and dashboard tier give consumers a trust signal at every layer of the stack, while retention and query-lane routing fall out of the dashboard tier automatically.

Agent enablement. The same structured metadata that powers discovery makes Metrics Board a foundational platform for analytics agents. We expose an MCP server so any LLM-based tools can find metrics, read their definitions, and reason about them. The clearest example is our Analytics Agent, which can answer natural-language questions about metrics. By referencing golden queries and certified metric definitions, the agent ensures its responses are grounded in trusted, governed data. As it turns out, the work we did to make metrics trustworthy for humans is what makes them usable by agents.

Impact

Adoption is the primary success measure we track, and we’ve exceeded adoption goals quarter after quarter — both in metrics created and in active users. Today nearly 150 new metrics are created each month (we also built out a lifecycle policy and engine to garbage-collect metrics no longer in use). Over 98% of metrics created for experimentation go through the Metrics Board today.

That adoption rides on dramatically improved velocity. On one recent high-profile project revamping key engagement metrics, Metrics Board saved roughly two engineering-months versus the old approach; metric additions that once took weeks now take days. At scale those savings translate to thousands of hours saved each month.

And velocity isn’t the whole story: the governance, quality, and cataloging we built along the way are quickly becoming the foundation for the LLM and agentic use cases we’re now shipping.

Acknowledgments

Analytics Platform: Krishna Gopal
Workflow Platform: Dinghang Yu, Martin Yau
Metrics Quality: Dennis Marcu, Raj Phull
Experiment Platform: Lu Yang, Xinyue Cao
Real-time Analytics: Charles Wu

Metrics Board: Building an Agent-ready Metrics Layer was originally published in Pinterest Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

This is the third post in our series on Pinterest’s next-generation database ingestion framework. Part 1 introduced the DB ingestion framework built on Kafka, Flink, Spark, and Iceberg, and Part 2 covered automated schema evolution. This post tackles another challenge in migrating downstream customers to the new ingestion framework: knowing when data is complete enough to read.

We’ll walk through that data completeness challenge and how we solve it: what “partition finalization” means, why it matters to downstream customers, how we built a unified mechanism to generate partition finalization markers, and how those markers are consumed. We’ll also look at how this mechanism generalizes beyond DB ingestion.

Background & Motivation

Before explaining partition finalization, it helps to define what a partition is and why a downstream consumer cares when one partition is “done.”

What is a Partition

A time partition is a subset of a table whose rows are grouped by a time column, e.g. all rows whose timestamp falls within a given hour. Grouping data this way lets a consumer read just the slice it needs, such as the 1 AM hour, instead of scanning the whole table.

How that grouping is physically represented depends on the table format:

• Hive uses explicit, directory-based partitioning: each partition maps to a physical directory in S3.
• Iceberg uses hidden partitioning: the partitioning is defined in the metadata layer rather than by directory layout, so consumers don’t need to know the physical file layout to query a partition.

What Finalization Means

Now consider a typical consumer of these tables: a downstream batch job that processes a single hour’s partition and runs only once for that hour. The consumer faces the question: when should my job run?

The answer is straightforward: Because the job runs only once, it should run only after the target partition is ‘finalized’, i.e. data in that partition is less likely to change. If it runs too early and the partition is still being updated after the run, it will miss later updates in that partition. And to know when a partition is finalized, it relies on the producer to put some signals on the data. For example, the producer can emit a lightweight marker in the table location when it thinks the partition is finalized, and the consumer will use a sensor to check that marker.

Challenge: Finalization in the Stream-based DB Ingestion Framework

In the previous batch-based system, finalization was trivial. The data was produced by a single batch job that read the whole source DB and wrote it in one shot, so once that job finished and there is data in a partition, we can consider that as finalized and downstream jobs could start consuming.

The new framework is stream-based. A Flink job writes change events continuously, committing many small batches into a partition over time. Because a record’s event time (when the change happened in the source) can lag when we process it, a partition can keep receiving data after its wall-clock hour has passed and no single moment marks it “done”, and the presence of data in a partition no longer means the partition is complete.

So finalization is no longer a byproduct of a job finishing. We have to infer it from the stream itself uniformly across the thousands of pipelines the framework supports, and a new mechanism is needed to support this.

Our Solution: EventTime Based Partition Finalization

We introduced a partition finalization mechanism into the streaming layer of the pipeline. The same Flink job that writes the CDC table also measures how far its event time has progressed and publishes a finalization marker into the destination Iceberg table, where downstream jobs can wait on it via a sensor before they start consuming.

Architecture Overview

There is no change in the underlying data flow as in Part 1 (see below figure, blue components are added for partition finalization): Kafka carries CDC events, Flink writes them into the CDC Iceberg table, and Spark upserts into the base table. The partition finalization logic mainly happens inside the Flink-to-Iceberg sink, plus one piece of published metadata that can be carried over in the Iceberg table.

Partition Finalization Logic

We’ll start with how a Flink-to-Iceberg pipeline works in general. As Flink processes the event stream, it first writes incoming records into data files. Those files aren’t visible to readers right away; they become visible during Flink’s checkpoint process, which runs every few minutes (configurable) to persist the job’s progress for fault tolerance. Every checkpoint produces a new snapshot in the Iceberg table: an atomic commit that also carries a small summary of metadata describing it. It works much like a Git commit, with each snapshot recording a new visible state of the table on top of the last.

With this process in mind, our partition finalization logic executes in the following procedures:

Event time extraction

Everything starts from a single value per record: the CDC event’s event time. Each pipeline configures which column carries it, and an extractor reads that column and normalizes it to a timestamp. Because the extractor is pluggable, the same machinery works for any table by simply pointing it at that table’s event-time column.

Fact collection: event time statistics

Between checkpoints, we collect the event time of every record processed within that window and track statistics over them. The core facts are the minimum and maximum event time seen in that window. On top of those, we also capture percentiles, e.g. the 99th and 95th percentile of event time, so we retain the shape of the distribution rather than just its extremes.

Fact storage: Iceberg snapshot summary

When a checkpoint commits, these statistics are written into the Iceberg snapshot’s summary, the per-snapshot metadata Iceberg already maintains for each commit. Every commit therefore carries its own self-describing record of what event times it contained, with no external store to keep in sync.

To keep the footprint small, we store the percentiles in a t-digest, a compact sketch that approximates a distribution in a small, fixed amount of space. The summary stays around a few hundred bytes whether a checkpoint saw a thousand records or millions, which is what makes it affordable to attach to every commit. The sketch is also mergeable, so partial statistics computed in parallel combine into a single summary at checkpoint time.

From fact to opinion: partition finalization watermark

After each commit, the framework turns those facts (the per-commit event-time statistics, such as the minimum event time seen in each checkpoint) into an opinion: how far the partition is finalized. It runs an algorithm over the statistics from recent commits and produces a watermark that marks the point up to which the partition is considered complete. By default, it takes the minimum of the per-commit minimum event times across the last X commits, then rounds down to the hour boundary.

The watermark is kept monotonically increasing. If late-arriving data would pull it backward, the value holds instead of regressing, so a finalized hour stays finalized. When that happens, the application can also send an alert to the data consumer, flagging that late data arrived after the partition was finalized.

Opinion storage: Iceberg table property

Unlike the facts which live on individual snapshots, the opinion is a single table-level value, so we store the partition finalization watermark as an Iceberg table property. Downstream jobs read it with an ordinary metadata lookup and gate their processing on it, without the need of any side channel or separate service.

Finalization Metadata Propagation

Since the statistics and finalization watermarks all live in Iceberg metadata, they are easy to propagate. Finalization is computed on the CDC table, but many consumers read the base table produced by the Spark upsert job, which carries the data but not the signal.

We close that gap in the upsert job: on each run it reads the finalization watermark from the CDC table and writes it onto the base table in the same commit. The signal follows the data to where consumers read it, and because these are ordinary Iceberg properties, propagation is just copying metadata during a commit the pipeline already makes without the need of side channel or services. Downstream jobs can then query those table metadata via sensors.

Underlying Implementation: Extensible Iceberg-Flink Sink

The above partition finalization logic is built entirely on top of the Flink-to-Iceberg sink. We needed two capabilities the standard sink did not have: (1) a way to collect per-record metrics inside the writer and merge them at commit time, and (2) a way to run custom logic immediately after every successful Iceberg commit. Rather than hard-coding partition finalization into the sink, we introduced two generic extension points — Accumulators and CommitProcessor — that any Flink-to-Iceberg pipeline can use.

Operator Topology

The standard sink has two Flink operators in sequence: a writer (one instance per parallelism unit) that flushes records to data files, and a committer (single instance) that makes those files visible by appending a new snapshot to the Iceberg table. The extensible sink keeps exactly the same topology and adds no extra operators.

When either extension point is configured, the writer and committer swap in extensible variants that carry accumulator and commit processor logic alongside. When neither is configured, the pipeline falls back to the standard operators with zero overhead.

Accumulators: collecting facts per record

An accumulator is a lightweight aggregator that runs inside each writer subtask and builds up a summary of the records it has seen since the last checkpoint. Between checkpoints it records and aggregates each record, extracting a field value and updating running statistics such as a count, a min/max, or a custom metric. At checkpoint time, each subtask takes a snapshot of its accumulator’s current state and resets it for the next window.

Those per-subtask snapshots travel to the committer alongside the data-file manifests. The committer, which sees results from every subtask, combines the individual snapshots into a single merged result. Because the subtasks may complete in any order, the merge operation must be order-independent: two snapshots combined must produce the same result regardless of which is folded into which.

Once merged, the accumulator’s value is written into the Iceberg snapshot summary under a namespaced key, sitting alongside the other metadata Iceberg already records per commit.

Commit processor: acting after each commit

A commit processor adds two hooks around the Iceberg commit: one that runs just before the snapshot is written, and one that runs just after it is durable. The pre-commit hook can set additional snapshot properties or abort the commit by throwing, which causes Flink to retry the checkpoint. The post-commit hook is where partition finalization lives: it reads the statistics just written to the snapshot summary, runs the watermark algorithm, and updates the table property if the watermark advances.

Both hooks receive the same context: a handle to the live table, the checkpoint identifier, and the merged accumulators from all writer subtasks.

Putting it together

The event-time accumulator collects the minimum, maximum, and a compact percentile sketch of event times across all records in a checkpoint window. At commit time those per-subtask observations are merged into a single global summary and written into the snapshot. The commit processor then reads that summary, together with recent snapshot history, runs the non-regressing watermark algorithm, and writes the result back to the table, all within the commit cycle without any external service.

Configurable Finalization Policies

Because each summary keeps the full distribution rather than a single number, the field the commit processor reads becomes a knob each table can turn. This is where the completeness-versus-latency trade-off is handled.

Most tables use the conservative default. Tables that value freshness and can accept a rare late record opt into a percentile instead, accepting that a small tail of records may arrive after their hour was marked safe to read.

Generalizing Beyond DB Ingestion

Nothing above is specific to database ingestion. The accumulator and commit processor only need a way to read an event time from each record; everything else, e.g. the sketch, the windowed lookback, the non-regressing watermark are generic. Any Flink-to-Iceberg pipeline that needs a completeness signal can adopt partition finalization by plugging in its own event-time extractor. And we are actively working on adopting this partition finalization mechanism into Pinterest’s next-generation Stream Ingestion framework.

Conclusion & What’s Next

Moving to streaming ingestion gives us fresh data in minutes, but it costs us the implicit completeness that batch loads provided for free. Partition finalization mechanism restores that guarantee: a cheap, mergeable event-time sketch on every commit, a non-regressing “safe to read” watermark derived from it, and a policy knob to trade-off completeness against freshness, and all of those are reusable beyond our DB ingestion framework.

In the following post, we will cover the incremental processing logic built on top of this ingestion framework and how we use it to support downstream use cases efficiently. Stay tuned for the future post: Incremental Processing on CDC Pipeline.

Acknowledgments

Huge thanks to my teammates Yisheng Zhou, Vi Nguyen, and Artem Tetenkin for building the Next Generation DB Ingestion at Pinterest together.

This project would not have been possible without the significant contributions and support of the following partners:

• Storage Services: Leonardo Marques Maciel Silva, Yuan Gao, Liqi Yi
• Storage Foundations: Tailin Lyu, Yu Su, Istvan Podor, John Grass
• Streaming Processing Platform: Kevin Browne
• Batch Processing Platform: Carlos Benavides
• Big Data Storage: Pucheng Yang, Mian Luo, Jenny Wang
• Data Warehouse: Bryant Xiao, Anita Narra, Pritam Pan
• Logging: Vahid Hashemian, Jeff Xiang, Jesus Zuniga

Special gratitude goes to Shardul Jewalikar, Ang Zhang, and Roger Wang for their continuous guidance, feedback, and support throughout the project.

Partition Finalization in Pinterest’s Next-Generation DB Ingestion Framework was originally published in Pinterest Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

Previously¹, we launched the next-generation serving stack for standard ads, which we call Nexus. Nexus decoupled candidate generation from scoring and moved us beyond the classic two-tower-only world, enabling richer model architectures while still meeting stringent latency and cost constraints.

Building on this system, we set out to design the first ads lightweight ranking model that goes beyond two towers. It jointly predicts three probabilities for each candidate ad: pCTR, the probability of a click; pGCTR30, the probability of a good click that lasts at least 30 seconds; and pOCTR, the probability of an outbound click to the advertiser’s destination. To support these objectives efficiently, we partition the query and Pin embeddings into task-specific CTR, gCTR30, and oCTR segments. For the CTR task, the fast two-tower prediction uses the first 64 dimensions of the CTR segment, while the three-tower prediction uses the full CTR segment together with richer cross features. For gCTR30 and oCTR tasks, full embeddings are shared between two-tower and three-tower predictions. This lets each task learn dedicated representations while sharing the overall model. In principle, Nexus places very few hard constraints on the architecture we can serve: cross-attention, sequence modeling, and more expressive interaction modules are all on the table.

However, in practice we quickly ran into the fundamental reality of ads lightweight ranking at Pinterest scale: for a typical request, we need to score on the order of hundreds of thousands of candidates (P99 post-targeting candidate counts can exceed 200K on some surfaces). We cannot simply keep increasing model complexity and expect to stay within our latency and cost budgets.

To strike a balance between latency and performance, we landed on a 3-tower co-train model design.

This design has a few key properties:

• We keep the query tower and Pin tower from the existing two-tower model, which lets us cache Pin embeddings offline and still obtain fast dot-product predictions for all candidates.
• We extend the architecture with a third cross tower that performs cross-attention between user sequences and candidate (Pin) features, plus an inter module that further mixes query, Pin, and cross embeddings.
• We co-train two-tower and three-tower predictions in a single model, giving us both fast but less accurate scores and slower but more accurate scores that we can deploy in different stages of the serving flow.

Put simply, the same model produces a fast two-tower score for every candidate and a richer three-tower score for a selected subset, so we can spend additional compute where it has the greatest impact.

Later in this blog, we will walk through the model architecture (cross tower and inter module), serving performance optimizations, and the two-stage scoring flow that leverages both two-tower and three-tower predictions.

By combining these changes, we maintained two-tower prediction quality while achieving around 30% reduction in offline loss for three-tower predictions across our engagement tasks, compared to the existing production model (details see below Offline Performance section). These offline gains translated into online lifts in CTR and gCTR30, reductions in cost per click (CPC), with a modest increase in infrastructure cost.

Model architecture

Our starting point was the existing two-tower engagement model, with separated query and Pin towers whose dot product feeds into task-specific heads. On top of this, we introduced two major components:

• A new cross tower that uses reduced-query cross-attention between user sequences and candidate features to capture high-order interactions.
• An inter module that jointly processes the query, Pin, and cross embeddings and produces a shared representation for all engagement tasks.

Below we describe the main design choices and trade-offs in each part.

Cross tower

The cross tower is responsible for modeling rich interactions between a user’s recent activity and a candidate ad. We use three on-site user sequence features (organic engagement, ads engagement, and search history) and four candidate features (advertiser ID, campaign ID, GraphSAGE embeddings, and Pin PinnerSAGE embeddings).

A natural first idea would be to build increasingly complex attention modules over these sequences and candidates. In practice, we explored several options:

• Merging full user sequences (across surfaces) and then running cross-attention with candidate features.
• Using shorter, truncated sequences to reduce compute and memory.
• Replacing attention with simpler interaction functions such as DIN-style pooling or average pooling.
• Crossing each sequence attribute with its corresponding candidate attribute individually, rather than merging first.

These variants exposed a clear trade-off: more expressive attention patterns (longer sequences, more attributes, per-attribute crossing) tended to improve offline loss but also increased latency, especially at high candidate counts. For example, using more complex cross architectures could reduce loss by several additional percentage points, but at the cost of tens of milliseconds of extra latency per request at 100K candidates.

We ultimately converged on an architecture that uses candidate side features to generate query tokens, which then interacts with user sequences to calculate attention. In this way, we can pick the most important candidate features and control the cost of transformer computation. This design gives us:

• Strong offline performance improvements versus production.
• A predictable compute profile that is easier to optimize and scale.
• A good balance between modeling capacity and serving latency.

The output of this cross tower is a cross embedding that summarizes how a user’s recent behavior interacts with a particular candidate ad.

Inter module

The inter module takes three inputs: the query embedding, the Pin embedding, and the cross embedding from the cross tower. Its goal is to produce a compact, shared representation that works well for all three engagement tasks (CTR, gCTR30, and oCTR), while keeping parameter count and serving latency under control.

Here as well, we evaluated multiple architectures:

• A deep MLP with DCN (Deep & Cross Network) layers.
• A standard MMoE (mixture-of-experts) with DCN.
• A top-K MMoE with DCN.
• A shared-bottom MLP with additive task-specific biases.

More complex structures such as MMoE with DCN achieved stronger loss reductions but also introduced noticeably higher latency compared to production. The shared-bottom MLP design provided a sweet spot: it delivered most of the performance gains while adding only modest latency, and it is architecturally simpler to optimize further.

In the final design, the inter module:

• Learns a shared logit for each task from the concatenated query, Pin, and cross embeddings.
• Adds task-specific biases computed from query and Pin embeddings for gCTR30 and oCTR.
• Outputs task logits that are then passed through sigmoid functions to produce probabilities.

This structure allows us to capture shared patterns across tasks while preserving enough task-specific flexibility.

Loss function

We train the model using a multi-task loss that combines main losses for the three-tower predictions with auxiliary losses for the two-tower co-train task.

The final loss takes the form of a weighted sum:

• Main three-tower losses for CTR, gCTR30, and oCTR.
• Co-train two-tower losses for CTR, gCTR30, and oCTR, computed from query–Pin dot products. For CTR, this fast auxiliary prediction uses the first 64 dimensions of the CTR embedding.

We tuned the task weights to balance learning stability and final performance, and landed on the following weighting scheme:

• Strong emphasis on main CTR loss.
• Moderate weight on main gCTR30 loss.
• Lower weight on main oCTR loss.
• Non-trivial but smaller weights on each of the co-train losses.

This configuration made the three-tower predictions the primary optimization target, while keeping the two-tower co-train task healthy enough to match or slightly improve on the production two-tower model.

The query tower produces a 192-dimensional task embedding: 144 dimensions for CTR, 32 for gCTR30, and 16 for oCTR. The Pin tower produces the corresponding task embedding and appends a 256-dimensional candidate-feature projection for the cross tower, producing a 448-dimensional Pin representation. For fast two-tower CTR scoring, we use only the first 64 dimensions of the 144-dimensional CTR segment. For three-tower CTR scoring, the inter module uses the full CTR segment together with the cross embedding. The additional Pin projection is computed in the Pin tower and cached offline, so the three-tower path can use these candidate features without per-candidate preprocessing at serving time.

Formally, our final loss is:

L = (100 * L_main_ctr + 5 * L_main_gctr30 + 1 * L_main_octr) + (10 * L_cotrain_ctr + 5 * L_cotrain_gctr30 + 1 * L_cotrain_octr)

Offline performance

We evaluated offline performance on held-out standard-ads data across three tasks (CTR, gCTR30, and oCTR), reporting relative loss reduction versus the production two-tower engagement model for both main (three-tower) and co-train (two-tower) predictions. The table reports ranges because we evaluated the model across multiple log sources; each endpoint is the result observed for a different source. For gCTR30, we observed a small degradation on the co-train loss which we deemed acceptable given the main task gains.

Across tasks, we observed:

Taken together, these results show that the co-train task maintains performance comparable to the existing production two-tower model, while the three-tower predictions deliver substantial improvements. This is important operationally: we can deprecate the standalone production two-tower engagement model and rely on the co-train head for fast scoring, without sacrificing quality.

Serving optimization

Serving a three-tower model over hundreds of thousands of candidates per request is expensive. To make the launch feasible, we invested heavily in model-level latency optimizations. Below are several techniques that had meaningful impact. Together, these changes reduced P99 model inference from over 200 ms to about 30 ms, leaving the final model only 1–2 ms slower than production.

Optimization 1: Move Pin pre-processing into the Pin tower

To perform cross-attention, sequence and candidate features must share the same dimensionality. In an initial design, we handled this with on-the-fly MLPs in the cross module, which added per-request compute proportional to the number of candidates.

Instead, we moved this preprocessing into the Pin tower. We append the processed Pin features to the original Pin embedding, increasing its dimension by an additional 256, and cache the resulting embedding offline. At serving time, the cross module can directly consume these enriched Pin embeddings with no additional per-request MLPs.

This change saved roughly 2 ms of latency at 100K candidates in our benchmarks.

Optimization 2: Lower precision

We also explored reduced-precision inference. By switching from FP32 to BF16 in the three-tower path, we significantly reduced model inference time while keeping model quality neutral.

On one representative benchmark, we observed:

• At 50K candidates, latency dropped from around 103 ms in FP32 to about 56 ms in BF16.
• At 20K candidates, latency dropped from around 45 ms to about 26 ms.

These gains played a key role in making the three-tower path practical at high candidate volumes.

Optimization 3: Late expansion of user features

In the three-tower engagement model, we compute predictions between one user and tens of thousands of candidate ads at once. User features are computed once and then expanded to match the batch size of candidates.

Earlier, this expansion happened just before the cross module to avoid duplicated computation. We realized we could delay expansion even further: instead of expanding before building the attention keys and values, we expand inside the cross module right before attention is computed.

This avoids redundant computation on large tensors and yields latency savings of around 15 ms at 100K candidates in our benchmarks.

Optimization 4: Pre-layer normalization in attention

Finally, we revisited how we apply LayerNorm inside the cross-attention module. Previously, we normalized the larger output sequence after attention, which has a batch size proportional to the number of candidates. We switched to normalizing the input sequence before attention instead; during serving this input has batch size 1, so the normalization work is much lower and independent of how many candidates we score.

During training, both options behave similarly. During serving, however, pre-layer normalization dramatically reduces the amount of work we do at large candidate counts. Flipping pre_lnorm from False to True reduced latency by about 3 ms at 100K candidates in our benchmarks.

Rethinking the serving flow

Even with model-level optimizations, running the full three-tower model on every candidate would still be too expensive. Post-targeting candidate counts can exceed 100K at P90 and reach up to over 200K at P99 on some surfaces. In early experiments where we scored all candidates with the three-tower model, model inference P99 latency exceeded 70 ms which is our timeout cutoff.

To tackle this, we redesigned the serving flow as a two-stage scoring pipeline that leverages both two-tower and three-tower predictions.

Stage 1: Fast scoring for all candidates

In the first stage, we use the two-tower head from the co-train model to score all candidates. These scores are combined into an initial utility: the overall ranking score that estimates a candidate ad’s value for the request and determines which candidates survive for later selection. This follows the existing production setup, but is powered by the new co-train model.

This stage is fast and inexpensive enough to run on the full candidate set.

Stage 2: Focused refinement with three-tower scoring

In the second stage, we identify the top-K candidates by utility and rescore only this subset with the three-tower model. Candidates outside this utility topK bypass the three-tower path and retain their two-tower scores.

We introduced a new hyperparameter, utility topK, which controls how many candidates the three-tower model sees. We tested several choices of the utility topK and measured both latency and downstream metrics such as clickthrough and web conversion impressions.

The trade-offs we observed:

• Smaller utility topK values reduce latency but can hurt web conversion impressions, because the final top-K selection stage needs a sufficiently large pool to satisfy different campaign groups and deduplication constraints.
• Very large utility topK values allow more candidates into the three-tower stage but directly increase latency, so we needed to pick a value that balanced candidate coverage and serving cost.
• Retaining candidates outside utility topK for later stages mitigates mixshifts with only a small additional latency cost.

We ultimately chose a utility topK of 40K. This threshold roughly corresponds to the 70th percentile for Home Feed and Related Pins and the 90th percentile for Search, which means the three-tower model scores the majority of candidates on most requests while keeping P99 latency within budget.

Final selection and bias considerations

After we obtain updated utility scores from the three-tower model for the utility topK subset, we blend them with the two-tower dot-product utilities. The final topK selection step then runs on this blended set of scores, selects pre-defined quotas from several candidate sources and performs deduplication to select the final set of ads shown to the user.

We made two design choices which may create concerns:

1. Heuristic selection of the utility topK subset for three-tower scoring.
2. Blending two-tower and three-tower utilities before final top-K selection.

Both heuristics are designed to favor high-utility candidates. In principle, they could bias the system toward items that already scored well in the two-tower stage, especially if three-tower predictions further amplify those scores.

To monitor this, we looked at calibration and mixshift. Encouragingly, we observed that the three-tower model actually reduces over-calibration for auction candidates, moving predicted CTR closer to realized CTR. We also did not observe a drop in standard web conversion impressions when using the blending option.

Online results and cost

In online A/B experiments on standard ads, the three-tower co-train model delivered around 1% gains in CTR, gCTR30, and oCTR, along with nearly 1% reductions in CPC, with only a modest increase in GPU spend.

Taken together, this represents a strong trade-off: meaningful engagement and efficiency gains for advertisers and users, with a small and well-understood increase in infra cost.

Conclusion

In this post, we walked through how we took Nexus beyond two towers by introducing a three-tower engagement co-train model for standard ads. On the modeling side, the cross tower and inter module allow us to capture richer interactions between user behavior and candidate ads. On the systems side, a combination of model-level optimizations and a two-stage scoring flow let us deploy this more powerful architecture while staying within tight latency and cost constraints.

Looking ahead, this work opens up several promising directions:

• Extending similar three-tower and co-train ideas to other objectives beyond engagement.
• Exploring even richer sequence modeling and attention patterns now that we have a scalable framework for late-stage scoring.
• Further tightening the feedback loop between offline architecture exploration, online performance, and infra-aware serving design.

Most importantly, it demonstrates that with the right system abstractions, we can continue to innovate on model architectures without losing sight of real-world constraints.

Acknowledgements

We thank Qingyu Zhou, Yuchen Shen, Li-Chien Lee, Qingmengting Wang, Zhixuan Shao, Tristan Nee, Sihan Wang, Lida Li, and Nuo Dou for their contributions to this project, and Renjun Zheng and Jamieson Kerns for their leadership support.

References

¹Yang, Xiao, et al. “Beyond Two Towers: Re-architecting the Serving Stack for Next-Gen Ads Lightweight Ranking Models (Part 1).” Pinterest Engineering Blog

Beyond Two Towers: Launching the 3-Tower Engagement Co-Train Model (Part 2) was originally published in Pinterest Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

The Billion-Embedding Challenge

At Pinterest, the “signal” is our lifeblood. Whether it’s a home decor enthusiast finding the perfect rug or a fashion seeker discovering a new aesthetic, our discovery engine relies on understanding deep semantic relationships to help our users find inspirations. Over the last few years, the explosive growth of embedding-based retrieval has fundamentally transformed how we surface these signals — and at the heart of that transformation is Manas, Pinterest’s in-house distributed search platform.

Embedding Retrieval is one of the core capabilities of Manas, supporting multiple approximate nearest neighbor search algorithms, hybrid queries with both token and embedding clauses, as well as real-time updates to ensure fresh contents become searchable within seconds. Deployed on over 80 clusters and serving billions of embeddings, Manas embedding retrieval powers all major product surfaces at Pinterest including Home Feed, Search, Related Pins, Ads, and Notifications.

However, as our corpus scales toward tens of billions of embeddings and our models capture increasingly complex interactions, we face mounting challenges around cost efficiency, scalability, and flexibility. On the infrastructure side, traditional ANN algorithms like HNSW are notoriously memory-hungry — they require the entire index to reside in RAM to maintain low query latency, making cost grow linearly with corpus size. On the modeling side, the classic two-tower retrieval paradigm is too restrictive: it reduces each candidate to a single embedding and scores relevance through a simple dot product, leaving little room to express richer, context-dependent notions of similarity.

To tackle these challenges, our team has been evolving Manas’s embedding retrieval stack across three fronts:

• Quantization. We reduce the memory footprint of embedding indices by compressing vectors into lower-bit representations with fewer effective dimensions. Quantization has been rolled out to all major use cases, delivering over 50% memory reduction in embedding indices and 20–30% cost savings in serving infrastructure.
• SSD-based Serving. Rather than holding entire indices in RAM, we serve ANN queries directly from SSD by carefully bounding the I/O per request — sustaining high throughput with low tail latencies at a fraction of the memory cost. Early experiments demonstrate a 10x reduction in memory usage and 40% CPU savings compared to in-memory serving
• Multi-embedding Retrieval. We move beyond the single-vector-per-candidate constraint of the two-tower model by supporting richer scoring functions that consider multiple embeddings per candidate. This unlocks more expressive ranking at the retrieval stage. We are currently partnering with a product team to launch a pilot use case.

In this blog post, we will delve into the technical details and results of each initiative, and discuss what’s next for embedding retrieval in Manas.

Quantization: Redefining the Footprint of High-Recall Search

For a long time, serving embeddings in 16-bit or 32-bit float precision was considered the common practice. But at Pinterest’s scale, raw precision is a luxury that often provides diminishing returns. We discovered that quantization — transforming these high-dimensional float vectors into compact integer representations — is one of our most effective ways to better cost efficiency.

Evaluating the Trade-offs: SQ vs. PQ

In the Manas stack, we focused our implementation on two primary quantization algorithms: Scalar Quantization (SQ) and Product Quantization (PQ). The core of these methodologies lies in partitioning the vector space into disjoint subspaces and mapping each vector into an integer representation: SQ applies uniform discretization from float to integer per dimension, while PQ performs a K-Means clustering in each subspace and maps a subvector to the ID of the closest cluster centroid.

Benchmarks on a 100-million-embedding GraphSage dataset confirmed this intuition. We evaluated both SQ and PQ across two ANN algorithms (HNSW and IVF) and observed a clear trade-off: PQ achieves higher compression but with a significant recall decrease, while SQ delivers strong compression with minimal loss on recall.

• PQ reduces the HNSW index by 74% and the IVF index by 93%, with a recall in the range of 70–80%
• SQ reduces the HNSW index by 59% and the IVF index by 75%, with a recall over 90% consistently

Given the trade-off demonstrated by the offline benchmarking exercise, we ran online A/B experiments in production to select the best performing quantizer for each use case, and ensure negligible impact on user engagement metrics when enabling quantization. We launched SQ and PQ across major product use cases, reducing the total memory allocation significantly and realizing 20–30% cost savings for serving.

Technical Deep Dive: SIMD and Linear Scaling

Shifting to 8-bit or 4-bit representations isn’t just a memory win; it’s a compute challenge. Usually, SQ requires a decoding step before distance computation, which can become a CPU bottleneck. To solve this, we implemented Linear Scaling SQ, which quantizes a vector by scaling only, and thus eliminates the decoding step before distance computation. The key enabler here is SIMD intrinsics, which allows the CPU to perform multiple 8-bit integer operations with each instruction, and hence reduces the total computing resources needed per query by 10–15% in our use cases.

SSD Serving: Moving Beyond the Constraints of RAM

Our journey of improving embedding retrieval cost efficiency led us to exploring alternative ANN algorithms that take advantage of recent NVMe SSD performance advancements. The current generation of SSD devices are roughly an order of magnitude cheaper per gigabyte, but with the latency increased from nanoseconds to microseconds, which can slow down queries if I/O operations are not carefully managed. To bridge this gap, we experimented with I/O-aware ANN algorithms that were designed to minimize the number of random reads issued per query while retaining a recall score as good as memory-based algorithms.

DiskANN vs. SPANN

In our experiments, we benchmarked two disk-based ANN algorithms, DiskANN and SPANN, with a 100M-embedding corpus collected from a Pinterest Search use case. While DiskANN is a robust graph-based approach, SPANN emerged as the better option for the Manas use cases.

Our team’s key observation was applying PQ quantization to the on-disk embedding store while retaining the full precision centroids helped the search accuracy and throughput significantly — a slight tweak from the original SPANN paper. This makes SPANN+PQ 4.5x faster than plain SPANN and achieves over 3x the QPS of DiskANN with 1/3 the latency, with a slight 5% recall drop.

Implementing SPANN in Manas

We implemented the SPANN algorithm in Manas, which stores the centroid index in the memory and the large posting lists in the disk, and guarantees both disk-access efficiency (low latency) and high recall by effectively reducing the disk access number per request. In the index-building stage, we adopt the hierarchical balanced clustering algorithm from SPANN for selecting the centroids, which ensures evenly distributed cluster sizes, and hence similar lengths of posting lists to keep the tail latency low. We build the centroid index using HNSW, which is well-suited for in-memory search over a relatively small set of centroids. In a preliminary evaluation with a Pin recommendation use case that indexes over 5 billion embeddings, our SPANN implementation saves over 40% of CPU time for production queries when compared with HNSW, with a <5% recall drop. Our next step is to adopt SPANN across all major use cases.

Multi-Embedding Retrieval: The Shift Toward Late Interaction

As we improve cost efficiency, we are also evolving the expressivity of the Manas embedding retrieval stack. The traditional “Two-Tower” model, while efficient, collapses an entire Pin or query into a single vector, often losing the nuanced, token-level semantics that define high-quality discovery.

Contrast in Paradigms: Sum of MaxSim

We are now moving toward Late Interaction models , such as ColBERT. Unlike the single dot product of two-tower models, late interaction represents documents and queries as lists of vectors. We use the “Sum of MaxSim” scoring logic to capture the maximum similarity between each query token and the document’s tokens.

Integrating this into Manas required comprehensive changes in our serving stack. We integrated the multi-embedding retrieval as a new query type and updated Manas to handle multiple query embeddings rather than a single vector per query. This required updating the Manas query parser to understand these complex queries, as well as running multiple ANN searches from a multi-embedding query simultaneously. Currently, we are working with a client team to launch a pilot use case for the multi-embedding query support in Manas. This represents the next frontier of Pinterest search: moving from “two tower” to true model-based retrieval.

Looking Ahead

The future of vector search at Pinterest lies at the intersection of infrastructure efficiency and retrieval model innovation. Over the next five years, our central goal is to evolve Manas embedding retrieval into an architecture that is scalable, cost-efficient, and open to new retrieval paradigms. We are pursuing this along three directions: adopting SPANN and SPFresh to push CPU and memory costs even lower for billion-scale indices; building first-class support for multi-embedding retrieval models like ColBERT that enable richer, interaction-based scoring at the retrieval stage; and embracing GPU-based retrieval systems like SilverTorch and TIGER to unlock new model capabilities.

Acknowledgements

Many people from Core and Ads Delivery Infra teams contributed to the projects discussed in this blog post. Special thanks to Ellie Madsen, Jennifer Kong, and Jiawei Kuang for working on various Manas embedding retrieval projects. Thanks to our collaborators from client teams, including Bowen Deng, Jiaxing Qu, Ryan Hou, Minhazul Islam SK, Wei-Ting Lin, J.J. Hu, Konik Kothari, Yujiao Guo, Hanlin Lu, Bella Huang, Ai Zhang, Janvi Palan. Last but not least, many thanks to Van Lam, Tao Yang, Deeksha Sharma, Kartik Paramasivam, Abhishek Tayal, Zheng Liu for leadership support.

Evolving Pinterest’s Embedding Retrieval Platform was originally published in Pinterest Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Why VLM Serving Matters at Pinterest

Pinterest is a visual search and discovery platform, so its AI systems must reason over both language and visual content. Vision-language models (VLMs), which can interpret images, compare visual candidates, and respond naturally to user intent, are becoming the foundation for the next generation of Pinterest experiences: Pinterest Assistant, hybrid search, multimodal reranking, content understanding, signal generation, content safety, and more.

This direction also reflects Pinterest’s broader strategy to customize open-source models to meet its product & scale needs. Pinterest Assistant is a standout example. This multi-turn conversational experience covers both user language and visual content. Serving it requires low-latency VLM inference over rich multimodal context as well as reworking Qwen3-VL with proprietary multimodal embeddings to cut runtime cost while improving performance.

Serving VLMs, however, introduces more challenges compared to text-only LLM workloads. Requests may carry multiple images, require extra vision encoder computation, incur larger and more variable prefill cost, and create higher KV cache pressure. To support this new class of models & product experiences, we built Pinterest’s VLM serving stack on top of NVIDIA Blackwell GPUs and NVIDIA Dynamo. Blackwell GPUs incorporate many architectural innovations that are uniquely positioned for today’s most demanding AI workloads — including higher BF16/FP8 compute throughput, increased memory bandwidth, and larger HBM memory capacity — that enable dramatically higher performance for inference. Dynamo provides a distributed inference orchestration layer that gives us the flexibility to optimize multimodal workloads across the full serving path, including disaggregated encoder/prefill/decode serving, multimodal support in the Dynamo frontend and vLLM, multimodal KV-aware routing with custom payloads, and KV cache offloading.

The Serving Challenge

As mentioned above, serving VLM for Pinterest use cases present several challenges:

Request payloads and preprocessing: For text-only serving, the request payload is just text (or tokens), so preprocessing is mostly tokenization plus applying a standard chat template; there are no external assets to fetch and no image-specific constraints. For VLM serving, the payload includes both text and images (URLs, base64, or precomputed embeddings), so the serving stack must download or load images, run image preprocessing (resize, enforce min/max pixels, normalization), map them into the model’s multimodal input format, and build prompts that mix both text and image content while inserting any required vision tokens or projectors, otherwise images get dropped or misinterpreted. Complexity is increased as requests include more images — Pinterest use cases sometimes require sending thousands of images per request.

Expensive prefill: In text-only serving, the expensive part is typically decode, with relatively modest and uniform prompt lengths, so KV cache pressure is more predictable. In VLM serving, requests often include many images or content items per query, which makes prefill dominant (encoding visual context is costly), drives much larger and more irregular KV caches, and requires our serving stack to support KV-aware routing, cache offloading, as well as careful prompt design to stay within latency and memory SLOs.

Multi-turn workloads: In text-only serving, multi-turn chat mainly increases prompt length and token costs but stays within a uniform text interface, so the serving logic is mostly about truncation and history management. In VLM serving, multi-turn workloads combine long dialog history with repeated or evolving visual context (e.g., revisiting or adding images/boards across turns), which makes prefill much heavier, complicates how visual state is represented across turns, and requires benchmarks and SLOs that reflect realistic multi-turn multimodal interaction patterns rather than single-shot prompts.

KV cache memory pressure: For text-only models, KV cache growth is driven by text token counts and is relatively predictable per request and per turn, so standard cache sizing and eviction often suffice. For VLM models, large visual contexts and long conversations can produce far bigger KV states per request, so serving must treat KV as a first-class constraint — using KV-aware routing, offloading, and disaggregated Encoder/Prefill/Decode designs — to avoid frequent evictions and maintain throughput under multimodal, prefill-heavy traffic.

Custom model and payload support: Text-only serving can often treat models as interchangeable behind a standard chat/completions API with generic JSON payloads and minimal per-model customization. VLM serving, by contrast, typically requires model-specific support for image fields, multimodal content arrays, projector layers (e.g., custom embedding projections), and custom routing or headers; the serving stack has to understand these payload shapes and model capabilities explicitly, and deployment artifacts and routers must be able to encode and route these richer, non-uniform multimodal requests correctly.

All of these VLM serving challenges required us to build a robust, flexible system that can meet our multimodal requirements. To build our serving stack, we relied on NVIDIA Dynamo’s multimodal serving capabilities.

Scaling with Blackwell and Building on NVIDIA Dynamo

Pinterest has been an early industry pioneer in adopting NVIDIA GPUs for online model serving at internet scale, starting with recommendation systems and expanding into LLM and VLM serving. Token costs and performance matters significantly for the viability of these products. Building on that foundation, we standardized on NVIDIA Blackwell GPUs B200 due to their market leading TCO for LLM/VLM inference. Blackwell also makes our LLM/VLM hardware stack future proof as our use cases and models continue to evolve. This cutting edge hardware enables Pinterest to be able to continue to get better TCO over time as we explore quantizations, improved kernels and multi-node inference.

Our in-house Gen AI Serving Solution is an end-to-end customizable stack centered around NVIDIA Dynamo as the serving orchestration framework. We use the OpenAI Chat Completions API, model-based Envoy routing and a model-aware gateway, Model Router, to provide a centralized way for all customers to call our system, ensuring a smooth client experience. Under the hood, we use vLLM as our inference engine and Weights and Biases for model management. Our compute infrastructure, PinCompute, is built on Pinterest’s internal centralized platform infrastructure that leverages AWS Elastic Kubernetes Service (EKS) along with NVIDIA GPUs. With the help of the Infra org, we manage dedicated EKS clusters that host all Gen AI Serving use cases. Notably, this is one of the first PinCompute EKS (PEKS) use cases at Pinterest. The Dynamo operator and components are installed through Helm charts, and our Dynamo workloads use the DynamoGraphDeployment CRD deployed as K8s manifests. To tailor the deployments to Pinterest’s requirements we inject additional sidecars and add custom containers to support functionality like Envoy (service mesh), model loading, and metrics scraping.

Our journey to Dynamo started with evaluating several Kubernetes-native frameworks that we found easy to start with but lacked flexibility in traffic management or forced reliance on a single inference ecosystem. We ultimately selected Dynamo as it is Kubernetes native, compatible with Pinterest Kubernetes and service discovery solution, inference engine agnostic, provides a flexible traffic solution, and uses a performant Rust-based router. During this process we developed a close relationship with the NVIDIA Dynamo team who have provided us with exceptional support. Pinterest utilizes many key features of Dynamo that provide flexibility and performance optimizations when powering our Gen AI Serving Stack.

P/D disaggregated serving

Our platform uses disaggregated prefilling and decoding inference to tailor serving to specific latency requirements (Time-to-First-Token (TTFT) or Inter-Token Latency (ITL)), optimizing hardware allocation by separating the distinct computational phases of LLM requests. This architecture is particularly effective for unblocking product launches with high traffic volume and tight latency requirements, especially for TTFT. Dynamo provides an easy to use solution to orchestrate distributed, disaggregated inference that allows us to explore the Pareto curve between the ratio of encoder (E), prefill (P), and decode (D) workers.

KV cache offloading

We leverage KV cache offloading, specifically via LMCache, for multi-tier offloading to CPU memory and disk, which is critical in high QPS, multi-turn scenarios. LMCache serves as a sophisticated extension for the inference engine, providing tiered storage across GPU, CPU DRAM, and local disk (NVMe) to preserve generation latency while managing high GPU memory pressure. This tiered approach includes asynchronous prefetching and compression, contributing to lower TTFT and increased throughput by effectively managing long-context scenarios where visual tokens would otherwise overwhelm available VRAM. LMCache fits seamlessly into Dynamo as one of the many KV cache integration option for KV cache offloading

Multimodal support in Dynamo frontend/vLLM

Pinterest’s image-based products have specific multi-modal serving requirements. We worked closely with the NVIDIA Dynamo team to develop corresponding multimodality features, including a frontend image decoder, multi-modal disaggregated serving, and multimodal KV router support to reduce recomputation for VLMs. These enhancements enable the serving stack to handle complex multimodal payloads, such as base64 encoded images or image URLs, and perform necessary image preprocessing directly in the Dynamo frontend. Furthermore, the implementation of multimodal KV-aware routing allows the system to track prefix cache overlap for visual content, which is essential for maintaining production latency in multi-turn interactions with high visual token counts.

Custom modality support: Projection Embeddings

Pinterest Assistant workloads often need to reason over large visual contexts: Pins, boards, products, and other image-heavy inputs that may appear across multi-turn interactions. Sending all of that context as raw image pixels is expensive for VLM serving because each request may require image loading, decoding, preprocessing, and online vision encoder computation before the language model can use the visual information. To reduce that cost, we added support for projection embeddings using PinCLIP, Pinterest’s internal image encoder, for generating Pin embeddings. Instead of sending raw images through the serving path, Assistant requests can send precomputed PinCLIP embeddings. Dynamo and the underlying inference engine then run a projector that maps those precomputed embeddings into the target VLM’s native visual token space, letting us reuse visual representations that already exist for many Pinterest entities while avoiding the most expensive parts of pixel-based image serving.

The performance impact of this approach has been significant. Compared with pixel-based image inputs in Dynamo, incorporating projection embeddings into Dynamo have yielded results that are substantially faster across our benchmarks: average gains are roughly 85x faster TTFT, 7.3x faster end-to-end latency, and 1.1x faster TPOT. Peak gains are even larger, reaching approximately 369x faster TTFT, 44x faster end-to-end latency, and 2.6x faster TPOT. Just as importantly, this makes much larger visual contexts practical: requests with 250 images represented as PinCLIP visual tokens reached latencies comparable to pixel-based requests with roughly 10 images, while carrying 25x more visual context. Even at that scale, the serving profile remained reasonable and production-ready.

Supporting this required changes across the API, artifact, serving, and engine layers. We introduced an updated ChatCompletions request format for projection embeddings, defined a model artifact contract so training and serving teams could package projector weights, model weights, and configs together into a single model artifact, added the new modality path in vLLM alongside image and video to decode embeddings, validate types and shapes, invoke the correct projector, and insert projected visual tokens into the model input sequence, and updated Dynamo to accept and route the new request format while preserving the multimodal contract. Adding multimodal KV-aware routing support for our modality delivered meaningful tail-latency gains: the strongest result improved TTFT p99 by 5.92x, and across the full benchmark matrix Multimodal(MM) KV-aware routing delivered about 1.42x average speedup on TTFT p99. Additionally, because embedding payloads are larger than ordinary text inputs, vLLM frontend processing latency became a bottleneck in some cases; Dynamo’s Rust frontend helps by providing an efficient path for receiving, parsing, and forwarding larger multimodal payloads.

Tool calling
Lastly, Dynamo fully supports tool calling with custom chat templates and multi-modal inputs, which is essential for providing flexibility for Pinterest’s agentic AI systems. This capability allows our agents to interact with internal tools and APIs in the Pinterest ecosystem, enabling more complex workflows that go beyond simple text for text and hybrid search, and other internal services. By leveraging custom chat templates, we can precisely define how the model should format its tool requests and handle the subsequent tool outputs, ensuring seamless integration with Pinterest’s internal services. Furthermore, the support for multi-modal inputs in tool calling means our agents can use visual information to inform their tool use, such as identifying an object in an image and then calling a specific search or recommendation tool to find similar products.

Benchmarking Real Multimodal Workloads with AIPerf

We use AIPerf, NVIDIA’s distributed benchmarking tool for standardizing our AI inference performance measurement, as the execution layer for our performance benchmarks. AIPerf is designed as a modular benchmarking framework, which makes it a better fit for complex generative AI workloads than tools focused mainly on single request/response patterns.

This is especially important as both Pinterest and the broader industry move toward agentic AI systems. These workloads are rarely a single model call. They often involve retrieval, routing, multiple model calls, tool use, multimodal inputs, and intermediate reasoning steps before producing a final response. This allows our optimizations to have grounding data and guardrails on whether we are improving or regressing.

AIPerf’s DAG support is a big part of why it works well for us. Instead of flattening an agentic workflow into one artificial request, we can model the actual execution graph: nodes represent meaningful stages in the system, and edges capture dependencies between steps. This lets us benchmark workflows that branch, fan out, join, or depend on earlier outputs, patterns that are increasingly common in real AI applications.

Just as importantly, AIPerf lets us shape the benchmark traffic to look more like real production usage. We can run benchmarks with configurable QPS, realistic Poisson request arrival patterns, multi-turn interactions, multimodal inputs, and configurable prompt characteristics such as system prompt size, prefix length, input token length, and number of visual/embedding items. This makes the benchmark less about testing an isolated model call and more about understanding how the full workload behaves under realistic load.

Internally, we pair AIPerf’s execution output with Pinterest-specific reporting. We use the results to power dashboards and summaries for latency, throughput, token usage, success rate, per-request details, and aggregate comparisons. That gives teams a practical way to compare runs, catch regressions, and understand whether a model or deployment can meet production SLOs under realistic multimodal and agentic workloads.

Product Use Cases Enabled

The VLM serving stack described above was built to support Pinterest Assistant, but the same architecture now serves as a reusable foundation for many GenAI and multimodal use cases across Pinterest. By standardizing on Dynamo for orchestration, vLLM for inference, and a common Chat Completions-compatible API, teams can launch new model-backed product experiences on top of this extensible serving platform.

Pinterest Assistant: Real-Time Multimodal Interactions

Pinterest Assistant is one of the first major product use cases enabled by this stack. As a conversational agent, Pinterest Assistant needs to support natural multi-turn interactions while reasoning over Pinterest’s visual content. A user may ask for help refining an idea, exploring a style, comparing products, or finding inspiration based on a set of Pins or images. Unlike a text-only assistant, this requires the serving system to handle both dialogue history and multimodal context in real time. Pinterest Assistant inference runs on NVIDIA B200 instances, which showed a greater than 2x latency improvement over Hopper during preliminary benchmarking.

Pinterest Assistant also benefits from custom modality support such as projection embeddings. Instead of always sending raw image pixels through the serving path, Assistant requests can use precomputed visual embeddings for Pinterest entities such as Pins, boards, and products. This allows the model to reason over much larger visual context while avoiding repeated image decoding and vision encoder computation, making richer real-time conversations practical.

A Shared Stack for Multimodal Product Patterns

As more Pinterest product surfaces adopt GenAI and multimodal models, this shared stack lets us support a growing range of patterns: conversational agents, re-rankers, OCR, safety systems, signal generation, and future VLM-powered experiences. The result is a serving platform that is not tied to a single product launch, but designed as a reusable foundation for multimodal AI at Pinterest.

Although Pinterest Assistant motivated many of the original requirements, the serving stack has grown to support a much broader set of use cases. Dynamo has become the out-of-the-box default for many GenAI serving workloads at Pinterest because it offers a flexible path for both text-only and multimodal deployment:

• Multimodal reranking uses VLMs to compare candidate content across textual and visual signals
• OCR workloads extract or reason over text in images
• Safety guardrails apply multimodal understanding to check whether responses or retrieved content meet product and policy requirements
• And more across signal generation, embedding-based workflows, and agentic systems

Dynamo’s LoRA hot loading has also accelerated experimentation under limited GPU capacity. Instead of standing up a separate full deployment per adapter — which increases GPU usage and operational overhead — client teams can load and evaluate multiple sets of LoRA weights dynamically against an existing base model. This shortens experimentation cycles and creates a smoother path from adapter training to production validation.

With a common serving foundation, teams reuse the same APIs, deployment patterns, routing layer, model management, observability, benchmarking, and GPU infrastructure rather than each building a custom solution. This gives product teams a paved path to focus on model behavior, integration, and evaluation. Dynamo is powering a reusable foundation for multimodal AI at Pinterest.

Lessons Learned and What’s Next

In building our Gen AI Serving platform, we’ve learned that VLM workloads are fundamentally prefill-heavy and cache-sensitive: encoding large visual contexts and long histories, not just decode, drives both latency and GPU memory utilization, so KV-aware routing, cache offload tiers, and disaggregated serving need to be designed explicitly. Dynamo’s multimodal KV-aware router and E/PD disaggregation and LMCache-based KV offloading turned out to be essential. We also found that payload design and routing are core serving problems, not just interface glue: the way we encode multimodal content arrays, choose image resolutions, and structure prompts directly determines whether Dynamo can reuse prefixes, route efficiently, and keep TTFT within product targets. On the evaluation side, we learned that benchmarks must reflect real multimodal product traffic — including multi-turn conversations, many images per request, and agentic DAGs — so we invested in AIPerf-based DAG benchmarks that mirror production QPS patterns instead of synthetic single-shot prompts. Finally, a shared serving platform built on Dynamo, vLLM, and EKS has significantly accelerated experimentation: once the stack supported multimodal routing, KV offload, and model management, new use cases like Pinterest Assistant and multimodal reranking could launch by reusing the same paved path instead of re-inventing infra per team.

Looking ahead, we’re investing in several new directions.

AI Configurator: Dynamo’s AI Configurator is a performance optimization tool that can simulate 10K+ deployment configurations in seconds, finding optimal prefill/decode worker counts, tensor/expert/data parallelism settings, and deployment parameters. It evaluates both aggregated and disaggregated serving architectures, and uses hardware-specific performance models to predict TTFT, ITL, and throughput across different GPUs. This tooling can help us create optimized deployments with lower lift, increasing performance and developer velocity across teams.

Dynamo Planner: As our workloads scale, so will the need to introduce autoscaling in order to maintain a highly available yet cost-efficient compute infrastructure. Dynamo’s Planner will provide a VLM/LLM-optimized autoscaler, which dynamically adjusts prefill and decode replica counts through four optimization targets: throughput (static queue/KV thresholds), latency (aggressive low-latency thresholds), load (user-defined prefill queue and decode KV utilization thresholds), and SLA (regression-based models targeting specific TTFT/ITL values)

Conclusion

NVIDIA Dynamo has given us a strong foundation for building Pinterest’s VLM serving stack and expanding it across emerging multimodal use cases. Its flexibility has been critical as we move from individual product launches toward a shared platform for production GenAI serving.

We’re excited to continue partnering with the NVIDIA Dynamo team and the broader community to push the limits of VLM and multimodal serving, and to make real-time multimodal AI systems faster, more efficient, and easier to deploy at scale.

Acknowledgements

This work would not be possible without the contributions from our partners and collaborators. Our thanks to:

Pinterest

AI Platform: Neha Upadhyay, Ananya Prabhu Angadi, Nazanin Farahpour, Howard Nguyen
Product ML Infra: Li Tang, Yayun Wang, Archer Liu
ATG: Yash Upadhyay, David Xue
Cloud Runtime Team: Vaibhav Shankar
CDP: Khoi Nguyen
Traffic: Peter Leng, James Fish, Scott Beardsley
Production Engineering: One Marino, Juan Pablo Daniel Borgna
Product Management: Colin Leatherbury
Leadership: Karthik Anantha Padmanabhan, Bo Liu, Roger Wang, Kartik Paramasivam, Matthias Zenger

NVIDIA
Elijah Soba, Qi Wang, Anthony Casagrande, Guan Luo, Kris Hung, Ryan McCormick, Harry Kim, Akshatha Kamath, Matthew Rawson

Building Pinterest’s VLM Serving Stack on NVIDIA Dynamo was originally published in Pinterest Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.