<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ben Weber on Medium]]></title>
        <description><![CDATA[Stories by Ben Weber on Medium]]></description>
        <link>https://medium.com/@bgweber?source=rss-a80e1f69e782------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*0Ku0pfCur-EdgYFFGHeboA.jpeg</url>
            <title>Stories by Ben Weber on Medium</title>
            <link>https://medium.com/@bgweber?source=rss-a80e1f69e782------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 06:45:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@bgweber/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com" rel="hub"/>
        <item>
            <title><![CDATA[Transformers for Mobile User Acquisition]]></title>
            <link>https://aiadvances.org/transformers-for-mobile-user-acquisition-f3816d8b0c41?source=rss-a80e1f69e782------2</link>
            <guid isPermaLink="false">https://medium.com/p/f3816d8b0c41</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[gaming]]></category>
            <dc:creator><![CDATA[Ben Weber]]></dc:creator>
            <pubDate>Tue, 10 Nov 2026 12:32:36 GMT</pubDate>
            <atom:updated>2026-11-10T12:32:36.542Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jOHv5FneizGkulR2" /><figcaption>Photo by <a href="https://unsplash.com/@aldwardcv31?utm_source=medium&amp;utm_medium=referral">Aldward Castillo</a> on <a href="https://unsplash.com?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4><em>A state of the art transformer architecture for install prediction.</em></h4><p>Transformer architectures have become the state of the art for modeling user behavior in ads and recommendations. Meta <a href="https://arxiv.org/abs/2402.17152">[1]</a>, Pinterest <a href="https://arxiv.org/abs/2205.04507">[2]</a>, and Alibaba <a href="https://arxiv.org/abs/1905.06874">[3]</a> have all published transformer-based sequence models that outperform earlier approaches, and the leading mobile ad platforms now describe their systems in similar terms. For example, Moloco writes that CARA <a href="https://www.moloco.com/blog/inside-molocos-cara-the-ai-that-compounds">[4]</a> trains “multi-head models, transformer-based and other sequence models, and deep cross networks,” and that its sequence models “learn from cross-app event histories to build dense user representations.” Liftoff <a href="https://aws.amazon.com/blogs/industries/liftoff-improved-conversion-performance-and-reduced-infrastructure-costs-with-cortex-using-aws-graviton">[5]</a> and Bigabid <a href="https://www.bigabid.com/how-dnns-predict-high-intent-users-faster/">[6]</a> describe moving from simpler models to neural networks with embeddings and attention, and AppLovin <a href="https://www.applovin.com/en/archived-blog/how-we-drive-value-and-handle-data">[7]</a> describes a system that learns from many streams of data. InMobi <a href="https://advertising.inmobi.com/company/press/inmobi-advertising-unveils-helix-a-groundbreaking-ai-layer-that-delivers-full-funnel-intelligence-for-the-inmobi-dsp">[8]</a> describes multi-objective neural networks for bidding, and Unity’s head of ads <a href="https://mdm.co/podcast-understanding-unitys-vector-with-felix-the/">[9]</a> has described Vector as one continuous learning model with a multi-head approach, replacing an earlier deep learning model.</p><p>Transformers are also the architecture behind large language models and much of today’s AI infrastructure <a href="https://arxiv.org/abs/1706.03762">[10]</a>. In this post, we’ll work through a lightweight version of a transformer. We won’t be working with context windows of a million tokens or models with billions of parameters, but the core idea is the same. An LLM represents a sentence as a sequence of tokens and learns to predict the next word, and in the same way, we can represent a device’s history as a sequence of tokens and train a transformer to predict which app the device will install next.</p><p>There are two reasons why transformers are such a good fit for mobile DSPs. The first is that they scale with data. A transformer keeps getting better as you give it more events and more compute, and research on sequential recommendation models has found scaling laws similar to those for LLMs <a href="https://arxiv.org/abs/2311.11351">[11]</a>. Mobile ad platforms have plenty of both data and compute. Moloco has written <a href="https://cloud.google.com/blog/products/containers-kubernetes/moloco-uses-gke-and-tpus-for-ml-workloads">[12]</a> that its platform ingests 10 petabytes of data a day and handles bid requests at a peak of 10.5 million queries per second, and that CPUs were not competitive for training at its scale. This led Moloco to train on Cloud TPUs, which cut its training times by up to ten times. Training workloads at this scale are what make specialized hardware worth the investment.</p><p>The second reason is that transformers can learn from non-attributed data. A transformer can be pretrained to predict a device’s next event from its history, which means that every install, purchase, and app session can be used as a training example, whether or not it was attributed to your DSP. This is a huge advantage for companies like AppLovin <a href="https://www.applovin.com/en/archived-blog/how-we-drive-value-and-handle-data">[7]</a>, which describes training on data streams beyond its own earned impressions, such as mediation auction outcomes, gaming usage patterns, and SDK and pixel data. Unity <a href="https://mdm.co/podcast-understanding-unitys-vector-with-felix-the/">[9]</a> is in a similar position, since Vector is built to take in very large volumes of gameplay data from the Unity engine, linking gameplay behavior, monetization signals, and campaign performance. Appodeal, which also runs mediation and its own exchange, has written that its models are deep neural networks and that more data typically equals better results <a href="https://appodeal.com/blog/ad-techs-ml-reality-check/">[13]</a>. Most DSP models only learn from the impressions they win, and a transformer removes that limit.</p><p>However, most of these posts are written for marketing purposes and skip over the details. They name the ingredients without providing the recipe. If you’re an MLE at a DSP, or leading a team that is deciding where to invest, the questions that matter most are the ones these posts don’t answer. What are the model inputs? What is the model trained on, when most conversions are never attributed to your DSP? How do you serve a large model inside an auction that has a latency budget of a few milliseconds? And how do you keep predictions calibrated, so that a bid of two dollars reflects the actual value of the impression?</p><p>In this post, I’ll walk through a state of the art architecture for mobile DSPs that is aligned with the methods these companies describe. I’m not claiming that this is what any of them run. The goal is to show what could be going on under the hood, with enough detail that a team could build it. In a previous post <a href="https://medium.com/@bgweber/pretrained-deep-learning-intent-signals-for-mobile-dsps-2eb1834c62db">[14]</a>, I described a simpler approach, where pretrained deep learning models generate intent signals that feed a boosted model for install and purchase prediction. This post builds on that approach, but takes advantage of the full power of a transformer. During my career I’ve worked on bidding models at Chartboost and Aarki, and many of the design choices in this post come from seeing which parts of the stack were difficult to get right in production.</p><p>We’ll focus on two predictions, whether an impression results in a click and whether it results in an install. To show why the transformer is worth the added complexity, I’ll walk through how feature engineering for these predictions has evolved over four stages, where each stage addresses a limitation of the one before it.</p><ol><li>Logistic regression over hashed one-hot features and hand-built crosses.</li><li>Embeddings for categorical and many-hot features, feeding a neural network.</li><li>Event sequences pooled with attention.</li><li>A transformer over tokenized events from many sources, pretrained on non-attributed data.</li></ol><p>Then we’ll cover how to co-train click and install heads on top of the transformer, how to serve it efficiently, and how to use a LightGBM model as the final step for calibration while still taking advantage of the embeddings learned by the transformer. The result is a state of the art architecture that is inexpensive in the bid path, learns from large-scale non-attributed data, and produces calibrated probabilities that can be used directly for bidding.</p><p>The figure below shows the architecture that we are building towards. It has three stages, pretraining a transformer on non-attributed data, co-training click and install heads on attributed impressions, and serving the model with dot products and a LightGBM model for calibration. Rather than starting with the full design, we’ll work towards it in stages, so that it’s clear why each component is needed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jmh0jPa6m3kTJhhm3FA5zQ.png" /><figcaption>The full architecture, from raw events to calibrated bids</figcaption></figure><h3>The prediction problem</h3><p>A DSP bidding on user acquisition campaigns needs to estimate the probability that an impression will result in an install. For a CPI or CPA campaign, the bid is roughly this probability multiplied by the value of the install to the advertiser, and then shaded for the auction. If the probability is off by 20%, then the bid is also off by 20%, and the DSP either overpays for the impression or loses the auction. This means that the model needs to do more than rank impressions well, it also needs to be calibrated.</p><p>Installs can happen through two different paths. With the click path, the user clicks on the ad, is sent to the app store, and installs the app. With the view path, the user sees the ad but does not click, and then installs the app later within the view-through window allowed by the MMP, which is usually around 24 hours. MMPs give priority to clicks and attribute the install to the last touch. To reflect both paths, we can define the install probability with the following equation.</p><pre>P(install) = p_click × p_install|click + (1 − p_click) × p_install|view</pre><p>This results in three quantities that the model needs to learn, a click head and two install heads, one for each path. It’s useful to split out the paths, because they behave differently. Click-through installs depend heavily on the creative and placement, while view-through installs depend more on the user’s existing intent. View-through installs are also where impression spamming and attribution fraud tend to show up, so it’s helpful to calibrate and monitor each path separately.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gbia9LLSCA_ppltmEq-vzA.png" /><figcaption>An install arrives by one of two paths</figcaption></figure><p>The click head and the two install heads combine into a single install probability for each impression.</p><p>The other half of the problem is the data. A DSP works with two different types of data.</p><ul><li><strong>Attributed data.</strong> These are impressions won by the DSP that are joined with MMP postbacks for the clicks, installs, and in-app events that the MMP credited to the DSP. This is the only data set with impression-level labels, but it’s small, it only covers auctions that the DSP won, and it’s biased towards the impressions that the current model chose to bid on.</li><li><strong>Non-attributed data.</strong> This is everything else that the DSP can legitimately see about a device, such as install and purchase events that advertisers share through their MMP even when another network was credited, the bid stream, and SDK data for platforms that have an SDK. This data does not have impression labels, but it’s orders of magnitude larger and describes what devices are actually doing across the market.</li></ul><p>Most DSP models only learn from the first type of data. The transformer is what makes the second type usable. The architecture in this post uses a transformer to learn a representation of each device from non-attributed data, and then uses attributed data only for the components that require impression-level labels.</p><h3>Stage 1. Logistic regression over hashed one-hots</h3><p>The first generation of DSP models used logistic regression, and there were good reasons for this. Logistic regression can train on billions of records, serves predictions in microseconds, and produces probabilities that are close to calibrated without additional work. Liftoff has written <a href="https://aws.amazon.com/blogs/industries/liftoff-improved-conversion-performance-and-reduced-infrastructure-costs-with-cortex-using-aws-graviton">[5]</a> that its stack was built on logistic regression before Cortex, and this was common across the industry, including the teams that I worked on.</p><p>The model itself is simple, and most of the work goes into feature engineering. Each raw field in the bid request is converted into a set of one-hot indicators, and the model learns a weight for each indicator. Here are the main types of features used in this approach.</p><ul><li><strong>Categoricals.</strong> Fields such as publisher app, placement, ad format, OS, OS version, device model, country, region, carrier, connection type, campaign, creative, and hour of day. Each value becomes an indicator.</li><li><strong>Many-hots.</strong> Fields that can have several values at once, such as the IAB categories of the publisher app or the set of apps installed on a device. Each value that is present turns on its own indicator.</li><li><strong>Numerics.</strong> Values such as bid floor, days since the device was first seen, and the number of impressions shown to the device today. These are bucketed into ranges, because a linear model cannot learn a nonlinear relationship on its own.</li><li><strong>Crosses.</strong> This is the most important part of the feature set. A linear model cannot learn that a puzzle game creative performs well on puzzle publishers unless it has an indicator for that pair, so teams hand-build crosses such as publisher by campaign, country by campaign, and device model by creative.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GUwMTRX7uf9_DDffl8NdrA.png" /><figcaption>Stage 1, hashed one-hots and crosses</figcaption></figure><p>Every interaction that the model uses needs to be defined by hand as a cross.</p><p>The vocabulary for these features grows quickly. For example, publisher by campaign alone can result in hundreds of millions of pairs. To keep the model a fixed size, the indicators are mapped to a fixed number of buckets using the hashing trick, often 2²⁴ or more, and the model learns a weight for each bucket with L1 or L2 regularization. Training typically uses FTRL <a href="https://research.google/pubs/ad-click-prediction-a-view-from-the-trenches/">[15]</a> or a similar online method, so that the model can update as new data arrives.</p><p>This approach has three limitations that motivate the later stages.</p><ol><li><strong>There’s no sharing between related values.</strong> The weight for one puzzle game provides no information about another puzzle game. Each publisher, campaign, and pair needs to learn its weight from its own data, which means that new and small values are poorly estimated.</li><li><strong>Crosses need to be hand-built.</strong> People decide which pairs matter, the number of pairs grows rapidly as fields are added, and interactions that no one thought of are never learned.</li><li><strong>The model has no memory of the device.</strong> The model sees the bid request and little else. If device history is used at all, it’s through a few hand-crafted counters, such as installs in the last 30 days, which discard which apps were installed and in what order.</li></ol><p>Factorization machines and field-aware factorization machines <a href="https://www.csie.ntu.edu.tw/~cjlin/papers/ffm.pdf">[16]</a> were the first improvement to this approach. They assign a small vector to each feature value and model the interaction between two values as the dot product of their vectors. This means that related values can share information, and every pair has an interaction term without needing to be hand-built. The idea of learning a vector for each value is what leads to the next stage.</p><h3>Stage 2. Embeddings for categoricals and many-hots</h3><p>The second stage replaces a single weight for each value with a learned vector for each value. Each categorical field gets an embedding table, which maps each value to a vector of 8 to 64 numbers that is learned during training. For example, publisher app, campaign, creative, device model, and country each become dense vectors, and a neural network on top of these vectors learns how they interact. This is the family of models that includes Wide &amp; Deep <a href="https://arxiv.org/abs/1606.07792">[17]</a>, DeepFM <a href="https://arxiv.org/abs/1703.04247">[18]</a>, and DCN <a href="https://arxiv.org/abs/2008.13535">[19]</a>, which most large ad and recommendation systems moved to in the late 2010s.</p><p>The main benefit of this approach is sharing. Two puzzle games that attract similar users will end up with similar vectors, so a new puzzle game with limited data can benefit from the games that the model already knows about. Interactions no longer need to be hand-built, because the network, or an explicit cross layer in a deep cross network, learns them from the vectors.</p><p>Many-hot fields need an additional step, because a device can have any number of values in a set. The typical approach is to look up a vector for each value that is present and then pool these vectors into a single fixed-size vector. Here are the most common pooling options.</p><ul><li><strong>Mean pooling.</strong> This summarizes the overall profile, such as the average of the app categories that a device has installed.</li><li><strong>Max pooling.</strong> This keeps the strongest signal in each dimension, so that a single rare but informative value is not averaged out.</li><li><strong>Mean and max pooling.</strong> This is a good default that works well in practice, where each field contributes both vectors to the input of the network.</li></ul><p>Numerics are standardized and passed in directly, or bucketed and embedded like a categorical. Embedding the buckets enables the model to learn a nonlinear relationship for each feature without additional work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NeHeLZK2l0bwWUV7bPToJw.png" /><figcaption>Stage 2, embeddings and pooling</figcaption></figure><p>The network learns the interactions that needed to be hand-built in stage 1.</p><p>This is a large improvement over stage 1, but there’s still a gap. A device’s installed apps are now represented as a pooled set, which describes what the device has but not when it installed each app or in what order. Two devices with the same five apps look identical, even if one device installed them all last week and the other installed them over three years. Recency and order are important for predicting the next install, which leads to the next stage.</p><p>This stage also introduces a practical cost. Embedding tables for high-cardinality fields such as publisher app, device model, and campaign can get large, and the network now needs to run in the bid path for every candidate campaign. This is where serving starts to become a challenge, and it’s one of the reasons that many DSPs stayed with linear models or gradient boosted trees longer than the research papers would suggest. We’ll come back to serving later in this post.</p><h3>Stage 3. Event sequences with attention</h3><p>The third stage represents a device’s history as a sequence of events rather than a set. Each install or purchase becomes a token, and each token is the sum of a few small embeddings.</p><ul><li><strong>App.</strong> The app where the event happened, using the same embedding table as the ad side of the model, so that a device’s history and a candidate campaign’s app are in the same space.</li><li><strong>Category.</strong> The category of the app, which helps rare apps learn from common apps.</li><li><strong>Event type.</strong> Whether the event is an install or a purchase.</li><li><strong>Spend band.</strong> For purchases, an order-of-magnitude bucket for the amount.</li><li><strong>Age band.</strong> How long before scoring time the event happened, bucketed into ranges such as less than a day, less than a week, and less than a month.</li></ul><p>The age band captures time, so the model does not need a positional encoding to know which events are recent. We keep the most recent 64 or so events, sorted from newest to oldest.</p><p>Attention is how the model reads this sequence. There are two common approaches, and they have very different serving costs.</p><ul><li><strong>Self-attention with pooling.</strong> Each event attends to the other events in the history, which enables the model to learn combinations such as a game install followed by a purchase in the same week. The outputs are then pooled with mean and max into a single vector for the device. Nothing in this step depends on the ad being scored, which means that it can be computed once per device.</li><li><strong>Target attention.</strong> The app of the candidate ad is used as the query, and the model weights each past event by how relevant it is to that app. This is the approach used by the Deep Interest Network <a href="https://arxiv.org/abs/1706.06978">[20]</a>, and Bigabid’s description <a href="https://www.bigabid.com/how-dnns-predict-high-intent-users-faster/">[6]</a> of attention that focuses on the most predictive signals sounds similar. It’s often more accurate, but because it depends on the candidate, it needs to run in the bid path once for each campaign.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u1n4LKTRKn1glosBUdhtPw.png" /><figcaption>Stage 3, two ways to attend over a device history</figcaption></figure><p>The self-attention approach works well with an inexpensive bid path, and the rest of this post builds on it.</p><p>In a related domain that I work in, adding a single self-attention layer over a dated event history made the history the most important input to the model by a wide margin, ahead of every static feature. The takeaway is that most of the signal is in the history, and even a lightweight encoder can capture most of it.</p><p>There are two limitations with this stage. First, the encoder is only as good as the labels used to train it. If the sequence encoder learns only from attributed impressions, then it gets one weak label for each won impression, and most devices never have a won impression that converted. Second, pooling removes order. The age bands capture recency, but patterns that depend on the order of several events, such as trying out three games in a genre before settling on one, are difficult to represent after mean and max pooling. Both of these limitations point to the same solution, which is to train the sequence model on its own task using data that does not require attribution. This is exactly what a transformer is designed to do.</p><h3>Stage 4. A transformer over tokens</h3><p>The fourth stage is the state of the art, and it’s where claims about foundation models and dense user representations come from. The model is a transformer over the device’s event history, but the architecture itself is not the main source of improvement. The improvement comes from two things that the transformer makes natural, a token format that can represent events from any data source, and a training task that does not require attribution.</p><h3>Tokens from every source</h3><p>Once every event is represented as a token, merging data sources means writing more tokens into the same timeline. Each token keeps the embeddings from stage 3 and adds a source embedding, which enables the model to learn how much to trust each source and how sources interact. Here are the token types that I would start with.</p><ul><li><strong>Install.</strong> From MMP data shared by advertisers. An input and a next-event target, one token per install.</li><li><strong>Purchase, with spend band.</strong> From MMP revenue postbacks. An input and a next-event target, one per purchase or per app per day.</li><li><strong>First seen in app.</strong> From the bid stream. Input only, one per app, the first time the bidder sees the device there.</li><li><strong>Weekly engagement, with volume band.</strong> From the bid stream or an SDK. Input only, one per app per week, capped at the latest few weeks.</li><li><strong>Gone dormant.</strong> From the bid stream. Input only, one when a device stops appearing in an app for several weeks.</li><li><strong>Profile context.</strong> From the bid request and device data. Input only, one token at position zero.</li></ul><p>The bid stream is the most interesting addition. If a device shows up in an app’s bid requests, then the app is installed and in use, which means that the first time your bidder sees the device in an app is an inferred install. This works for every app that sells ads, not only the apps of your own advertisers. Similarly, when a device stops appearing in an app, this is an inferred churn. These tokens fill in many of the gaps in MMP coverage, but they need to be aggregated. A device can generate thousands of bid requests a day, and without aggregation these requests would push every install and purchase out of the context window.</p><p>The profile context token describes what the device is rather than what it did, such as country, OS, device model tier, and connection type, pooled in the same way as the many-hots in stage 2. Placing it at position zero means that attention can use it, and a device with no history still gets a valid state. This is important for mobile, because many devices, especially on iOS, have short histories.</p><h3>A training task that needs no attribution</h3><p>The transformer is trained with a causal mask to predict the next outcome event from all of the events before it, which is the same objective used by SASRec for next-item recommendation <a href="https://arxiv.org/abs/1808.09781">[21]</a>. At every position where the next token is an install, the model predicts which app was installed, using a softmax over the app vocabulary. The output layer reuses the input app embedding table, and a log-popularity correction <a href="https://research.google/pubs/sampling-bias-corrected-neural-modeling-for-large-corpus-item-recommendations/">[22]</a> is subtracted from the logits so that the model doesn’t learn to predict the top grossing apps for every device. With tens of thousands of apps, a sampled softmax keeps training costs manageable.</p><pre>logit(app_j | state_t) = h_t · e_j + b_j − α × log(q_j)</pre><p>In this equation, h is the state at position t, e is the app’s embedding, b is a learned bias, and q is the app’s share of installs. With this approach, every install in every history becomes a training label. Compared with stage 3, where the encoder received one weak label for each won impression, this provides far more supervision, and none of it depends on which network was credited for the install. This is the main way that the token structure improves model performance. Pretraining a user model on unlabeled behavior and then fine-tuning it for downstream tasks has also been studied in academic work, if you want to go deeper on this idea <a href="https://arxiv.org/abs/2010.01494">[23]</a>.</p><p>Only outcome tokens are used as targets. If bid stream tokens were also targets, the model would spend its capacity learning to predict which apps a device opens next, which is an easier task but a less useful one. Bid stream and profile tokens are used as inputs, and installs and purchases are what the model learns to predict.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*msSfHuBdLBGoZ4XuGKJ2Dw.png" /><figcaption>Stage 4, tokens, transformer and pretraining</figcaption></figure><p>The next-install head is used to train the transformer, and the device state is what the rest of the system uses.</p><h3>Details that make it work</h3><ul><li><strong>Size.</strong> Two to four pre-norm transformer blocks, with a model width of 64 to 128 and a context of a few hundred tokens, is sufficient. This is a small model compared to language models, because the task does not require a large model. The training data, on the other hand, can be enormous, since pretraining runs over every event in every device history. This is where GPUs or TPUs pay off, and the model can be scaled up as more data sources are added.</li><li><strong>Time.</strong> Keep the age bands from stage 3 and add a bucket for the gap since the previous event, so that the model sees both recency and cadence. Adding a learned position embedding is also inexpensive.</li><li><strong>Warm start.</strong> Initialize the app embeddings from app co-install statistics, such as a PPMI matrix factored with SVD. This helps training converge faster and gives rare apps a reasonable starting point.</li><li><strong>Observation gaps.</strong> You’ll only see MMP events for apps whose data you receive, and each advertiser’s feed starts on a different date. Only count an install as a target if the app’s feed was live at the time, and add a flag for apps that are newly observed. Otherwise, the model learns when feeds started rather than what devices want.</li><li><strong>Time cutoffs.</strong> Pretraining needs to stop at each training window’s cutoff. If the encoder sees events after the cutoff, they leak into the device state that the downstream heads are trained on.</li><li><strong>Identity.</strong> Sequences can only be joined across sources when the IDs match. The advertising ID works across apps on Android and on iOS devices that have given consent, but IDFV does not. You should expect much longer sequences on Android and evaluate the two platforms separately.</li><li><strong>Data rights.</strong> Every token comes from someone’s data. MMP events fall under what advertisers agree to share and pool, and exchange terms often limit how bid request data can be used. Deciding which sources to combine is a policy decision as well as a modeling decision.</li></ul><p>The output of this stage is a device state, h, which is the transformer’s state at the last position. It summarizes everything the device did across every source available to the DSP, and it was learned without using any attributed labels. The next step is to connect this representation to impressions.</p><h3>Co-training the click and install heads</h3><p>The pretrained transformer learns a lot about devices, but nothing about ads. In the second training stage, we connect the transformer to impressions. This is a multi-task model trained on attributed impressions, where the transformer is the device tower and there is one head for each term in the install equation.</p><h3>Three towers</h3><p>The model has three towers, one for each group of inputs, and each tower outputs a 64-dimensional vector.</p><ul><li><strong>Device tower, h.</strong> The pretrained transformer from stage 4, plus a few bid-time device fields, such as how many times the device has seen the campaign today.</li><li><strong>Ad tower, a.</strong> The campaign’s app, using the same app embedding table as the transformer, plus the creative, campaign, and ad format.</li><li><strong>Context tower, c.</strong> The publisher app, also using the shared app table, plus placement, OS version, country, connection type, and hour of day.</li></ul><p>Sharing the app table across all three towers is a small detail that has a large impact. The app that a device installed last week, the app that it’s currently using, and the app being advertised are all represented in the same space, which enables the model to relate them directly.</p><h3>Interactions built for serving</h3><p>The heads use the three vectors in two ways. A small MLP reads the concatenated vectors and their element-wise products, which enables it to learn any interactions that it needs. In addition, each head has three explicit dot product terms, one for each pair of towers. This is the same idea as factorization machines from stage 1, applied to towers.</p><pre>logit_k = MLP_k([h, a, c, h ⊙ a, h ⊙ c]) + w_k1 × (h · a) + w_k2 × (h · c) + w_k3 × (a · c)</pre><p>The dot products are included to support serving. Because they’re part of every head’s logit, training encourages the towers to encode useful signal in them. In the bid path, computing a dot product between two vectors of 64 numbers takes nanoseconds, while running the MLP for every candidate would be much more expensive. The dot products are the bridge between this model and the model that is used for serving, which we’ll cover in the next two sections.</p><h3>Three heads, trained over every impression</h3><p>Each head predicts a conditional probability. However, the losses are not applied to the conditional probabilities directly, because the install heads would then only be trained on the narrow slice of traffic where they apply. Instead, we follow the entire space multi-task model (ESMM) approach <a href="https://arxiv.org/abs/1804.07931">[24]</a>, where every loss is defined over all impressions.</p><ul><li><strong>Click head</strong> predicts p_click. Its loss is applied to p_click, and the label is whether the impression was clicked.</li><li><strong>Install after click head</strong> predicts p_install given a click. Its loss is applied to p_click × p_install given click, and the label is whether the impression was clicked and then led to an install.</li><li><strong>Install after view head</strong> predicts p_install given a view. Its loss is applied to (1 − p_click) × p_install given view, and the label is whether the device installed without clicking, inside the view window.</li></ul><p>With this setup, the install after click head only receives gradient through impressions where the click head predicts that a click is likely, and the model never needs to train on the biased set of clicked impressions alone. The view head works the same way on the impressions without a click.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ASaQzaSohhTs1Lwar1_SsA.png" /><figcaption>Co-trained model, three towers and three heads</figcaption></figure><p>The app embedding table is shared by the device, ad, and context towers, so every app is represented in the same space.</p><h3>Training details</h3><ul><li><strong>Start from the pretrained tower.</strong> Freeze the transformer for the first epoch while the heads and the other towers converge, and then unfreeze it with a learning rate that is around ten times lower. Keep the next-install loss running on a sample of non-attributed data as an auxiliary task, so that fine-tuning on the smaller attributed data set doesn’t erase what was learned during pretraining.</li><li><strong>Downsample negatives.</strong> Most impressions do not result in a click and almost none result in an install. Keep every positive and a fraction of the negatives, and weight the negatives by one over the sampling rate so that the model’s outputs stay on the correct scale.</li><li><strong>Wait for labels to mature.</strong> Installs arrive hours or days after the impression. Only train on impressions that are old enough for the install window to have closed, or use a delayed feedback model <a href="https://dl.acm.org/doi/10.1145/2623330.2623634">[25]</a> that accounts for conversions that have not yet arrived.</li><li><strong>Keep some exploration.</strong> Attributed data only covers auctions won with the current model. Using a small share of randomized or deliberately varied bids keeps the training data from narrowing around the model’s own decisions.</li><li><strong>Treat SKAN separately.</strong> On iOS without consent, installs are reported as aggregated SKAdNetwork postbacks rather than impression-level labels. These need a separate model, so I’ve left them out of this post.</li></ul><h3>Heads you can add later</h3><p>This post focuses on clicks and installs, but the same model can support additional heads at low cost, because the heads share the towers. For example, purchase after install, first week spend, day seven retention, and predicted SKAN conversion value all fit the same pattern, where each head is defined over every impression by multiplying through the funnel. Each new head provides additional supervision for the shared towers, which often helps the original heads as well.</p><h3>Serving the foundation model efficiently</h3><p>Moloco’s CARA post <a href="https://www.moloco.com/blog/inside-molocos-cara-the-ai-that-compounds">[4]</a> includes a line that any DSP engineer will recognize, which is that its largest models “learn the richest representations, but they’re too slow and too expensive to run inside an auction.” A DSP may receive hundreds of thousands to millions of bid requests a second, score each request against dozens or hundreds of eligible campaigns, and respond within a few milliseconds. Running a transformer over a few hundred tokens for each candidate is not going to fit within this budget.</p><p>The common solutions are distillation, where a small student model is trained to mimic the large model, or selecting a less expensive model based on a price and performance trade-off. This architecture takes a different approach, which makes it possible to use a state of the art model without paying for it in the auction. It splits the work based on what each component depends on, so that the expensive parts never run in the bid path. Here’s where each component is computed and stored.</p><ul><li><strong>Device state h, from the transformer.</strong> Depends on the device’s history. Computed in an offline batch daily, plus a refresh for devices with new events, and stored in a key-value store keyed by advertising ID, with a model version.</li><li><strong>Ad vectors a.</strong> Depend on the campaign, app, and creative. Computed offline whenever a campaign or creative changes, and kept in bidder memory.</li><li><strong>Publisher part of the context vector c.</strong> Depends on the publisher app and placement. Computed offline daily and kept in bidder memory.</li><li><strong>Rest of the context.</strong> OS version, country, connection, and hour, read once per request as plain features.</li><li><strong>Dot products h · a, h · c, and a · c.</strong> Computed per candidate in the bid path from two stored vectors each.</li><li><strong>Final probabilities.</strong> Computed per candidate by LightGBM from the dot products and plain features.</li></ul><p>The transformer runs as a batch job, once per device per day, on GPUs or a large CPU cluster, where cost depends on throughput rather than latency. The ad and publisher vectors are small tables, with thousands to tens of thousands of rows, that fit in the bidder’s memory. In the bid path, each request requires one key-value lookup, which the bidder is already doing for frequency capping and other device features. For each candidate, the work is three dot products of 64 numbers and one pass through a gradient boosted tree model. None of the steps in the bid path require a neural network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KT07SuyLLwIADpLuw99IeA.png" /><figcaption>Serving, offline batch and bid path</figcaption></figure><p>Only two arrows connect the offline and bid path components, a device lookup and the vector tables loaded in memory.</p><p>Here are a few practical details that determine whether this works in production.</p><ul><li><strong>Versioning.</strong> A dot product between vectors from two different training runs is meaningless. Store the model version with each device record, keep the previous ad and publisher tables loaded during a rollover, and use the table that matches each record.</li><li><strong>Devices with no record.</strong> A new device, or a device without a usable ID, has no device state. To handle this case, precompute a fallback state for each profile segment, such as country by OS by device tier, by running the transformer on the profile token alone. Flag these states as fallbacks so that the final model can treat them differently.</li><li><strong>Freshness.</strong> A device that installed an app an hour ago is not reflected in yesterday’s state. A daily batch job is sufficient to get started, and a streaming job can later recompute the state for devices with new events, since running the transformer for a single device is inexpensive. Pinterest’s TransAct <a href="https://arxiv.org/abs/2306.00248">[26]</a> takes a related approach, combining a precomputed user embedding with a small real-time transformer over recent actions.</li><li><strong>IDFV-only iOS traffic.</strong> Without a cross-app ID, there’s no history to join, so these requests use the fallback state and rely on context features. Report performance for these requests separately, so that the gains on Android don’t hide the gap.</li></ul><p>The main benefit of this design is that the large model and the bid path are decoupled. You can make the transformer deeper, extend its context, or add new data sources, and the additional cost is incurred in the batch job rather than in the auction.</p><h3>Calibration with LightGBM at the end</h3><p>The final model in the stack is a LightGBM <a href="https://papers.nips.cc/paper/6907-lightgbm-a-highly-efficient-gradient-boosting-decision-tree">[27]</a> model trained on recent attributed impressions. It uses the dot products from the foundation model along with standard bid request features, and it outputs the click and install probabilities that the bidder uses. The model is small, fast, and retrained frequently, and its role is to get the level of the probability right, not only the ranking.</p><h3>Why not bid on the neural network’s output directly</h3><p>The co-trained heads rank impressions well, but there are several reasons why their probabilities can drift from the true rates.</p><ul><li><strong>Overconfidence.</strong> Deep networks trained with log loss are often overconfident <a href="https://arxiv.org/abs/1706.04599">[28]</a>, especially large networks trained for many epochs.</li><li><strong>Products of heads.</strong> The install probability is a product of two or three heads, and small errors in each head compound.</li><li><strong>Staleness.</strong> The foundation model is expensive to retrain, so it lags behind the market. New campaigns, new creatives, holiday seasons, and supply shifts change install rates faster than a weekly retrain can keep up with.</li><li><strong>Sampling.</strong> Negative downsampling and the weights used to correct for it are easy to get slightly wrong, and a small error is enough to shift every bid.</li></ul><p>A calibration layer addresses these issues in one place, and a gradient boosted tree model is a good fit for this role. It can train on a few days of data in minutes, which means that it can be retrained every few hours. It handles features that neural networks have trouble with, such as bid floor, frequency, and hour of day. It serves predictions on a CPU in microseconds per candidate. And its feature importances and split points are easy to inspect when an account manager asks why a campaign’s bids changed.</p><h3>What goes into it</h3><p>Here are the feature groups that I would use for the calibration model.</p><ul><li><strong>Foundation model scores.</strong> h · a, h · c, and a · c, the learned affinity between device and ad, device and placement, and ad and placement.</li><li><strong>Device scalars, stored with h.</strong> Fallback flag, history length, days since last install, and model version, which let the trees trust the scores less when the history is thin.</li><li><strong>Ad and campaign.</strong> Campaign, app category, creative format, smoothed recent install rate, and campaign age, the fast-moving levels that the foundation model lags on.</li><li><strong>Context.</strong> Placement, ad format, OS version, country, connection type, and hour of day, plain features that trees handle well.</li><li><strong>Request.</strong> Bid floor and how often this device has seen the campaign today, signals that only exist at bid time.</li></ul><p>The 64-dimensional vectors themselves are not used as features. Trees split on one feature at a time, so 64 dense dimensions provide many weak features that are not meaningful on their own. The dimensions also change meaning each time the foundation model is retrained, which would require the trees to be retrained at the same time. A dot product is a single, stable number that already encodes how well two things match, and it stays meaningful across versions as long as both vectors come from the same training run.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Elq4asAGDokRb5O459kdFg.png" /><figcaption>Calibration with LightGBM at the end</figcaption></figure><p>The foundation model contributes three numbers for each candidate, and the trees set the level of the probability.</p><h3>How it is trained</h3><p>We train two LightGBM models on the most recent few days of attributed impressions, one for clicks and one for installs from either path, both using log loss. If negatives are downsampled for this model as well, the output needs to be corrected back to the true rate, where w is the share of negatives that were kept.</p><pre>p = p_s / (p_s + (1 − p_s) / w)</pre><p>Next, we check calibration directly rather than relying only on AUC. This means plotting predicted versus observed rates in bins, computing expected calibration error, and comparing predicted to actual installs by publisher, country, OS, campaign age, and the fallback flag. Trees are usually well calibrated on the data they were trained on, but if a segment drifts, fitting a lightweight isotonic regression on top, per path or per segment, is inexpensive. If you need the click and view paths separately, for example to monitor view-through fraud, you can train a third model for view-through installs on impressions without a click and combine the models using the equation from earlier in the post.</p><h3>Three cadences</h3><ul><li><strong>Transformer pretraining</strong> trains on non-attributed events from every source and is retrained weekly to monthly. It serves device states, recomputed daily in batch.</li><li><strong>Co-trained towers and heads</strong> train on attributed impressions and are retrained weekly. They serve the ad and publisher vectors and the dot products.</li><li><strong>LightGBM</strong> trains on the last few days of attributed impressions and is retrained every few hours. It serves the click and install probabilities in the bid path.</li></ul><p>The slow model learns what devices and apps are, while the fast model learns what the current market is paying for them. The bid is then based on the calibrated install probability, such as the probability multiplied by the advertiser’s target cost per install, and then shaded for the auction.</p><h3>What to measure</h3><p>It’s easy to overstate the impact of a design like this, so the evaluation should separate out what each stage contributes. The most useful experiment is a sequence of model variants, all trained on the same time windows and evaluated on the same holdout window, with LightGBM as the final layer in every case so that calibration is held constant. Here are the variants that I would compare.</p><ol><li><strong>Plain features only.</strong> LightGBM on bid request features with no device history. This is the baseline that every DSP already has.</li><li><strong>Pooled device embeddings.</strong> Stage 2 many-hots of installed apps, pooled. This shows what a static profile adds.</li><li><strong>One attention layer.</strong> The stage 3 sequence encoder, trained on attributed labels only. This shows what order and recency add.</li><li><strong>One attention layer plus next-install pretraining.</strong> The same encoder, pretrained on non-attributed events. This shows what the training task adds, separate from depth.</li><li><strong>Transformer with pretraining.</strong> Stage 4 with MMP tokens only. This shows what depth adds.</li><li><strong>Transformer with all sources.</strong> Stage 4 with bid stream tokens added. This shows what the additional sources add.</li></ol><p>My expectation is that most of the improvement will show up between variants 3 and 4 from the pretraining task, and between variants 5 and 6 from the additional data sources, with a smaller improvement from depth alone between variants 4 and 5. Mobile histories are longer and change faster than the histories where I’ve seen this tested, so depth may have a larger impact here, and this is what the experiment is designed to measure.</p><h3>Offline</h3><ul><li><strong>Always evaluate out of time.</strong> Train on earlier windows, select the model on a later validation window, and report results on a test window after that. Hold out a hash of devices from every training window, so that test devices are never seen during pretraining.</li><li><strong>Measure ranking and calibration.</strong> Report log loss and AUC for clicks and installs, and measure calibration using expected calibration error and the ratio of predicted to actual conversions.</li><li><strong>Break results out by segment.</strong> Report results by OS, by whether the device has a cross-app ID, by history length, by campaign age, and by app category, because averages hide where the improvements are.</li><li><strong>Check for leakage.</strong> Assert that every token used for a training example comes before that example’s cutoff, and watch for any variant that improves much more than expected. In my experience, an improvement that looks too good is usually caused by a leak.</li></ul><h3>Online</h3><p>Improvements in offline log loss don’t always translate into lower cost installs, because the auction reacts to changes in your bids. To validate the model, run the final candidate as an A/B test on a share of traffic and measure cost per install, installs per thousand impressions, win rate, and spend at a fixed budget. It’s also important to monitor the daily ratio of predicted to actual installs. If this ratio drifts, then the bids are wrong even if the ranking is accurate.</p><h3>Where this is heading</h3><p>The four stages in this post are less about model architecture and more about what the model is able to learn from. Logistic regression learns from the bid request. Embeddings enable related values to share information. Sequences bring in the device’s history. And the transformer, through its token format and pretraining task, brings in everything that a device does across every available data source, without needing to wait for an attributed label. This is why transformers have become the state of the art for mobile DSPs. They scale with data and compute, and they can learn from the data that most DSP models never use.</p><p>This is also why vertically integrated platforms are difficult to compete with. A company that runs mediation, an exchange, and an MMP can generate far more tokens per device than a standalone DSP. The architecture in this post doesn’t close that gap, but it gets the most out of whatever data a DSP does have, and it’s the same structure that a larger company could use to combine all of its data sources.</p><p>Here’s a summary of the proposed architecture.</p><ol><li><strong>Pretrain a small transformer</strong> on tokenized device histories from MMP data, the bid stream, and any other sources that you have rights to use, with next-install prediction as the training task.</li><li><strong>Co-train click and install heads</strong> on attributed impressions, using device, ad, and context towers that share an app embedding table and include explicit dot products between the towers.</li><li><strong>Serve with lookups and dot products</strong>, keeping the transformer in a batch job and out of the auction.</li><li><strong>Calibrate with LightGBM</strong> trained on the most recent few days of attributed data, using the dot products as features and retraining every few hours.</li></ol><p>The result is an architecture that is efficient to serve, learns from data at the scale of the whole market, and produces probabilities that can be used directly for bidding. Next steps that I’d like to explore include adding purchase and retention heads to the co-trained model, testing target attention for the top few candidates in each request, and seeing how much richer bid stream tokens can help devices with short MMP histories. If you’re working on any of these problems, I’d be interested to hear what you’re finding.</p><h3>References</h3><ol><li>Zhai et al. (Meta), <a href="https://arxiv.org/abs/2402.17152">Actions speak louder than words, trillion-parameter sequential transducers for generative recommendations (HSTU)</a>, 2024</li><li>Pancha et al. (Pinterest), <a href="https://arxiv.org/abs/2205.04507">PinnerFormer, sequence modeling for user representation at Pinterest</a>, 2022</li><li>Chen et al. (Alibaba), <a href="https://arxiv.org/abs/1905.06874">Behavior sequence transformer for e-commerce recommendation</a>, 2019</li><li>Moloco, <a href="https://www.moloco.com/blog/inside-molocos-cara-the-ai-that-compounds">Inside Moloco’s CARA, the AI that compounds</a></li><li>AWS and Liftoff, <a href="https://aws.amazon.com/blogs/industries/liftoff-improved-conversion-performance-and-reduced-infrastructure-costs-with-cortex-using-aws-graviton">Liftoff improved conversion performance and reduced infrastructure costs with Cortex using AWS Graviton</a></li><li>Bigabid, <a href="https://www.bigabid.com/how-dnns-predict-high-intent-users-faster/">How DNNs predict high-intent users faster than traditional models</a></li><li>AppLovin, <a href="https://www.applovin.com/en/archived-blog/how-we-drive-value-and-handle-data">Performance advertising, how we drive value and handle data</a></li><li>InMobi, <a href="https://advertising.inmobi.com/company/press/inmobi-advertising-unveils-helix-a-groundbreaking-ai-layer-that-delivers-full-funnel-intelligence-for-the-inmobi-dsp">InMobi Advertising unveils Helix</a></li><li>Mobile Dev Memo, <a href="https://mdm.co/podcast-understanding-unitys-vector-with-felix-the/">Podcast, understanding Unity’s Vector (with Felix Thé)</a></li><li>Vaswani et al., <a href="https://arxiv.org/abs/1706.03762">Attention is all you need</a>, 2017</li><li>Zhang et al., <a href="https://arxiv.org/abs/2311.11351">Scaling law of large sequential recommendation models</a>, 2023</li><li>Google Cloud, <a href="https://cloud.google.com/blog/products/containers-kubernetes/moloco-uses-gke-and-tpus-for-ml-workloads">Moloco uses GKE and TPUs for ML workloads</a></li><li>Appodeal, <a href="https://appodeal.com/blog/ad-techs-ml-reality-check/">Ad tech’s ML reality check</a></li><li>Weber, <a href="https://medium.com/@bgweber/pretrained-deep-learning-intent-signals-for-mobile-dsps-2eb1834c62db">Pretrained deep learning intent signals for mobile DSPs</a></li><li>McMahan et al., <a href="https://research.google/pubs/ad-click-prediction-a-view-from-the-trenches/">Ad click prediction, a view from the trenches</a>, 2013</li><li>Juan et al., <a href="https://www.csie.ntu.edu.tw/~cjlin/papers/ffm.pdf">Field-aware factorization machines for CTR prediction</a>, 2016</li><li>Cheng et al., <a href="https://arxiv.org/abs/1606.07792">Wide and deep learning for recommender systems</a>, 2016</li><li>Guo et al., <a href="https://arxiv.org/abs/1703.04247">DeepFM</a>, 2017</li><li>Wang et al., <a href="https://arxiv.org/abs/2008.13535">DCN V2</a>, 2021</li><li>Zhou et al., <a href="https://arxiv.org/abs/1706.06978">Deep interest network for click-through rate prediction</a>, 2018</li><li>Kang and McAuley, <a href="https://arxiv.org/abs/1808.09781">Self-attentive sequential recommendation (SASRec)</a>, 2018</li><li>Yi et al., <a href="https://research.google/pubs/sampling-bias-corrected-neural-modeling-for-large-corpus-item-recommendations/">Sampling-bias-corrected neural modeling for large corpus item recommendations</a>, 2019</li><li>Wu et al., <a href="https://arxiv.org/abs/2010.01494">PTUM, pre-training user model from unlabeled user behaviors via self-supervision</a>, 2020</li><li>Ma et al., <a href="https://arxiv.org/abs/1804.07931">Entire space multi-task model (ESMM)</a>, 2018</li><li>Chapelle, <a href="https://dl.acm.org/doi/10.1145/2623330.2623634">Modeling delayed feedback in display advertising</a>, 2014</li><li>Xia et al. (Pinterest), <a href="https://arxiv.org/abs/2306.00248">TransAct, transformer-based realtime user action model for recommendation at Pinterest</a>, 2023</li><li>Ke et al., <a href="https://papers.nips.cc/paper/6907-lightgbm-a-highly-efficient-gradient-boosting-decision-tree">LightGBM, a highly efficient gradient boosting decision tree</a>, 2017</li><li>Guo et al., <a href="https://arxiv.org/abs/1706.04599">On calibration of modern neural networks</a>, 2017</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f3816d8b0c41" width="1" height="1" alt=""><hr><p><a href="https://aiadvances.org/transformers-for-mobile-user-acquisition-f3816d8b0c41">Transformers for Mobile User Acquisition</a> was originally published in <a href="https://aiadvances.org">AI Advances</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pretrained Deep Learning Intent Signals for Mobile DSPs]]></title>
            <link>https://bgweber.medium.com/pretrained-deep-learning-intent-signals-for-mobile-dsps-2eb1834c62db?source=rss-a80e1f69e782------2</link>
            <guid isPermaLink="false">https://medium.com/p/2eb1834c62db</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Ben Weber]]></dc:creator>
            <pubDate>Sat, 03 Oct 2026 15:51:21 GMT</pubDate>
            <atom:updated>2026-10-03T15:51:21.755Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1U1I6aDf4ocKOjFo" /><figcaption>Photo by <a href="https://unsplash.com/@robman?utm_source=medium&amp;utm_medium=referral">Rob Hampson</a> on <a href="https://unsplash.com?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><em>Borrowing retrieval and ranking methods from the largest recommendation systems to turn non-attributed data into install and purchase intent for fast, calibrated boosting models</em></p><p>Deep learning doesn’t need to be in the bid path to improve a DSP’s bids. In my experience, the biggest gains came from using deep models offline to learn from data that can’t be credited to any campaign, and then passing their outputs as features to fast, well calibrated classic models. In this post, I’ll walk through why serving deep models at bid time is hard, the deep features that worked well for me in the past, and the design I would build today for capturing install and purchase intent from non-attributed data.</p><h3>Why deep learning in the bid path is hard</h3><p>I spent about a decade building machine learning systems for mobile DSPs, at Chartboost and then Aarki. At Aarki, we moved the bidder from a logistic regression model with over 100 million hand-engineered features to deep models such as DeepFM. These models were trained on more than a billion records a day on GPUs and served at over 5 million requests per second (<a href="https://www.aarki.com/insights/aarkis-bidding-brain-evolution-part-1-deep-learning-for-mobile-growth/">part 1</a>, <a href="https://www.aarki.com/insights/aarkis-bidding-brain-evolution-part-2-scaling-with-gpus-at-billion-row-scale/">part 2</a>). We got there, but it was harder than it needed to be, for three reasons.</p><ul><li><strong>Latency.</strong> A bidder has a few milliseconds to respond to each request and handles millions of requests per second. I’ve tried both of the main approaches for serving, scoring ONNX models in-process inside the bidder and calling out to model servers such as TensorFlow Serving (<a href="https://bgweber.medium.com/options-for-ml-serving-with-golang-3b1afa3a6182">Options for ML Serving with Golang</a>, <a href="https://medium.com/data-science-collective/serving-deep-learning-in-adtech-0ff9467ed946">Serving Deep Learning in AdTech</a>). In-process scoring keeps latency low, but it limits model size and ties model releases to bidder releases. Endpoints add a network hop to every bid.</li><li><strong>Calibration.</strong> A bid is only as good as the probability behind it. Deep models trained on sparse, heavily downsampled conversion data are difficult to calibrate, and a miscalibrated model will overpay or underbid at scale.</li><li><strong>Tabular models are good at this task.</strong> Propensity prediction on tabular, high-cardinality ad data is what gradient boosted trees and logistic regression do well. These models are fast, handle missing values and mixed feature types natively, and their calibration is well understood.</li></ul><p>There are places where deep learning is a natural fit for the serving path. For example, <a href="https://blog.bytebytego.com/p/how-snapchat-serves-a-billion-predictions">Snapchat’s ad serving</a> uses learned embeddings and approximate nearest neighbor search to narrow millions of ads down to hundreds of candidates before running more expensive ranking models, at over a billion predictions per second. The embeddings are computed ahead of time, which means that the work at request time is mostly lookups and dot products. This pattern of precomputing the deep part and keeping the online part cheap is the approach this post builds on.</p><h3>Deep features for classic models</h3><p>Some of the strongest signals I’ve used did not require deep learning in the bid path.</p><ul><li><strong>App embeddings from GloVe.</strong> We trained GloVe embeddings on advertiser and publisher apps, and then grouped the apps into clusters using k-means with several values of k. Our logistic regression model used the cluster IDs as categorical features, which meant that the embeddings were pretrained offline and trivial to serve.</li><li><strong>Store description embeddings.</strong> We used a BERT model to create embeddings from the advertiser and publisher app descriptions on the App Store and Google Play. This gave every app a useful representation, including brand new apps with no history.</li></ul><p>In both cases, deep models trained on non-attributed data created stronger features for a classic model that remained fast and well calibrated.</p><p>Looking back, I think going all in on deep learning for serving was a misstep. We would have been better off taking a broader look at how our data sets could produce stronger features, and testing new signals without needing to rebuild the bidder first. At Chartboost, the team did early work on device embeddings, but we didn’t yet have the tooling to build the large scale training and scoring pipelines needed for models such as lightweight transformers for install prediction. Getting hands on with deep models was part of how we built that expertise. If I were starting over, I would first build an offline platform for embedding and transformer models, using a tool such as Vertex AI Pipelines (now part of Google’s <a href="https://en.wikipedia.org/wiki/Gemini_Enterprise_Agent_Platform">Gemini Enterprise Agent Platform</a>), before making deep learning the core of the bid path. There was a lot of pressure at the time to keep up with competitors that were getting coverage for their AI, but taking the time to build the right foundations is a viable path.</p><h3>The approach in this post</h3><p>The rest of this post describes the design I would build today. Mobile ad networks describe their machine learning systems at a high level. For example, <a href="https://www.moloco.com/blog/inside-molocos-cara-the-ai-that-compounds">Moloco</a> has shared that it uses sequence models and multi-head networks, but not how they are trained, what data is used, or how they are calibrated. However, companies such as Pinterest, Spotify, Netflix, Airbnb, YouTube and Twitter have published a decade of detailed work on retrieval and ranking at large scale, and most of it applies directly to user acquisition.</p><p>The main idea is to <strong>use large models pretrained offline on non-attributed data to produce a few strong signals of install and purchase intent, and to use these signals as features in fast, well calibrated boosting models trained on attributed data.</strong> The deep models never set a bid directly. Their job is to produce signals, and the boosting models make the decision.</p><p>Intent is a strong signal to learn from, and non-attributed data is full of it. App level features such as GloVe clusters and description embeddings already showed that deep features work well within classic models. Embeddings and transformer signals learned from the install and purchase histories of individual devices should be even stronger.</p><h3>Two kinds of data, two kinds of model</h3><p>A DSP has two very different data sets, and each one should train the type of model it suits best.</p><ul><li><strong>Attributed data</strong> is the DSP’s own won impressions, joined to the install and purchase postbacks credited to them. This is the only data that tells you what happens after you show an ad, so it needs to be used for training the bidding models. It’s also scarce. Install rates are often around one in ten thousand impressions, and purchases are even rarer.</li><li><strong>Non-attributed data</strong> is everything else that advertisers share through their MMP, such as installs and in-app purchases from organic users and other ad networks. It doesn’t tell you anything about your ads, but it tells you a lot about devices and apps, such as which devices install which types of apps, and which devices make purchases and how much they spend. It’s also orders of magnitude larger.</li></ul><p>A common mistake is to treat non-attributed data as noise because it can’t be credited to a campaign. A better approach is to learn from it before bidding, by pretraining models offline and then passing a handful of their outputs to the bidding models as features. Deep models are good at learning representations from large, sparse and sequential data, while boosting models such as LightGBM and CatBoost are fast to score and easier to calibrate. This design doesn’t try to solve deep model calibration. <strong>Deep models produce signals, and boosting models produce bids.</strong></p><p>I’ve also seen this pattern work outside of ad tech, with household data and nonprofit giving. A two-tower model that matches households to nonprofits produces an affinity score for prospecting, and a transformer trained on each household’s giving history powers propensity and major gift models. Both feed boosting models as features. The data is different, but the structure, the leakage issues and the questions about pooled data are nearly the same.</p><h3>The architecture at a glance</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sp4h5y_xCh0mDWc9gKAGIw.png" /></figure><p>Each pretrained model passes only its outputs to its own boosting model. Attributed data is used to train both boosting models, and their two predictions are multiplied to determine the bid.</p><h3>Install intent with a two-tower affinity model</h3><p>User acquisition is a question of breadth. Out of hundreds of millions of devices, which ones are a good fit for this app? The device usually has no history with the advertiser’s app, so the model needs to match on taste rather than past behavior. A two-tower model works well for this problem.</p><ul><li><strong>Device tower.</strong> Encodes the set of apps installed on a device, along with context such as OS, device model and geo. The set is built from dated MMP and SDK events as of a cutoff time.</li><li><strong>App tower.</strong> Encodes an app ID along with its metadata, such as category, genre and the store description. The metadata is what enables cold start. A new app gets a useful embedding on day one, before it has a single install, because the tower is trained to rebuild an app from its metadata alone by randomly dropping the ID during training.</li><li><strong>Training.</strong> The model is trained with a contrastive loss using in-batch negatives, where for each device and app install pair, the other apps in the same batch are used as negatives. This makes training efficient at large scale. There’s no separate negative sampling step, the app tower runs once per batch, and a single matrix multiply scores every device against every app in the batch, which means that a batch of 8,192 pairs produces millions of negative comparisons almost for free. This is what makes it possible to train on billions of install pairs with a small number of GPUs. Install distributions are extremely skewed, so popularity correction is essential. The logQ correction from <a href="https://research.google/pubs/sampling-bias-corrected-neural-modeling-for-large-corpus-item-recommendations/">YouTube’s two-tower work</a> is the standard way to prevent the model from learning that everyone installs the top 20 apps.</li></ul><p>The output passed to the boosting model is not the embedding itself, but a few <strong>similarity scores</strong>, each answering a different question.</p><ul><li><strong>Long-term affinity.</strong> The cosine similarity between the device embedding and the advertiser app.</li><li><strong>Recent affinity.</strong> The similarity between the advertiser app and the device’s last few installs, computed from app vectors only.</li><li><strong>Negative affinity.</strong> The similarity to apps the device uninstalled quickly.</li><li><strong>Placement affinity.</strong> The cosine similarity between the placement and the advertiser app, which works without a device ID.</li><li><strong>Affinity percentile.</strong> The long-term cosine similarity ranked within the advertiser app’s own distribution.</li></ul><p>The percentile matters more than it might seem. Raw cosine values are not comparable across apps, and a value of 0.4 might be the top 1% for one app and the median for another. A single boosting model shared across thousands of campaigns needs a signal that means the same thing everywhere.</p><p>The placement signal comes from adding placements as a third entity in the embedding space, learned from the DSP’s own bid logs. This is the main way to address the iOS gap, where many requests don’t include an IDFA and the device level cosine is missing. <a href="https://arxiv.org/pdf/2202.05387">Twitter’s TwHIN</a> showed the value of embedding several entity types, such as users, content and advertisers, in one space, and reported a meaningful reduction in cost per conversion for ads.</p><p>The closest published example of this approach is <a href="https://dl.acm.org/doi/10.1145/3219819.3219885">Airbnb’s search ranking</a>. Their ranker was a GBDT model, and listing embeddings were used only as cosine similarity features, such as similarity to listings that a user had recently clicked or skipped. Several of these features ranked among the most important of roughly 100 features. This is the same pattern used here, where embeddings stay upstream and the trees see scalar values.</p><h3>Why scores rather than raw embeddings</h3><p>It’s also possible to use the embeddings directly as features, but this creates a few challenges for the classic models that need to consume them.</p><ul><li><strong>Model support.</strong> Many classic models don’t accept a vector as a single input. A random forest or logistic regression model treats a 64 dimension embedding as 64 separate columns, and a dimensionality reduction step such as PCA might be needed to make the input manageable. That reduction becomes one more model to train, version and keep in sync.</li><li><strong>Stricter versioning.</strong> A retrain of the affinity model can rotate or reshape the embedding space, so dimension 17 in one version may mean something completely different in the next. The cosine similarity between a device and an app changes much less from one retrain to the next, because it captures the relationship between the two vectors rather than their coordinates. If you use the raw embedding, the boosting models need to be retrained in lockstep with every new embedding version.</li><li><strong>Feature importance.</strong> Adding 64 columns from a single embedding can overwhelm the other features in the model. The trees may spread their splits across many weak embedding dimensions, which makes the model harder to interpret and can crowd out the request and campaign features that matter most for bidding.</li></ul><p>A small number of similarity scores captures most of the signal from the embedding, works with any classic model, and keeps the coupling between the pretrained models and the boosting models loose.</p><h3>Purchase intent with a sequence transformer</h3><p>Value prediction is a question of depth. Will this device make a purchase, how much will it spend, and how soon? Order and timing matter more here than they do for taste. A device whose purchases are growing is different from one with the same total spread over two years, and a device that stopped paying six months ago is different again. This calls for a sequence model.</p><ul><li><strong>Input.</strong> Each device’s events in time order, including installs, purchases with spend buckets, uninstalls, and the gaps between events. Static context such as OS, device model and geo is included alongside the sequence.</li><li><strong>Encoder.</strong> A causal transformer that summarizes the sequence into a single <strong>device state</strong>, which is the hidden vector at the last position.</li><li><strong>Objective.</strong> Predict purchases and spend over a <strong>future window</strong>, such as the next 7 or 30 days, rather than only the next event. <a href="https://arxiv.org/pdf/2205.04507">Pinterest’s PinnerFormer</a> found that a future window objective made daily batch embeddings nearly as good as real-time embeddings, which is a good fit for a model that is refreshed offline. It also matches the horizon of D7 and D30 LTV labels.</li><li><strong>App-agnostic heads.</strong> The probability of a purchase in any app within the window, and spend bucket probabilities such as P(spend ≥ $10) and P(spend ≥ $100).</li><li><strong>App-conditional head.</strong> P(purchase | advertiser app), computed by scoring the device state against an app embedding. This head is trained the same way as the affinity model, with in-batch negatives and logQ correction, so it scales to the full device event history just as efficiently.</li></ul><p>Spending behavior carries across apps, which is why non-attributed purchase data is so valuable. A device that spends heavily in three strategy games is a much better prospect for a fourth game than its install history alone would suggest. The app-agnostic heads will usually provide most of the signal, while the app-conditional head adds genre fit.</p><p>The large platforms have converged on this pattern of one pretrained user model feeding many downstream systems. <a href="https://arxiv.org/pdf/2205.04507">PinnerFormer</a> embeddings are used as features in Pinterest’s ranking models, including ads, with reported CTR gains of 7 to 10%. <a href="https://research.atspotify.com/2025/9/generalized-user-representations-for-large-scale-recommendations">Spotify</a> built general user representations that are used for retrieval, ranking and search. <a href="https://netflixtechblog.com/foundation-model-for-personalized-recommendation-1a0bd8e02d39">Netflix</a> trains one foundation model on member interaction sequences and builds other systems on top of it. Closer to ad tech, <a href="https://pith.science/paper/2607.28019">Teads</a> pretrained a transformer on open web user event sequences and used the resulting user embedding in production CTR and win rate models. In mobile ad tech, <a href="https://www.moloco.com/blog/inside-molocos-cara-the-ai-that-compounds">Moloco</a> describes using transformer based sequence models to build dense user representations from cross-app event histories, in order to deal with sparse signals and fragmented identity.</p><p>One simplification compared to those systems is that the bidding models receive the <strong>head outputs</strong> rather than the device state. A probability or a spend bucket has a stable meaning across retrains, while dimension 17 of an embedding does not.</p><h3>Two boosting models on attributed data</h3><p>The bid is the product of two predictions.</p><p><strong>Bid value = P(install | impression) × E[value | install]</strong></p><p>Each factor is predicted by a LightGBM or CatBoost model that is trained only on attributed data.</p><ul><li><strong>Install model.</strong> Trained on won impressions and labeled by install postbacks. The features include the bid request, such as placement, publisher, OS, device model, geo and time, the advertiser app and campaign, serving counters such as daily impression frequency, and the affinity scores.</li><li><strong>LTV model.</strong> Trained on attributed installs and labeled by purchases in the D7 or D30 window after the install. The features include the same request and advertiser features, plus the transformer’s head outputs. A Tweedie objective handles revenue that is mostly zero with a long tail, or you can predict spend buckets directly.</li></ul><p>This is a hurdle model, but the hurdle is two GBDT models with pretrained features, rather than a stack of models plus a combiner and a separate calibration step. Designs like this often add a combiner model that relies on the affinity prior when attributed data is sparse, but a GBDT can learn this on its own. For a new app with no attributed history, the ID based features are empty while the affinity scores still carry signal, so the trees learn when to rely on them.</p><h3>Calibration</h3><p>Bidding the true value is what makes a DSP profitable, so calibration is essential. GBDT models trained with log loss are reasonably well calibrated on their training distribution, and the remaining practices are well documented, most notably in <a href="https://quinonero.net/Publications/predicting-clicks-facebook.pdf">Facebook’s paper on predicting ad clicks</a>.</p><ul><li><strong>Correct for negative downsampling.</strong> If non-installs are kept at rate <em>w</em>, the corrected probability is <em>p / (p + (1 − p) / w)</em>.</li><li><strong>Retrain often.</strong> Freshness matters a lot in ad prediction. Retrain the boosting models daily, while the pretrained models change slowly and can be refreshed weekly.</li><li><strong>Measure normalized entropy, not only AUC.</strong> Log loss relative to a model that predicts the base rate captures calibration, which AUC ignores. You should also report AUC within each campaign, because overall AUC is dominated by differences between placements and campaigns.</li><li><strong>Monitor the product.</strong> Errors in P(install) and E[value] are multiplied, so check the calibration of the bid value by campaign and by bid range, and not just each model on its own.</li></ul><p>The deep models never need to be calibrated, because they never produce a probability that sets a price. Their outputs are features, and the trees learn how much weight to give each one.</p><p>Some DSPs go further and make the decision layer itself a deep model, using multi-head networks and deep cross networks. This approach can capture more feature interactions, but it moves the calibration problem onto the deep model, which is what this design avoids. In my own experiments on click prediction, a deep model slightly outperformed logistic regression on AUC but was worse calibrated. For teams without the scale to invest heavily in calibrating deep rankers, boosting models are a safer choice for the decision layer.</p><h3>Serving with offline scoring and fast lookups</h3><p>The expensive models never run in the bid path. Both pretrained models are scored offline in batch, and what reaches the bidder is a set of vectors keyed by app and by device. At bid time, the bidder performs a few key-value lookups, a dot product, one small ONNX model, and two tree ensembles. This fits within a single digit millisecond budget at a million or more requests per second.</p><p>This serving pattern is proven at DSP scale. At Aarki, models trained in TensorFlow and PyTorch were exported to ONNX and scored in-process on CPUs in a Rust bidder using the ort library, across four data centers handling more than 5 million bid requests per second. In-process scoring avoids a network hop to a model server, which matters more in a bid path than raw model speed.</p><p>The offline half is where GPUs are worth the cost. At Aarki, moving training to NVIDIA L4 GPUs reduced batch processing time from 200 ms to 20 ms, which made daily training on more than a billion rows practical. The pretrained models in this design have a similar profile, as large models that run offline and are refreshed on a schedule.</p><p>Here’s where each component lives.</p><ul><li><strong>App and placement embeddings</strong> are produced weekly by the affinity model and held in memory in every bidder, since they are small enough to load in full.</li><li><strong>Device embeddings</strong> are produced weekly by the device tower and stored in a NoSQL key-value store keyed by device ID, such as Bigtable.</li><li><strong>Device states</strong> are the last position output of the transformer encoder, produced daily or weekly and stored in the same NoSQL record.</li><li><strong>The transformer head</strong> is exported to ONNX when the model is registered, loaded in every bidder, and scored with ONNX Runtime.</li><li><strong>The install and LTV models</strong> are retrained daily on attributed data and loaded in every bidder.</li></ul><p>Here are the steps in the bid path for a single request.</p><ol><li><strong>Fetch the device record</strong> from the NoSQL store in one read, which includes the affinity embedding and the transformer state. The app and placement vectors are already in memory.</li><li><strong>Compute the affinity scores</strong>, including the cosine similarities and the per-app percentile from a lookup table built along with the embeddings.</li><li><strong>Score the transformer head</strong> with ONNX Runtime using the device state and the advertiser app. The head is a couple of small layers, so this takes microseconds rather than milliseconds.</li><li><strong>Score the install and LTV models</strong> using the request features plus these scores.</li><li><strong>Bid</strong> on P(install) × E[value | install], after applying any pacing or margin logic.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7PLRq_axJV2TLTnqH0izbw.png" /></figure><p>Both towers are scored offline. App and placement vectors are small enough to fit in every bidder’s memory, and device vectors are stored in the NoSQL database. At bid time, the affinity signals require only a few dot products and a percentile lookup.</p><p>Splitting the transformer into an encoder that runs offline and a head that runs online is what makes app-conditional scores affordable. The encoder is the expensive part, and its output doesn’t depend on the advertiser, so it runs once per device. The head depends on the app, so it runs at bid time. It’s important to verify the split when exporting the model, by checking that the head applied to the encoder output matches the full model on a sample of records, both in the original framework and in ONNX Runtime, before the model version can be used.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0WLMzrv1_FOvUMaq7QIjkg.png" /></figure><p>The device state is computed once per device in batch and stored by device ID. At bid time, one read fetches the state, and the ONNX head combines it with the advertiser app’s embedding to produce the purchase and spend signals used by the LTV model.</p><p>There are two additional details that make this approach work in production.</p><ul><li><strong>Missing is a value.</strong> Devices without an ID, or without a record in the store, get missing values for the device signals. GBDT models handle missing values natively, and the placement affinity still provides a contextual signal.</li><li><strong>Every signal carries its version.</strong> Store a model version ID with each device record, and don’t score when the device record, the in-memory app vectors and the ONNX head come from different versions. <a href="https://research.atspotify.com/2025/9/generalized-user-representations-for-large-scale-recommendations">Spotify</a> describes the same problem and uses batch IDs to keep downstream models aligned across retrains. This prevents a silent failure, where scores from mismatched versions look normal but are wrong.</li></ul><h3>Keep MMP data separate from bid request data</h3><p>The MMP and SDK feeds should be treated as their own data asset, rather than as extra columns on bid requests. The two sources describe the same devices using different vocabularies. For example, MMP data might use two-letter country codes while bid requests use three-letter codes, list device models by marketing name while bid requests use hardware identifiers, or report OS versions at a different level of detail. These fields also drift independently over time, as exchanges, SDKs and MMPs change what they send.</p><p>If the pretrained models are trained on joined features from both sources, every one of these mismatches becomes a feature parity issue, where the value a model saw in training is not the value it receives at bid time. A cleaner approach is to split the data as follows.</p><ul><li><strong>Train the embedding models only on MMP and SDK data</strong>, using that data’s own vocabulary and the dated event logs.</li><li><strong>Precompute the outputs offline</strong> and write them to the NoSQL store, keyed by the device’s advertising ID, which is the IDFA on iOS and the GAID on Android.</li><li><strong>Join on the ID only at bid time.</strong> The bidder looks up the precomputed embedding and device state using the request’s advertising ID, so no MMP attribute ever needs to match a bid request attribute.</li><li><strong>Keep bid request attributes on the boosting side</strong>, where the install and LTV models learn them directly from attributed data in the RTB vocabulary.</li></ul><p>This also means that a legacy device profile that stores only set membership, with no timestamps, doesn’t need to be redesigned. The ordering and as-of logic happen offline using the raw logs, and the new design adds precomputed fields such as the embedding, device state and version ID next to what the profile already contains. However, you should avoid building boosting model training features from that legacy profile, because a set with no dates can’t be rewound to what was true at the time of a past impression.</p><h3>Avoiding leakage, version drift and data rights issues</h3><p>The architecture is simple to draw, but most of the work is in three areas where it can fail without obvious errors.</p><h3>Target leakage</h3><p>The non-attributed feed contains the attributed events. A device acquired by the DSP still has the app installed, and its purchases show up in the MMP’s event stream like any other device. If these events are used to train the pretrained models, and the same installs and purchases then appear as labels for the boosting models, the signal already encodes the label. The offline AUC will look excellent, but production results won’t match.</p><ul><li><strong>Cut off by time.</strong> Device embeddings and states used for a training row should be built only from events before that row’s impression, and the training window for the pretrained models should end before the label window for the boosting models starts.</li><li><strong>Score historical data with the version that would have been live.</strong> Backfilling training rows with this week’s embeddings leaks the future into the past.</li><li><strong>Mask the target in the inputs.</strong> If the device tower sees the device’s installed apps, the advertiser app needs to be removed from that set when computing the device’s affinity to it.</li></ul><h3>Versions</h3><p>The boosting models learn what a cosine similarity of 0.42 means for one embedding version, and a retrain can shift the entire distribution. Passing scores rather than embeddings keeps this coupling small, and the per-app percentile stays stable across versions even when raw cosine values move. You should still retrain or validate the boosting models whenever a pretrained model changes, and keep version IDs on every signal, as covered in the serving section.</p><h3>Data rights</h3><p>This is a topic that published systems rarely discuss, because a platform like Pinterest owns its users’ activity data. A DSP does not. Non-attributed installs and purchases are the advertisers’ first-party data, shared through their MMP. A pretrained model built from this data is pooled, because it learns from every advertiser and serves every advertiser.</p><ul><li><strong>Make pooled use explicit in the agreement.</strong> Most advertisers are happy to share data that improves performance, but “helps the DSP perform” might be interpreted as “helps my campaigns.” The pooled use should be stated in the contract.</li><li><strong>Handle opt-outs at the data layer, not at the label.</strong> An advertiser that doesn’t allow pooling needs to be removed entirely from the training data for the pretrained models, both as labels and as features. Otherwise, their installs will still shape other devices’ embeddings. Their app can still be scored with an embedding built from metadata, and their own data can still train their own boosting models, since those models serve only them. This should be controlled with a contract flag rather than a rule inferred from the data.</li><li><strong>Follow platform and privacy rules.</strong> On iOS, combining data across companies’ apps for targeting requires ATT consent, so the pooled models can only use devices that have opted in. US state privacy laws and GDPR also apply to cross-advertiser use. Sensitive app categories, such as health, dating and religion, should be excluded, along with anything directed at children.</li></ul><p>Purchase data requires extra care. Advertisers that are comfortable pooling installs may still see their payer lists as their most valuable asset, so it’s a good idea to get permission for installs and purchases separately.</p><h3>Where this is heading</h3><p>The frontier in recommendation systems is moving toward end-to-end sequence models. <a href="https://arxiv.org/abs/2402.17152">Meta’s HSTU</a> treats recommendation as sequence generation and reports large online gains from a model with over a trillion parameters. <a href="https://arxiv.org/abs/2508.14948">Tencent’s LFM4Ads</a> transfers user, item and cross representations from one large model into many downstream ad ranking models. Both point to a future where one large model handles much of what this design splits into four models.</p><p>For a mobile DSP today, I would stop short of that. Attributed labels are scarce, the bid budget is a few milliseconds, and calibration needs to be reliable. This design is intentionally state of practice rather than state of the art, because it uses the frontier’s representation learning without its serving and calibration costs. The deep models learn from the large data set that you didn’t pay for, and the trees make well calibrated decisions on the data that you did.</p><p>The design also doesn’t need to be adopted all at once. It can be rolled out in low risk steps, with each one measured on its own.</p><ol><li><strong>Add the affinity signals</strong>, including the cosine similarity and the per-app percentile, to the existing install GBDT and run an A/B test.</li><li><strong>Add the transformer’s head outputs</strong> to the LTV model and run an A/B test.</li><li><strong>Merge the pretrained models</strong> into one device foundation model with install, purchase and spend heads.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tg-VtR6mv3B3OdNabpOT7Q.png" /></figure><p>In the merged version, one encoder with a shared app table replaces the two pretrained models, and the install head replaces the two-tower cosine similarity. The boosting layer doesn’t change, and still sees only scores. The tradeoff is that install and purchase data rights become tied together, and the model needs devices with timestamped histories.</p><p>At each step, compare the boosting models with and without the pretrained signals. Offline, compare AUC within each campaign and normalized entropy. Online, run an A/B test on cost per install and D7 ROAS, along with an incrementality holdout. The holdout is what shows whether the signals are finding devices that your ads actually influence, rather than devices that would have installed anyway.</p><p>Non-attributed data is the cheapest source of install and purchase intent that a DSP has. <strong>Pretrain on the data you didn’t pay for, and make decisions on the data you did.</strong></p><p>If your team has tried a version of this approach, or chosen a different path, I’d love to hear how it went.</p><h3>References</h3><ul><li><a href="https://www.aarki.com/insights/aarkis-bidding-brain-evolution-part-1-deep-learning-for-mobile-growth/">Aarki’s Bidding Brain Evolution, Part 1</a>. Moving from logistic regression to DeepFM, with ONNX serving in a Rust bidder.</li><li><a href="https://www.aarki.com/insights/aarkis-bidding-brain-evolution-part-2-scaling-with-gpus-at-billion-row-scale/">Aarki’s Bidding Brain Evolution, Part 2</a>. GPU training on more than a billion rows a day.</li><li><a href="https://medium.com/data-science-collective/serving-deep-learning-in-adtech-0ff9467ed946">Serving Deep Learning in AdTech</a>. In-process vs. endpoint serving, and library vs. portable formats.</li><li><a href="https://bgweber.medium.com/options-for-ml-serving-with-golang-3b1afa3a6182">Options for ML Serving with Golang</a>. ONNX Runtime, model endpoints and other options for Go bidders.</li><li><a href="https://blog.bytebytego.com/p/how-snapchat-serves-a-billion-predictions">How Snapchat Serves a Billion Predictions</a> (ByteByteGo). Embedding retrieval before expensive ranking, within a 100 ms budget.</li><li><a href="https://dl.acm.org/doi/10.1145/3219819.3219885">Real-time Personalization using Embeddings for Search Ranking at Airbnb</a> (KDD 2018). Embedding similarities as GBDT features.</li><li><a href="https://research.google/pubs/sampling-bias-corrected-neural-modeling-for-large-corpus-item-recommendations/">Sampling-Bias-Corrected Neural Modeling for Large Corpus Item Recommendations</a> (Google, RecSys 2019). Two-tower retrieval with logQ correction.</li><li><a href="https://arxiv.org/pdf/2202.05387">TwHIN, Embedding the Twitter Heterogeneous Information Network</a> (KDD 2022). Multi-entity embeddings for ads ranking.</li><li><a href="https://arxiv.org/pdf/2205.04507">PinnerFormer, Sequence Modeling for User Representation at Pinterest</a> (KDD 2022). Future window objective, with batch user embeddings used as ranking features.</li><li><a href="https://research.atspotify.com/2025/9/generalized-user-representations-for-large-scale-recommendations">Generalized User Representations for Large-Scale Recommendations</a> (Spotify, RecSys 2025). One user representation for many downstream tasks, versioned across retrains.</li><li><a href="https://netflixtechblog.com/foundation-model-for-personalized-recommendation-1a0bd8e02d39">Foundation Model for Personalized Recommendation</a> (Netflix Tech Blog, 2025).</li><li><a href="https://pith.science/paper/2607.28019">Building a User Foundation Model for the Open Web</a> (Teads, 2026). Transformer user embeddings in production ad rankers.</li><li><a href="https://www.moloco.com/blog/inside-molocos-cara-the-ai-that-compounds">Inside Moloco’s CARA</a> (Moloco). Sequence models and multi-head networks in a mobile DSP, described at a high level.</li><li><a href="https://quinonero.net/Publications/predicting-clicks-facebook.pdf">Practical Lessons from Predicting Clicks on Ads at Facebook</a> (ADKDD 2014). Boosted trees, calibration, downsampling and freshness.</li><li><a href="https://arxiv.org/abs/2402.17152">Actions Speak Louder than Words, Trillion-Parameter Sequential Transducers for Generative Recommendations</a> (Meta, ICML 2024).</li><li><a href="https://arxiv.org/abs/2508.14948">LFM4Ads, Large Foundation Model for Ads Recommendation</a> (Tencent, 2025).</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2eb1834c62db" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Structuring an ML Org for a DSP]]></title>
            <link>https://medium.com/data-science-collective/structuring-an-ml-org-for-a-dsp-9fae5a01c671?source=rss-a80e1f69e782------2</link>
            <guid isPermaLink="false">https://medium.com/p/9fae5a01c671</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Ben Weber]]></dc:creator>
            <pubDate>Tue, 19 May 2026 23:00:19 GMT</pubDate>
            <atom:updated>2026-05-21T09:28:24.948Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KYyOQjiuvaLbAZHJ" /><figcaption>Photo by <a href="https://unsplash.com/@shelbyfigueroa?utm_source=medium&amp;utm_medium=referral">Shelby Murphy Figueroa</a> on <a href="https://unsplash.com?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Pods not Silos for Effective Model Delivery</h4><p>Machine learning is central to how a demand-side platform operates. A DSP runs click, install, purchase, and LTV models in real time, making bid decisions in under 100 milliseconds across billions of daily impressions. The quality of those models directly determines advertiser outcomes and the platform’s ability to compete. Getting the organizational structure right for the team that builds and operates those models matters as much as getting the architecture right.</p><p>This post covers how to structure an ML org for a DSP, why the standard separation between data science and engineering creates predictable problems, and what a better structure looks like in practice. To keep the discussion focused, we will talk about serving models in-process using the ONNX Runtime rather than through a separate model endpoint service, which sidesteps some of the infrastructure ownership questions and lets us focus on the team structure itself.</p><h3>The Standard Approach: Separate Data Science and Engineering Teams</h3><p>The most common org structure at a DSP is a data science team responsible for model training and a bidder engineering team responsible for the serving infrastructure. This division has a reasonable logic behind it. The two teams have different technical requirements and different work rhythms.</p><p>The data science team works in Python. Their job is to explore new model architectures, engineer features, run offline experiments, and produce model artifacts for deployment. The training pipeline they own needs to handle large-scale datasets efficiently, but it does not have real-time latency requirements. The output of their work is a model artifact, typically an ONNX file or a TensorFlow model, which gets handed off to the engineering team for deployment.</p><p>The bidder engineering team works in a systems language, typically Go, Rust, or Java. Their job is to process bid requests in real time, maintain strict latency SLAs, and run model inference either in-process using the ONNX Runtime or via a remote call to a model endpoint. They also calculate the ML features at request time that the model needs to produce a prediction.</p><p>This structure works reasonably well when models are simple and stable. When models are more complex or when they change frequently, three recurring problems emerge.</p><h3>Three Challenges with the Two-Team Handoff</h3><p><strong>Feature parity issues.</strong> The data science team calculates ML features in Python during training. The engineering team recalculates a subset of those features in Go or Rust at serving time. These two implementations will diverge. A lookup table that maps app names to app bundle categories, for example, needs to exist in both pipelines and the values need to stay in sync. When the data science team adds a new feature or changes how an existing feature is calculated, the engineering team needs to implement the same change in the serving pipeline, often weeks later. Small implementation differences between the two pipelines accumulate over time and degrade model performance in ways that are difficult to diagnose. Feature parity issues are one of the most common sources of silent model degradation in production, and they are structurally produced by the two-pipeline approach.</p><p><strong>Model architecture changes are hard to ship.</strong> The handoff model works reasonably well when the output of a model is a single scalar prediction value that the serving pipeline can use directly as a bid adjustment. It breaks down when model architectures become more complex. Consider moving from a propensity model for click prediction to a two-tower architecture for campaign selection. The propensity model produces a scalar; the serving pipeline reads it and adjusts the bid. The two-tower model produces a vector representing the user, and the serving pipeline also needs to compute a cosine similarity between that vector and pre-computed vectors for each campaign under consideration. The engineering team now needs to understand the model architecture well enough to implement the post-processing step correctly. For compositions of models, where the output of one component model feeds into another, the serving implementation becomes more complicated with each architectural iteration. You can define a formal contract for how models should be chained together, but the potential for misimplementation grows with complexity, and the data science team is rarely in a position to catch those errors.</p><p><strong>Model performance ownership is ambiguous.</strong> The data science team trains the models and monitors their performance. The engineering team deploys and serves them. When model performance degrades in production, determining whether the cause is a model quality issue or a serving implementation issue requires coordination between two teams with different priorities and codebases. In practice, this ambiguity tends to resolve slowly. The data science team investigates the model and finds nothing wrong. The engineering team investigates the serving layer and finds nothing obviously broken. The subtle interaction between the two that is causing the problem takes longer to surface. No single team is accountable for the outcome.</p><h3>A Better Structure: Cross-Functional ML Pods with End-to-End Ownership</h3><p>My recommendation for a DSP ML org is to keep separate bidder engineering and machine learning teams, but to make the ML team fully responsible for models from training through serving through monitoring. The ML team owns the prediction microservice: the code that calculates ML features at bid time, runs model inference, and returns prediction values to the bidder.</p><p>This structure is cleanest when the DSP is organized as microservices, with a dedicated prediction service that the ML team owns independently. It can also work in a monolithic codebase, but only if the codebase is approachable for people with ML backgrounds. Rust and Scala are difficult learning curves for most data scientists and ML engineers. Go is a much better fit: it has a small surface area and has ONNX Runtime bindings that make it straightforward to load and run models that were trained in PyTorch or TensorFlow.</p><p>Within the ML team, I recommend organizing into cross-functional pods aligned to model types rather than functional roles. A conversion model pod can be responsible for click, install, and purchase prediction end to end. A pricing model pod can own bid price models. A ranking model pod can own campaign selection. Each pod includes backend engineers, machine learning engineers, and data scientists working together, with shared ownership of the full lifecycle. The titles matter less than the structure: one pod, one set of models, clear accountability for performance.</p><p>Start with a single pod for conversion models and a single prediction microservice. Get that working end to end before adding more pods or expanding scope. The organizational and technical complexity of adding a second pod is much lower once you have proven the model with the first.</p><h3>Logging Features at Inference Time</h3><p>One of the most important practices this structure enables is logging ML features at bid time rather than recalculating them in the training pipeline.</p><p>In many ML systems, impression data and device profiles are logged at bid time and then a subset of features are recalculated offline during training. This gives the data science team flexibility to experiment with new feature definitions without changing the serving pipeline, but it is the root cause of feature parity issues. The feature definition exists in two places, and the two definitions will drift.</p><p>When the ML team owns the serving pipeline, the responsibility for where features get defined shifts. Features are defined first in the serving pipeline, at bid time, and the training pipeline reads those logged values directly rather than recalculating them. The training data reflects exactly what the model saw at serving time. Feature parity issues become structurally much harder to introduce.</p><p>To keep training datasets compact when logging full feature sets at bid time, protocol buffers are a good choice for serialization. Protobuf’s binary encoding is compact and efficient to deserialize, and the schema definition serves as a useful contract between the serving pipeline and the training pipeline.</p><p>The tradeoff is that adding a new feature now requires a change to the prediction microservice first, and if that service is written in Go, the ML team needs to be comfortable working in Go to ship new features. This is a higher bar than working entirely in Python, but it is the right bar. The team that defines the features should be the same team that trains on them.</p><h3>The Role of Analytics and Data Science</h3><p>With this structure, there is still a valuable role for a separate data science or analytics function. That team focuses on building standardized experiment readouts, running exploratory data analysis to understand changes in model performance over time, and providing measurement infrastructure for A/B tests. Their job is to help the ML pods understand what is happening and why, not to own model architecture or tuning decisions. Those decisions stay with the pod that owns the models.</p><p>This separation is important. A centralized analytics team that also has opinions about model architecture creates ambiguity about who is accountable for model performance. Keeping analytics focused on measurement and the ML pods focused on models gives each function a clear mandate.</p><h3>Conclusion</h3><p>The right ML org structure for a mobile DSP is a team that owns model training, serving, and performance end to end. The cleanest way to implement this is a cross-functional ML pod that owns a prediction microservice, with features logged at inference time and training pipelines that read those logged features directly.</p><p>This structure requires ML engineers and data scientists to work in Go or another systems language in addition to Python. That is a real ask. But the alternative, two teams with a handoff between them, produces feature parity problems, slow architecture iteration, and ambiguous ownership of model performance. Those are expensive problems to live with at DSP scale, where model quality directly determines revenue.</p><p>Starting with a single pod, a single prediction service, and a clear commitment to end-to-end ownership is the fastest path to a system that can ship new models quickly and trust that they are performing as intended.</p><p><em>This post is adapted from an upcoming book on deploying machine learning models to production at scale, covering real-time and batch inference patterns using PyTorch, ONNX, PySpark, and Go.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9fae5a01c671" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/structuring-an-ml-org-for-a-dsp-9fae5a01c671">Structuring an ML Org for a DSP</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[OnPrem vs Public Cloud for AdTech ML]]></title>
            <link>https://medium.com/data-science-collective/on-prem-vs-public-cloud-for-ml-3fb4568bd589?source=rss-a80e1f69e782------2</link>
            <guid isPermaLink="false">https://medium.com/p/3fb4568bd589</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Ben Weber]]></dc:creator>
            <pubDate>Fri, 15 May 2026 23:47:31 GMT</pubDate>
            <atom:updated>2026-05-16T17:52:24.890Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iH8y8O1w7vGrv8nj" /><figcaption>Photo by <a href="https://unsplash.com/@pericakalimerica?utm_source=medium&amp;utm_medium=referral">Pero Kalimero</a> on <a href="https://unsplash.com?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Tradeoffs in Training and Inference at Ad Tech Scale</h4><p>Mobile advertising runs on prediction. Every time an ad is shown, a conversion model has already estimated the likelihood that the impression will result in a click, an install, or a purchase. This happens in under 100 milliseconds, billions of times per day. The models making those predictions need to be trained frequently, on enormous datasets, and served with latency budgets that leave almost no margin for error.</p><p>The infrastructure that supports this workflow (training pipelines, GPU clusters, model serving, and observability) can be built on public cloud, on-premise, or a combination of both. Each approach involves real tradeoffs across cost, flexibility, development speed, and operational overhead. This post breaks down those tradeoffs for both training and serving, with the goal of helping teams evaluate which deployment makes sense for their situation, particularly for new projects where these decisions are still open.</p><h3>The Scale of Ad Tech Data</h3><p>To understand why infrastructure choices matter so much in this domain, it helps to start with the scale. A mid-size mobile DSP may serve hundreds of millions to several billion impressions per day. Each impression is a labeled training example: the ad was shown, and the downstream outcome (click, install, or purchase) is recorded via postback from an MMP. Over weeks and months, this accumulates into training datasets with billions of rows.</p><p>Conversion prediction models, which estimate the probability of a click or install given an impression, need to be retrained frequently to remain effective. User behavior shifts, seasonal patterns emerge, new apps enter the market, and advertiser campaign mixes change. A model trained three months ago may no longer reflect the current distribution of traffic well enough to make good bidding decisions. Most mobile ad tech companies now train on a daily or weekly cadence, which means the training pipeline runs continuously and reliably matters as much as how fast the first model was shipped.</p><p>The industry has largely moved to deep learning for conversion prediction. GPU training has become standard practice, with DNNs, attention-based architectures, and models with embedding layers replacing the gradient boosted trees that dominated earlier years. At the same time, serving these models under real-time bidding latency constraints (typically well under 100 milliseconds for the full request cycle) means that GPU serving is often impractical, and most teams serve deep learning models on CPU using optimized runtimes. Training on GPU and serving on CPU is now the dominant pattern in the industry.</p><h3>Part 1: Training with Spark, GPUs, and Pipeline Orchestration</h3><p>The standard training pipeline for a deep learning conversion model has two stages: data preparation on a Spark cluster, followed by model training on GPU instances. Spark handles the heavy lifting of reading billions of impression records, joining them with auxiliary data, enriching features, and reshaping data into tensor formats that a PyTorch DataLoader can consume. The GPU stage then trains the model on the prepared data. This separation of concerns works well because Spark is CPU and memory intensive while training is GPU intensive, and the two workloads suit different hardware profiles.</p><p>This pattern is common to both on-premise and public cloud deployments. The tooling for the training step itself is largely open source and runs identically in both environments: PyTorch for model development, MLflow for experiment tracking and model registry, and ONNX for model export. The output of the pipeline in either case is a model artifact, typically an ONNX file, stored in a shared location accessible to the serving layer. Where the environments diverge is in how the pipeline is orchestrated and how the underlying hardware is provisioned.</p><p><strong>Workflow Orchestration</strong></p><p>Connecting a Spark data preparation job to a GPU training job requires some form of orchestration: something that runs Spark, waits for it to complete, and then kicks off training. On-premise, there are several approaches. A lightweight option is a simple Flask application running on the GPU machine: when the Spark job finishes, it calls the Flask endpoint to trigger model fitting. This approach is operationally simple and easy to debug, but it lacks the visibility and retry logic of a proper workflow tool. For teams that want more structure, Airflow and Prefect both run well on-premise and give you DAG-based pipeline management, scheduling, alerting, and a UI for monitoring run history.</p><p>Public cloud offers managed versions of these tools. AWS Managed Workflows for Apache Airflow (MWAA), Google Cloud Composer, and Databricks Workflows are all production-ready managed orchestration options that eliminate the operational overhead of running the scheduler infrastructure yourself. For teams without a dedicated platform engineering function, this can be a meaningful reduction in toil. Databricks Workflows in particular handles the Spark-to-GPU handoff cleanly if your data preparation is already running on Databricks.</p><p><strong>Hardware Provisioning</strong></p><p>This is one of the sharpest differences between the two environments. On public cloud, you can launch a GPU instance in minutes (A100s, L4s, H100s) and terminate it when training completes. If you need a different GPU type for a new architecture, you provision it immediately. If demand spikes, you scale out.</p><p>On-premise, provisioning new hardware means physically ordering, installing, and configuring machines. If your GPU fleet is sized for your current daily training workload, there is no spare capacity available for experimentation or for running multiple architectures in parallel. Teams often overprovision to create headroom, which adds cost. On the other hand, if your training workloads are stable and predictable, with a consistent daily pipeline and a known GPU budget, the economics of buying versus renting GPUs typically favor on-premise after roughly two years of operation. For a large-scale ad tech company running GPU training continuously, owned hardware can represent significant savings over a multi-year horizon.</p><p>For teams that need distributed training across multiple GPUs or multiple nodes, public cloud also makes this operationally simpler. Ray on Vertex AI, for example, lets you spin up a Ray cluster with GPU worker nodes in a few lines of Python and run distributed PyTorch training without managing the cluster infrastructure yourself. Doing the same on-premise is possible with Ray or PyTorch’s native distributed training, but it requires more setup and ongoing maintenance.</p><h3>Part 2: Serving ONNX Models in Production</h3><p>The serving side of the equation has its own set of tradeoffs, and some of them are less obvious than the training side.</p><p><strong>Instance Flexibility</strong></p><p>The same on-demand provisioning advantage that public cloud offers for training applies to serving. If request volume grows faster than expected, you scale out serving instances on public cloud with a few clicks or an autoscaling policy. On-premise, expanding serving capacity requires new hardware, and that process is measured in weeks or months, not minutes. For serving infrastructure that needs to handle the highly variable traffic patterns of real-time bidding, where volume can spike significantly during peak hours or around major events, this flexibility has real operational value.</p><p>There is also a subtler issue with on-premise serving that is easy to underestimate: library and OS compatibility. The ONNX Runtime and its language-specific bindings are actively developed, and recent versions often require recent operating system versions. The Rust ONNX Runtime library (ORT), for example, requires Ubuntu 22. On-premise machines may have been provisioned years ago and configured for Ubuntu 20, and upgrading them is a non-trivial operation, particularly on machines running production traffic. Public cloud instances can be launched with any OS version from the start, and upgrading is a matter of launching new instances rather than migrating running ones.</p><p><strong>Model Serving Architecture</strong></p><p>Once you have resolved the infrastructure questions, the actual mechanics of serving ONNX models are more similar across environments than the surrounding differences might suggest.</p><p>If you are serving models in-process, loading the ONNX Runtime directly into your serving application written in Go, Python, or Rust, there is not much practical difference between on-premise and cloud. The model artifact is loaded into memory at startup, inference runs on each incoming request, and the serving layer is a relatively straightforward application regardless of where it is deployed. This in-process pattern is common in latency-sensitive ad tech serving stacks, where the overhead of a network call to a separate model endpoint is difficult to justify within a tight latency budget.</p><p>If you prefer an endpoint-based approach, using a dedicated model server that receives inference requests over a network, the environments differ more. Public cloud provides managed offerings like Vertex AI Model Endpoints, which handle deployment, scaling, and health monitoring for TensorFlow and PyTorch models with relatively little configuration. On-premise, NVIDIA Triton Inference Server is the standard open-source option for this pattern. Triton supports ONNX natively, handles batching, and provides a well-defined gRPC and HTTP interface for inference requests. It is a capable solution, but it requires more setup and operational investment than a managed endpoint.</p><p><strong>Observability</strong></p><p>Observability is another area where managed cloud offerings provide value out of the box that on-premise teams typically have to build themselves. Vertex AI Model Endpoints emit system metrics automatically, including request latency, requests per second, and error rates, and they surface in Cloud Monitoring without any additional instrumentation. Getting the same visibility on-premise requires standing up Prometheus for metrics collection, Grafana for dashboards, and building the instrumentation into your serving application to export the metrics you care about, such as average model prediction, feature value distributions, and per-segment latency breakdowns. This is achievable and the open-source ecosystem is mature, but it is engineering work that does not directly advance the model.</p><h3>A Hybrid Approach</h3><p>The most practical architecture for many ad tech ML teams is not a pure choice between on-premise and cloud, but a hybrid that uses each environment where it fits best.</p><p>Public cloud is well suited for model training. The ability to launch GPU instances on demand, use managed orchestration tools, and scale distributed training jobs without managing hardware makes it a strong fit for a workflow that is bursty by nature, running once a day or once a week and then sitting idle. The cost premium over owned hardware is real, but for teams without consistent high-volume training workloads, or for teams that want to experiment with different GPU configurations without capital investment, the flexibility justifies the cost.</p><p>On-premise is well suited for model serving. Real-time bidding serving stacks run 24 hours a day at consistent high throughput. The latency requirements are strict, and co-locating serving infrastructure with exchange connectivity reduces the network hops in the critical path. For this workload, the economics strongly favor owned hardware over rented instances, and the operational model is well understood.</p><p>ONNX is what makes the hybrid approach practical. Because both the training pipeline and the serving layer agree on the ONNX format as the exchange artifact, the training environment and the serving environment are decoupled. A model trained on a cloud GPU cluster and exported to ONNX can be deployed to an on-premise Go or Rust serving application without any changes to the serving code. The variety of ONNX Runtime bindings available in Python, Go, Rust, C++, and Java means you can serve the same model artifact across a heterogeneous serving fleet. Open source tools like NVIDIA Triton extend this further for teams that want an endpoint-based architecture on-premise.</p><p>The choice between on-premise and cloud is rarely made once and then revisited easily. Switching environments is a lengthy and costly process, and the operational muscle memory a team builds around a particular stack has real value that is difficult to account for in an infrastructure comparison. For new projects where the decision is genuinely open, the framework above provides a starting point, but the right answer depends heavily on existing infrastructure, team size, traffic volume, and how much the organization values infrastructure flexibility versus cost optimization.</p><p><em>This post is adapted from a chapter in an upcoming book on deploying machine learning models to production at scale, covering real-time and batch inference patterns using PyTorch, ONNX, PySpark, and Go.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3fb4568bd589" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/on-prem-vs-public-cloud-for-ml-3fb4568bd589">OnPrem vs Public Cloud for AdTech ML</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using DNN Embedding Models in PySpark at AdTech Scale]]></title>
            <link>https://medium.com/data-science-collective/using-dnn-embedding-models-in-pyspark-at-adtech-scale-8f8fdd00fc17?source=rss-a80e1f69e782------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f8fdd00fc17</guid>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Ben Weber]]></dc:creator>
            <pubDate>Tue, 12 May 2026 01:09:24 GMT</pubDate>
            <atom:updated>2026-05-12T01:09:24.691Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wZJayYK5cFfFTTkj" /><figcaption>Photo by <a href="https://unsplash.com/@fabioha?utm_source=medium&amp;utm_medium=referral">fabio</a> on <a href="https://unsplash.com?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Inferencing on billions of records with PyTorch and ONNX</h4><p>In a real-time bidding system, every millisecond counts. A mobile DSP has somewhere Around 120 milliseconds to evaluate a bid request, retrieve features, run model inference, and return a bid price. Deep learning models that use attention layers and transformer components are powerful, but they are expensive and running a transformer over a sequence of behavioral events for every single bid request burns through that latency budget fast.</p><p>One solution is to split the model in two. The computationally expensive part in the the transformer layers that process device behavior sequences can runs offline on a daily batch pipeline, and its output is precomputed as a fixed-size embedding vector stored in a NoSQL database like Aerospike. At inference time, the serving layer retrieves the embedding with a key-value lookup in under a millisecond and passes it to a lightweight conversion model that runs the remaining inference in around one millisecond. The transformer’s rich behavioral representation is available at serving time without paying its latency cost on every request.</p><p>This post walks through the full implementation of this pattern for an install prediction model at a mobile DSP. We want to predict the likelihood that showing an impression will result in an install callback from a Mobile Measurement Partner (MMP). Our inputs include sequences of postback events from the DSP — actions like prior installs, purchases, and engagement signals across apps — along with numeric and categorical features available at bid time.</p><p>Here is the architecture we will build:</p><ol><li><strong>Feature encoding:</strong> reshape impression data into tensor inputs using PySpark and write to HDFS as Parquet for PyTorch DataLoader consumption</li><li><strong>Composite model training:</strong> train an embedding model with transformer and DCNv2 layers alongside a multi-task conversion model in PyTorch</li><li><strong>ONNX export: </strong>export the embedding model for batch scoring</li><li><strong>Distributed embedding generation: </strong>run the ONNX model at scale using PySpark’s mapInPandas with a serializable broadcast wrapper</li><li><strong>Install model training:</strong> train a focused install propensity model on the precomputed embeddings and export it for online serving</li></ol><h3>Why Precompute Embeddings</h3><p>The device-level features that we feed the transformer include sequences of app activity, install history, and purchase events that change slowly. A user’s behavioral profile from yesterday is still a good representation of their intent today. Recomputing that transformer inference from scratch on every bid request is wasteful: we would be running a 40-millisecond computation to produce a result that barely changes between requests.</p><p>By running the transformer daily in a batch pipeline and storing its output vector in Aerospike, the serving layer gets the full representational power of a deep behavioral model at the cost of a single NoSQL lookup. The conversion model that consumes the embedding is a lightweight DCN network that runs in about one millisecond and handles the context-specific features that do change between requests, like the specific ad being shown and the publisher app.</p><p>The daily batch pipeline scores the full population of active devices using PySpark distributed across a cluster. This is where ONNX becomes essential. PySpark workers are JVM processes, and distributing a PyTorch model across them is complicated because PyTorch is not designed for this environment, and serializing a model across executor nodes is fragile. The ONNX Runtime, by contrast, is lightweight, has a clean serialization workaround we will cover below, and runs efficiently on CPU without any framework overhead.</p><h3>Step 1: Feature Encoding with PySpark</h3><p>We can gather raw impression data as event logs. We need to reshape it into tensor inputs that a PyTorch DataLoader can consume: a numeric tensor, a categorical tensor, and one tensor per sequence feature. The output is Parquet files written to HDFS, with label columns for click, install, and purchase.</p><p>Sequence features capture ordered behavioral history. The most important is active_model_apps — the apps a device has previously installed or engaged with, ordered from most recent to least recent. We want more recent apps used to have a higher weight.</p><pre>from pyspark.sql import SparkSession<br>from pyspark.sql import functions as F<br>from pyspark.sql.types import ArrayType, FloatType, IntegerType<br>import numpy as np<br><br>spark = SparkSession.builder.appName(&quot;feature-encoding&quot;).getOrCreate()<br><br># Load raw impression logs<br>df = spark.read.parquet(&quot;/data/impressions/raw/&quot;)<br># --- Numeric features ---<br># Normalize to float32; fill nulls with zero<br>numeric_cols = [<br>    &quot;bid_floor&quot;, &quot;hour_of_day&quot;, &quot;day_of_week&quot;,<br>    &quot;device_age_days&quot;, &quot;prior_impressions_7d&quot;, &quot;prior_clicks_7d&quot;,<br>]<br>for c in numeric_cols:<br>    df = df.withColumn(c, F.col(c).cast(FloatType()).fillna(0.0))<br># Pack into a single array column<br>df = df.withColumn(<br>    &quot;numeric_tensor&quot;,<br>    F.array([F.col(c) for c in numeric_cols])<br>)<br># --- Categorical features ---<br># Integer-encoded categories (vocabulary mapping applied upstream)<br>categorical_cols = [<br>    &quot;country_id&quot;, &quot;os_id&quot;, &quot;device_type_id&quot;,<br>    &quot;publisher_app_id&quot;, &quot;ad_format_id&quot;,<br>]<br>for c in categorical_cols:<br>    df = df.withColumn(c, F.col(c).cast(IntegerType()).fillna(0))<br>df = df.withColumn(<br>    &quot;categorical_tensor&quot;,<br>    F.array([F.col(c) for c in categorical_cols])<br>)<br># --- Sequence features ---<br># active_model_apps: list of app IDs ordered most recent to least recent<br># Pad or truncate to fixed sequence length<br>SEQ_LEN = 20<br>@F.udf(ArrayType(IntegerType()))<br>def pad_sequence(seq):<br>    if seq is None:<br>        return [0] * SEQ_LEN<br>    truncated = seq[:SEQ_LEN]<br>    return truncated + [0] * (SEQ_LEN - len(truncated))<br>df = df.withColumn(<br>    &quot;app_sequence_tensor&quot;,<br>    pad_sequence(F.col(&quot;active_model_apps&quot;))<br>)<br># --- Labels ---<br>df = df.withColumn(&quot;has_click&quot;,    F.col(&quot;click_event&quot;).cast(FloatType()).fillna(0.0))<br>df = df.withColumn(&quot;has_install&quot;,  F.col(&quot;install_event&quot;).cast(FloatType()).fillna(0.0))<br>df = df.withColumn(&quot;has_purchase&quot;, F.col(&quot;purchase_event&quot;).cast(FloatType()).fillna(0.0))<br># --- Output ---<br>output_cols = [<br>    &quot;device_id&quot;,  # retain for embedding store key<br>    &quot;numeric_tensor&quot;, &quot;categorical_tensor&quot;, &quot;app_sequence_tensor&quot;,<br>    &quot;has_click&quot;, &quot;has_install&quot;, &quot;has_purchase&quot;,<br>]<br>df.select(output_cols) \<br>  .write \<br>  .mode(&quot;overwrite&quot;) \<br>  .parquet(&quot;/data/impressions/encoded/&quot;)</pre><p>The encoded Parquet files are now ready for PyTorch. Each row contains three tensor columns: numeric, categorical, and sequence, plus the three label columns. The device_id is retained so we can key the embedding store by device when writing results back to Aerospike.</p><h3>Step 2: Training the Composite Model in PyTorch</h3><p>Our architecture is a composition of three PyTorch modules. The first is an embedding model that processes the sequence input through transformer encoder layers and a DCNv2 cross-network, producing a 100-dimensional device embedding. The second is a conversion model that takes the embedding alongside the numeric and categorical inputs and predicts all three labels. The third is a wrapper that composes both into a single trainable unit.</p><p>We use a lightweight transformer with two encoder layers and four attention heads rather than a full-scale architecture. The goal is expressive sequence modeling, not language understanding. The DCNv2 cross-network on top of the pooled transformer output efficiently captures interactions between sequence-derived features.</p><pre>import torch<br>import torch.nn as nn<br>import math<br><br># --- Sequence Embedding Model ---<br>class PositionalEncoding(nn.Module):<br>    def __init__(self, d_model, max_len=20):<br>        super().__init__()<br>        pe = torch.zeros(max_len, d_model)<br>        position = torch.arange(0, max_len).unsqueeze(1).float()<br>        div_term = torch.exp(<br>            torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model)<br>        )<br>        pe[:, 0::2] = torch.sin(position * div_term)<br>        pe[:, 1::2] = torch.cos(position * div_term)<br>        self.register_buffer(&quot;pe&quot;, pe.unsqueeze(0))  # (1, max_len, d_model)<br>    def forward(self, x):<br>        return x + self.pe[:, :x.size(1), :]<br><br>class DCNv2Cross(nn.Module):<br>    &quot;&quot;&quot;Single DCNv2 cross layer: x0 * W(x) + x&quot;&quot;&quot;<br>    def __init__(self, input_dim):<br>        super().__init__()<br>        self.weight = nn.Linear(input_dim, input_dim, bias=True)<br>    def forward(self, x0, x):<br>        return x0 * self.weight(x) + x<br><br>class EmbeddingModel(nn.Module):<br>    &quot;&quot;&quot;<br>    Processes device behavior sequences through transformer encoder layers<br>    and a DCNv2 cross-network to produce a 100-dim device embedding.<br>    Input:  app_sequence (batch, seq_len) - integer app IDs<br>    Output: device_embedding (batch, 100)<br>    &quot;&quot;&quot;<br>    def __init__(self, vocab_size=50000, d_model=64, seq_len=20,<br>                 nhead=4, num_layers=2, embedding_dim=100):<br>        super().__init__()<br>        self.app_embedding = nn.Embedding(vocab_size, d_model, padding_idx=0)<br>        self.pos_encoding  = PositionalEncoding(d_model, max_len=seq_len)<br>        encoder_layer = nn.TransformerEncoderLayer(<br>            d_model=d_model, nhead=nhead, dim_feedforward=128,<br>            dropout=0.1, batch_first=True<br>        )<br>        self.transformer = nn.TransformerEncoder(encoder_layer, num_layers=num_layers)<br>        self.cross1 = DCNv2Cross(d_model)<br>        self.cross2 = DCNv2Cross(d_model)<br>        self.projection = nn.Sequential(<br>            nn.Linear(d_model, embedding_dim),<br>            nn.ReLU(),<br>        )<br>    def forward(self, app_sequence):<br>        x  = self.app_embedding(app_sequence)   # (batch, seq_len, d_model)<br>        x  = self.pos_encoding(x)<br>        x  = self.transformer(x)                # (batch, seq_len, d_model)<br>        x  = x.mean(dim=1)                      # mean pool -&gt; (batch, d_model)<br>        x0 = x<br>        x  = self.cross1(x0, x)<br>        x  = self.cross2(x0, x)<br>        return self.projection(x)               # (batch, 100)<br><br># --- Conversion Model ---<br>class ConversionModel(nn.Module):<br>    &quot;&quot;&quot;<br>    Multi-task conversion model.<br>    Inputs: numeric tensor, categorical tensor (embedded), device embedding.<br>    Outputs: click prob, install prob, purchase prob.<br>    &quot;&quot;&quot;<br>    def __init__(self, num_numeric=6, num_categories=5,<br>                 cat_vocab_size=10000, cat_embed_dim=8,<br>                 device_embed_dim=100, hidden_dim=128):<br>        super().__init__()<br>        self.cat_embeddings = nn.ModuleList([<br>            nn.Embedding(cat_vocab_size, cat_embed_dim)<br>            for _ in range(num_categories)<br>        ])<br>        input_dim = num_numeric + (num_categories * cat_embed_dim) + device_embed_dim<br>        self.cross1 = DCNv2Cross(input_dim)<br>        self.cross2 = DCNv2Cross(input_dim)<br>        self.deep = nn.Sequential(<br>            nn.Linear(input_dim, hidden_dim), nn.ReLU(),<br>            nn.Linear(hidden_dim, hidden_dim), nn.ReLU(),<br>        )<br>        combined_dim = input_dim + hidden_dim<br>        self.click_head    = nn.Linear(combined_dim, 1)<br>        self.install_head  = nn.Linear(combined_dim, 1)<br>        self.purchase_head = nn.Linear(combined_dim, 1)<br>    def forward(self, numeric, categorical, device_embedding):<br>        cat_embeds = [<br>            emb(categorical[:, i]) for i, emb in enumerate(self.cat_embeddings)<br>        ]<br>        x  = torch.cat([numeric] + cat_embeds + [device_embedding], dim=1)<br>        x0 = x<br>        x_cross = self.cross1(x0, x)<br>        x_cross = self.cross2(x0, x_cross)<br>        x_deep  = self.deep(x)<br>        x_out   = torch.cat([x_cross, x_deep], dim=1)<br>        return (<br>            torch.sigmoid(self.click_head(x_out)).squeeze(1),<br>            torch.sigmoid(self.install_head(x_out)).squeeze(1),<br>            torch.sigmoid(self.purchase_head(x_out)).squeeze(1),<br>        )<br><br># --- Composite Model ---<br>class CompositeModel(nn.Module):<br>    &quot;&quot;&quot;Wraps EmbeddingModel and ConversionModel into a single trainable unit.&quot;&quot;&quot;<br>    def __init__(self, embedding_model, conversion_model):<br>        super().__init__()<br>        self.embedding_model = embedding_model<br>        self.conversion_model = conversion_model<br>    def forward(self, app_sequence, numeric, categorical):<br>        device_embedding = self.embedding_model(app_sequence)<br>        return self.conversion_model(numeric, categorical, device_embedding)</pre><p>Training the composite model jointly ensures that the embedding learns representations useful for all three conversion types. The loss weights reflect both label frequency and business priority and installs receive the highest weight.</p><pre>def train_composite(model, loader, optimizer, epochs=5):<br>    model.train()<br>    criterion = nn.BCELoss()<br><br>for epoch in range(epochs):<br>        total_loss = 0.0<br>        for app_seq, numeric, categorical, y_click, y_install, y_purchase in loader:<br>            optimizer.zero_grad()<br>            pred_click, pred_install, pred_purchase = model(<br>                app_seq, numeric, categorical<br>            )<br>            loss = (<br>                0.2 * criterion(pred_click,    y_click) +<br>                0.6 * criterion(pred_install,  y_install) +<br>                0.2 * criterion(pred_purchase, y_purchase)<br>            )<br>            loss.backward()<br>            optimizer.step()<br>            total_loss += loss.item()<br>        print(f&quot;Epoch {epoch+1}: loss={total_loss/len(loader):.4f}&quot;)<br><br>embedding_model  = EmbeddingModel()<br>conversion_model = ConversionModel()<br>composite_model  = CompositeModel(embedding_model, conversion_model)<br>optimizer = torch.optim.Adam(composite_model.parameters(), lr=1e-3)<br>train_composite(composite_model, train_loader, optimizer)</pre><p>After training, we evaluate AUC on a holdout test set for each label before proceeding. The install AUC is the primary signal for this example, and in practice we would creative multiple conversion models to supplement the install prediction shown here.</p><h3>Step 3: Exporting the Embedding Model to ONNX</h3><p>We export only the EmbeddingModel component. The conversion model will be retrained on fresh daily embeddings and exported separately in Step 5. Keeping the two exports separate is deliberate because the embedding model changes on a slower cadence (weekly or monthly retraining), while the conversion model can be retrained daily. Either side can be updated independently as long as the embedding dimension stays consistent. We do need to ensure that the same model version used to create the embedding for model training is the embedding version used during serving.</p><pre>import torch.onnx<br><br>embedding_model = composite_model.embedding_model<br>embedding_model.eval()<br>SEQ_LEN = 20<br>dummy_app_sequence = torch.zeros(1, SEQ_LEN, dtype=torch.long)<br>torch.onnx.export(<br>    embedding_model,<br>    dummy_app_sequence,<br>    &quot;embedding_model.onnx&quot;,<br>    input_names=[&quot;app_sequence&quot;],<br>    output_names=[&quot;device_embedding&quot;],<br>    dynamic_axes={<br>        &quot;app_sequence&quot;:     {0: &quot;batch_size&quot;},<br>        &quot;device_embedding&quot;: {0: &quot;batch_size&quot;},<br>    },<br>    opset_version=17,<br>)<br># Verify the export<br>import onnxruntime as ort<br>import numpy as np<br>session    = ort.InferenceSession(&quot;embedding_model.onnx&quot;)<br>test_input = np.zeros((4, SEQ_LEN), dtype=np.int64)<br>result     = session.run([&quot;device_embedding&quot;], {&quot;app_sequence&quot;: test_input})<br>print(result[0].shape)  # (4, 100)</pre><p>We use the dynamic_axes declaration, because without it the exported model&#39;s batch dimension is fixed to 1 and it cannot process partitions in bulk. The opset_version=17 setting ensures broad compatibility across ONNX Runtime versions.</p><h3>Step 4: Distributed Embedding Generation with PySpark</h3><p>This is where the pipeline scales to billions of records. We need to run the embedding model across billions of impressions and write the output vectors to HDFS for downstream training and to NoSQL for serving.</p><p>The core challenge is that onnxruntime.InferenceSession is not directly serializable. Spark must serialize the UDF and everything in its closure in order to distribute work across executor nodes. If we include the session object directly, Spark cannot pickle it and the job fails.</p><p>The solution is to serialize the ONNX model as a byte string, broadcast that string across the cluster using Spark’s broadcast mechanism, and reconstruct the InferenceSession from bytes inside the worker. The byte string is serializable, while the session object is not. We wrap this pattern in a class that initializes the session lazily on first use within each executor.</p><pre>from pyspark.sql import SparkSession<br>from pyspark.sql.types import StructType, StructField, StringType, ArrayType, FloatType<br>import pandas as pd<br>import numpy as np<br>import onnxruntime as ort<br><br>spark = SparkSession.builder.appName(&quot;embedding-generation&quot;).getOrCreate()<br># Read the ONNX model as bytes and broadcast across the cluster.<br># Broadcasting sends the bytes once per executor and caches them there,<br># rather than shipping the model with every task.<br>with open(&quot;embedding_model.onnx&quot;, &quot;rb&quot;) as f:<br>    model_bytes = f.read()<br>broadcast_model = spark.sparkContext.broadcast(model_bytes)<br><br>class OnnxSessionWrapper:<br>    &quot;&quot;&quot;<br>    Lazily initializes an ONNX Runtime session from a serialized model byte string.<br>    InferenceSession is not serializable and cannot be used as a Spark broadcast<br>    variable directly. This wrapper holds the serializable byte string and creates<br>    the session on first use within each executor, avoiding repeated initialization<br>    overhead across records in the same partition.<br>    &quot;&quot;&quot;<br>    def __init__(self, model_bytes):<br>        self._model_bytes = model_bytes<br>        self._session = None<br>    def get_session(self):<br>        if self._session is None:<br>            self._session = ort.InferenceSession(self._model_bytes)<br>        return self._session<br><br>def generate_embeddings_partition(iterator):<br>    &quot;&quot;&quot;<br>    mapInPandas function: receives one partition as a stream of Pandas DataFrames.<br>    Runs ONNX batch inference on the full partition rather than row by row.<br>    &quot;&quot;&quot;<br>    wrapper = OnnxSessionWrapper(broadcast_model.value)<br>    session = wrapper.get_session()<br>    for pdf in iterator:<br>        # Stack sequence arrays into shape (batch, seq_len) as int64<br>        sequences  = np.stack(pdf[&quot;app_sequence_tensor&quot;].values).astype(np.int64)<br>        # Run batch inference - far more efficient than one record at a time<br>        embeddings = session.run(<br>            [&quot;device_embedding&quot;], {&quot;app_sequence&quot;: sequences}<br>        )[0]  # (batch, 100)<br>        yield pd.DataFrame({<br>            &quot;device_id&quot;: pdf[&quot;device_id&quot;].values,<br>            &quot;user_emb&quot;:  embeddings.tolist(),<br>        })<br><br>output_schema = StructType([<br>    StructField(&quot;device_id&quot;, StringType()),<br>    StructField(&quot;user_emb&quot;,  ArrayType(FloatType())),<br>])<br>df = spark.read.parquet(&quot;/data/impressions/encoded/&quot;)<br>df_embeddings = df.mapInPandas(generate_embeddings_partition, schema=output_schema)<br>df_embeddings.write.mode(&quot;overwrite&quot;).parquet(&quot;/data/device_embeddings/&quot;)</pre><p>A note on the choice of mapInPandas over the alternatives. Spark offers three ways to run custom Python logic on partitions. The oldest is rdd.mapPartitions, which operates on the RDD API and processes data as iterators of Row objects. It works, but it bypasses the DataFrame optimizer, requires manual schema handling, and does not support batch-oriented processing natively. A Pandas UDF with the @pandas_udf decorator is better suited to column-level transformations than to multi-input batch inference jobs like this one. mapInPandas is the best tool here: it operates on the DataFrame API, passes full Pandas DataFrames to the function, and lets us send the entire partition through the ONNX Runtime as a single batch. Running batch inference through ONNX is significantly faster than row-by-row inference because the runtime can vectorize operations and amortize session overhead across the batch.</p><p>Once the daily job completes, the device_id and user_emb pairs are written to Aerospike alongside the partition output. The online serving layer retrieves them at bid time with a single key-value lookup that completes in under one millisecond — well within the latency budget that the transformer inference would have consumed entirely.</p><h3>Step 5: Training the Install Model on Precomputed Embeddings</h3><p>With embeddings available for every device in the training data, we now train a focused install propensity model. This model predicts only thehas_install label from the multi-task training in Step 2, and now we are solving a single well-defined problem.</p><p>The install model is a lightweight DCNv2 network that takes the numeric tensor, categorical tensor, and the precomputed user_emb vector as inputs. Because it does not run transformer inference, it is fast enough for the online serving path.</p><pre>class InstallModel(nn.Module):<br>    &quot;&quot;&quot;<br>    Lightweight install propensity model.<br>    Inputs: numeric tensor, categorical tensor, precomputed device embedding.<br>    Output: install probability.<br>    &quot;&quot;&quot;<br>    def __init__(self, num_numeric=6, num_categories=5,<br>                 cat_vocab_size=10000, cat_embed_dim=8,<br>                 device_embed_dim=100, hidden_dim=64):<br>        super().__init__()<br><br>self.cat_embeddings = nn.ModuleList([<br>            nn.Embedding(cat_vocab_size, cat_embed_dim)<br>            for _ in range(num_categories)<br>        ])<br>        input_dim = num_numeric + (num_categories * cat_embed_dim) + device_embed_dim<br>        self.cross1 = DCNv2Cross(input_dim)<br>        self.cross2 = DCNv2Cross(input_dim)<br>        self.deep = nn.Sequential(<br>            nn.Linear(input_dim, hidden_dim), nn.ReLU(),<br>            nn.Linear(hidden_dim, hidden_dim), nn.ReLU(),<br>        )<br>        self.output = nn.Linear(input_dim + hidden_dim, 1)<br>    def forward(self, numeric, categorical, user_emb):<br>        cat_embeds = [<br>            emb(categorical[:, i]) for i, emb in enumerate(self.cat_embeddings)<br>        ]<br>        x  = torch.cat([numeric] + cat_embeds + [user_emb], dim=1)<br>        x0 = x<br>        x_cross = self.cross1(x0, x)<br>        x_cross = self.cross2(x0, x_cross)<br>        x_deep  = self.deep(x)<br>        x_out   = torch.cat([x_cross, x_deep], dim=1)<br>        return torch.sigmoid(self.output(x_out)).squeeze(1)<br><br>def train_install_model(model, loader, optimizer, epochs=10):<br>    model.train()<br>    criterion = nn.BCELoss()<br>    for epoch in range(epochs):<br>        total_loss = 0.0<br>        for numeric, categorical, user_emb, y_install in loader:<br>            optimizer.zero_grad()<br>            pred = model(numeric, categorical, user_emb)<br>            loss = criterion(pred, y_install)<br>            loss.backward()<br>            optimizer.step()<br>            total_loss += loss.item()<br>        print(f&quot;Epoch {epoch+1}: loss={total_loss/len(loader):.4f}&quot;)<br><br># train_loader reads the Parquet embeddings from Step 4<br># joined back to the encoded impression features from Step 1<br>install_model = InstallModel()<br>optimizer     = torch.optim.Adam(install_model.parameters(), lr=1e-3)<br>train_install_model(install_model, train_loader, optimizer)</pre><p>After validating AUC on the holdout set, we export the install model and register it for online serving.</p><pre>install_model.eval()<br><br>dummy_numeric     = torch.zeros(1, 6)<br>dummy_categorical = torch.zeros(1, 5, dtype=torch.long)<br>dummy_user_emb    = torch.zeros(1, 100)<br>torch.onnx.export(<br>    install_model,<br>    (dummy_numeric, dummy_categorical, dummy_user_emb),<br>    &quot;install_model.onnx&quot;,<br>    input_names=[&quot;numeric&quot;, &quot;categorical&quot;, &quot;user_emb&quot;],<br>    output_names=[&quot;install_prob&quot;],<br>    dynamic_axes={<br>        &quot;numeric&quot;:      {0: &quot;batch_size&quot;},<br>        &quot;categorical&quot;:  {0: &quot;batch_size&quot;},<br>        &quot;user_emb&quot;:     {0: &quot;batch_size&quot;},<br>        &quot;install_prob&quot;: {0: &quot;batch_size&quot;},<br>    },<br>    opset_version=17,<br>)</pre><p>In production we publish this ONNX file to a model registry, and MLflow works well for this, storing the artifact alongside the training run metadata, evaluation metrics, and a reference to the embedding model version it was trained against. The serving layer pulls the latest registered model on deployment and loads it with the ONNX Runtime. Because the serving code depends only on the ONNX file and its declared input schema, the model server does not need to change when the model is updated and only the artifact in the registry changes.</p><h3>Summary</h3><p>The system separates model complexity from serving latency. The transformer processes behavioral sequences and produces rich device representations, but it does so offline where its 40-millisecond inference time is not a constraint. The install model runs online where it needs to be fast and we use a small DCNv2 network reading a precomputed embedding vector runs in around one millisecond per record.</p><p>The PySpark pipeline makes this practical at scale. Distributing ONNX inference across a cluster using mapInPandas with a broadcast session wrapper lets us score tens of millions of devices in a daily batch job. Each executor loads the ONNX model once from the broadcast bytes, processes its partition as a single batch, and returns results that Spark assembles back into a DataFrame. The parallelism is handled entirely by Spark&#39;s scheduler with no changes to the model code.</p><p>This pattern stores results for fast retrieval and generalizes beyond install propensity. The same approach applies to LTV scoring, churn prediction, or any problem where a deep behavioral model is too expensive to run in the online path but too valuable to leave out of the feature set.</p><p><em>This post is adapted from a chapter in an upcoming book on deploying machine learning models to production at scale, covering real-time and batch inference patterns using PyTorch, ONNX, PySpark, and Go. It was primarily generated by Claude after training on my previous book and nearly 100 blog posts.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f8fdd00fc17" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/using-dnn-embedding-models-in-pyspark-at-adtech-scale-8f8fdd00fc17">Using DNN Embedding Models in PySpark at AdTech Scale</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Options for ML Serving with Golang]]></title>
            <link>https://bgweber.medium.com/options-for-ml-serving-with-golang-3b1afa3a6182?source=rss-a80e1f69e782------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b1afa3a6182</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Ben Weber]]></dc:creator>
            <pubDate>Thu, 07 May 2026 01:09:30 GMT</pubDate>
            <atom:updated>2026-05-07T01:09:30.402Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KvTkKn6kQ237gt6z" /><figcaption>Photo by <a href="https://unsplash.com/@jeshoots?utm_source=medium&amp;utm_medium=referral">JESHOOTS.COM</a> on <a href="https://unsplash.com?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Deep Learning Inference at Ad-Tech Scale</h4><p>I’m a big fan of using Golang to write microservices and have used it for projects at multiple gaming and ad tech companies. One of the challenges that comes up with using Golang for ad tech is serving machine learning, and in particular deep learning models. In this post I’ll cover a few different options for running deep learning models in Golang and discuss some tradeoffs. I’ll highlight the GoMLX library, which supports training and serving in Golang, provides compatibility with the ONNX industry standard, provides hardware acceleration on CPU via XLA, and supports model fine tuning.</p><p><strong>Model Endpoints<br></strong>Perhaps the easiest way to serve deep learning models with Golang is to not use Go for model inference, and instead have your Golang server call a separate service where your model is hosted. This can be an endpoint set up yourself using FastAPI and Docker, or a managed service like Vertex AI where a cloud provider hosts your model. This approach generally uses Python for serving rather than Golang, but you can still use Go for reshaping your feature data into tensors, which is needed for calls to services such as TensorFlow serving.</p><p>Using endpoints for model serving does introduce some challenges for serving, because it’s another service to manage in addition to your Golang application. Additionally, coordination is needed when deploying a retained model in order to make sure that requests are routing to the correct endpoint when updating a model. For example, you may have a click prediction model that you retrain daily, and each model deployment requires setting up a new endpoint or model route, and the vocabularies used for encoding feature data into tensors may change day-to-day, meaning that the encoding step needs to align with the model endpoint that traffic is being routed to, and there is a period of time where both model endpoints need to be live in order to handle the migration of traffic. When using an in-memory approach for model inference, which we’ll cover in the following sections, you can replace the new model in one step, since there is no routing logic needed to cleanly switch to your new model version.</p><p>The endpoint approach is a common approach for many industries, but at ad tech scale it can introduce challenges due to latency considerations. I recommend this approach when first working with deep learning models in Golang, because you can use existing tooling like NVIDIA Triton or Vertex AI to set up a model as an endpoint, which can reduce the complexity of your Golang application. You can use this approach for ad tech scale, but using managed services like Vertex AI gets expensive. Using endpoints also makes deployments more complicated if you are targeting on-prem deployments, because you need to handle model refreshes and model routing to get good throughput.</p><p><strong>Deep Learning Libraries<br></strong>Another option for serving deep learning models in Golang is to use a library that directly supports serving the models that you train. TensorFlow has the <a href="https://github.com/galeone/tfgo">tfgo</a> library and PyTorch has <a href="https://github.com/orktes/go-torch">go-torch</a> and <a href="https://github.com/sugarme/gotch">gotch</a>, for serving models directly in Go. The main issue with these libraries is that they are no longer actively maintained and building and packaging these libraries for deployments may result in large binaries given all of the dependencies that TensorFlow includes.</p><p>There’s another library I explored for working with deep learning models not in Golang, but Rust, which is Hugging Face’s <a href="https://github.com/huggingface/candle">candle</a> library. This library enables model training and serving directly in Rust, and something similar available for Golang might be the best fit for the ad tech use case.</p><p><strong>Onnx Runtime<br></strong>A good option if you want to serve models in-process in Golang is ONNX, which is a serialization format that supports both TensorFlow and PyTorch models. PyTorch has native support for exporting models into the ONNX format while TensorFlow models require additional libraries such as <a href="https://github.com/onnx/tensorflow-onnx">tf2onnx</a>. Once a model is converted to ONNX, you need a runtime to support running inference on the computation graph stored in ONNX format. The most common implementation of this is Microsoft’s <a href="https://github.com/microsoft/onnxruntime">ONNX Runtime</a> which is written in C++ and has robust support for most of the functionality provided in the different ONNX op sets. The <a href="https://github.com/yalue/onnxruntime_go">onnxruntime_go </a>library provides a wrapper for the ONNX runtime that enables running ONNX models in process in a Golang application.</p><p>ONNX Runtime works great for serving deep learning models, but it does require some tuning to get working well for ad tech scale. I had to tune the <a href="https://onnxruntime.ai/docs/performance/tune-performance/threading.html">threading </a>settings to get good latency measures when using the ONNX runtime, reducing the parallelism enabled by the default configuration. There’s also limited options for enabling hardware acceleration for the ONNX Runtime when running inference on CPUs, which is a common use case in ad tech with on-prem deployments.</p><p><strong>GoMLX<br></strong><a href="https://github.com/gomlx/gomlx">GoMLX</a> is a Golang library focused on enabling hardware acceleration for training and serving deep learning models with Go. It provides an XLA backend for accelerating model serving on CPU and it can load ONNX models into its computational graph format. If you are already using the ONNX Runtime, you should be able to explore GoMLX as an option without needing to make any changes to your training and model publishing pipelines. It can take some time to get the XLA backend configured properly, but this backend can deliver high throughput for model inference when running on CPUs on smaller batch sizes. While GoMLX does support loading and running ONNX models, the support is not as complete as Microsoft’s runtime, and some of the <a href="https://onnxruntime.ai/docs/performance/model-optimizations/graph-optimizations.html">graph optimizations</a> are not currently supported in GoMLX.</p><p>Another key benefit of GoMLX is that it supports online retraining and fine tuning. Once an ONNX model is loaded into the GoMLX computational graph, you can train the model on new examples to update the weights of the models. This is useful for ad tech where you have real-time feedback in the form and win and loss notifications that you can feed into pricing models.</p><p><strong>Conclusion<br></strong>Both ONNX Runtime and GoMLX provide good options for serving deep learning models in-process in Golang. ONNX Runtime provides more robust support for op sets and optimizations, but GoMLX can be a better choice if you are targeting CPU hardware and what to do fine-tuning or your models using real-time data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b1afa3a6182" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Out of the Cloud: Building On-Prem Machine Learning Pipelines]]></title>
            <link>https://medium.com/data-science-collective/out-of-the-cloud-building-on-prem-machine-learning-pipelines-597d5603502f?source=rss-a80e1f69e782------2</link>
            <guid isPermaLink="false">https://medium.com/p/597d5603502f</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Ben Weber]]></dc:creator>
            <pubDate>Sat, 27 Sep 2025 18:17:49 GMT</pubDate>
            <atom:updated>2025-09-28T07:52:34.375Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YSru4-9oPsB1RFkK" /><figcaption>Photo by <a href="https://unsplash.com/@tvick">Taylor Vick</a> on <a href="https://unsplash.com/photos/cable-network-M5tzZtFCOfs">Unsplash</a></figcaption></figure><h4>Building ML Workflows without Managed Services</h4><p>In most of my past data science roles, I’ve worked with data and model training pipelines on public clouds including Amazon Web Services (AWS) and Google Cloud Platform (GCP). These platforms now provide managed services for ML models in the form of SageMaker and Vertex AI that greatly simplify the process of going from model concept to prototype to production. Another managed service offered on these platforms is Databricks, which makes it trivial to set up Python and PySpark workloads for model training. In my current role, I don’t have the luxury of using these managed services for PySpark and model pipelines, because we own and operate our servers in a private data center. I’ve previously blogged about our journey to deep learning models <a href="https://www.aarki.com/insights/aarkis-bidding-brain-evolution-part-1-deep-learning-for-mobile-growth/">here </a>and our investment in GPUs <a href="https://www.aarki.com/insights/aarkis-bidding-brain-evolution-part-2-scaling-with-gpus-at-billion-row-scale/">here</a>. In this post, I’ll focus on the challengse and opportunities that I’ve faced as a data scientist working with on-prem infrastructure.</p><h4>Spark</h4><p>Most of my experience with the Spark ecosystem has been through the Databricks platform, which provides a notebook interface where the Spark session is already initialized and you can jump in and start working with dataframes. I’ve worked with Databricks on AWS and GCP and have used it to build production pipelines for ML models that train dozens of models daily for adtech prediction tasks. Most of these pipelines first unload data from a data warehouse or data lake and then read and write prepared data sets and model artifacts to cloud storage, such as S3 or GCS. Databricks makes it easy to spin up new clusters on-demand, using the elastic compute capabilities of public cloud platforms.</p><p>In my current role, we have a fixed number of machines that we use for our Spark cluster. Instead of spinning up new clusters for each job, we have an over-provisioned cluster that we share for all of our Spark jobs. We also have a cluster set up for HDFS, which we use for cloud storage. We are able to use the cluster interactively with the PySpark kernel for Jupyter and for job runs using spark-submit. We primarily use Spark for dataset preparation, and use separate machines for model training, where we leverage GPUs.</p><h4><strong>Model Training</strong></h4><p>My first models in adtech were large logistic regression models that we trained on Spark using MLlib. When moving to deep learning models for these prediction tasks, I continued to use the Spark ecosystem and Databricks for the model training portion of the pipeline, leveraging Databricks pipelines. However, we weren’t able to get proper hardware acceleration with GPUs using this setup, and needed to set up separate dedicated GPU machines to get a noticeable speedup in model training.</p><p>We have the same challenge with our on-prem setup, where we need to use separate infrastructure for data preparation and model training, using Spark for preparation and GPUs machines for model fittings. Our initial approach to this problem was to set up a lightweight Flask app on the GPU machines, where we exposed endpoints that the data preparation jobs trigger when the data set for a model is ready. This worked okay to start, but we ran into scheduling issues where different model training jobs were trying to grab the same GPU and resources became exhausted. We migrated to Prefect for orchestration to help alleviate this issue, and use interactive workflows to manage the GPU resources.</p><h4>Model Serving</h4><p>In a public cloud it’s easy to spin up a load balanced model serving endpoint using services like SageMaker and Vertex AI. These services also provide auto-scaling capabilities for dealing with changes in prediction volume throughout the day. In the past I used Vertex AI to set up model serving endpoints for models trained with TensorFlow and used the Vertex AI <a href="https://cloud.google.com/vertex-ai/docs/predictions/optimized-tensorflow-runtime">optimized runtime</a> to get good model throughput.</p><p>With on-prem servers, we need to create and manage endpoints for model serving ourselves. We could leverage a public cloud option, but the latency would be too high, and the data transfer and compute cost would be very costly. The first option that we explored was to avoid model endpoints altogether and serve our deep learning models in our existing servers, which are written in Rust. We used the <a href="https://crates.io/crates/ort">ort</a> package which provides Rust bindings to the ONNX Runtime. We load our deep learning models into memory and serve them in process to minimize latency and simplify model deployments. We are also exploring moving to NVIDIA Triton for model serving, to provide more flexibility in configuring our model deployments and leveraging both CPU and GPU execution providers.</p><h4>Observability</h4><p>Many of the tools that we use for observability on-prem are also used with public cloud setups. While Databricks provides a managed offering of MLflow, you can also run this on-prem by running your own instance of MLflow server. We’re also using Evidently AI for tracking feature and model prediction drift.</p><p>Where things differ on-prem is how we manage operational metrics for our models. In the past I used GCP’s StackDriver (now called Cloud Monitoring) for logging metrics such as average and p90 model outputs, such as click rate prediction. I’ve also used Datadog for this functionality, and we were able to set up alerts that are triggered when a threshold is exceeded for a model metric. Our current setup uses OpenTelemetry for Rust, which we use to aggregate model metrics.</p><h4><strong>GPUs</strong></h4><p>When building machine learning pipelines on a cloud provider like AWS you can spin up a variety of GPU instances on-demand for any of your workloads. There may be quota limits and higher-end GPUs may be harder to reserve, but if you need more resources for hardware acceleration and you have the budget, you can get access to cards to help complete your workloads. Cloud providers offer a variety of solutions in this space, and we worked with both AWS and GCP when determining which types of graphics cards to buy for our on-prem workflows. We created sample data sets for the models that we wanted to train, and then transferred these from our data centers to cloud storage while exploring different cloud options. Google’s Colab provided a flexible option for our testing, and we evaluated different NVIDIA cards for our workflows. We found that the L4 cards were a good option for our model throughput and also provided a good energy efficiency for our data centers.</p><p>To move to GPU training for on-prem model training, we had to make an upfront investment in GPUs in order to keep everything local within our data centers, but we were able to do due diligence and find a cost efficient solution for the deep learning models that we train. Our largest models train on over one billion records daily, and we are able to achieve a 10x speedup with GPU versus CPU training for our models.</p><p><strong>Summary</strong><br>It takes more work to set up model training and serving capabilities using on-prem hardware versus using managed services offered by public cloud providers, but you can achieve the same result using open source tools. We were able to achieve a more cost efficient solution for model training and model serving by doing everything in house.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=597d5603502f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/out-of-the-cloud-building-on-prem-machine-learning-pipelines-597d5603502f">Out of the Cloud: Building On-Prem Machine Learning Pipelines</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Serving Deep Learning in AdTech]]></title>
            <link>https://medium.com/data-science-collective/serving-deep-learning-in-adtech-0ff9467ed946?source=rss-a80e1f69e782------2</link>
            <guid isPermaLink="false">https://medium.com/p/0ff9467ed946</guid>
            <category><![CDATA[adtech]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[rust]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Ben Weber]]></dc:creator>
            <pubDate>Sun, 18 May 2025 02:28:24 GMT</pubDate>
            <atom:updated>2025-05-18T02:28:24.298Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pOX-KbaRLkS-oj16" /><figcaption>Photo by <a href="https://unsplash.com/@dulhiier?utm_source=medium&amp;utm_medium=referral">Nastya Dulhiier</a> on <a href="https://unsplash.com?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Scaling model inference to millions of requests per second</h4><p>Machine learning models are a core component of the AdTech ecosystem, where applications and websites show advertisements to generate revenue and advertisers buy ad placements to drive conversions. In the mobile app ecosystem, AdTech drives user acquisition and app monetization, and the key players in this space serve deep learning models to gain a competitive advantage. Mobile Demand Side Platforms (DSPs) for programmatic user acquisition including <a href="https://www.moloco.com/blog/dr-sechan-oh-explains-molocos-machine-learning">Moloco</a>, <a href="https://liftoff.io/blog/cortex-ml-platform-announcement/">Liftoff</a>, <a href="https://cloud.google.com/blog/products/compute/applovin-builds-ai-ad-platform-on-google-cloud-g2-vms">Applovin</a>, and <a href="https://www.aarki.com/pr/aarki-redefines-the-app-hive-economy-with-supervised-ai-and-deep-neural-networks-for-mobile-app-retention/">Aarki</a> have shared details about their deep learning approaches for AdTech.</p><p>Deep learning can be used for a variety of tasks in AdTech including conversion prediction (e.g. click and install prediction for mobile apps), ranking (e.g. campaign and creative selection), and <a href="https://liftoff.io/blog/maximize-spend-with-dynamic-impression-pacing/">pacing</a>. It’s common to use different model architectures for these tasks, such as using <a href="https://arxiv.org/pdf/2008.13535">DCN V2</a> for conversion predictions and <a href="https://cloud.google.com/blog/products/ai-machine-learning/scaling-deep-retrieval-tensorflow-two-towers-architecture">Two Towers</a> for ranking tasks. Regardless of architecture, you need to deal with two key challenges: latency and throughput. AdTech systems need to process more than 1 million requests per second (QPS), and need to respond to requests from ad exchanges within 100 milliseconds. These requirements have major implications for the viability of different approaches for serving deep learning models.</p><p>It’s a common pattern to use Python to train deep learning models and to serve these models in a production system running a different language, such as Go, Rust, or Java. I like to categorize the options for how to achieve this interoperability along two dimensions:</p><ol><li><strong>Model Serialization</strong><br>Are you storing the model in a library specific or portable format?</li><li><strong>Model Serving</strong><br>Are you running the model directly or calling an endpoint?</li></ol><p>The best approach to take for your use case will vary based on a variety of factors including your deployment environment, requirements for hardware acceleration (GPUs), and latency requirements. We’ll first provide an overview of the landscape of tools in this space, and then discuss tradeoffs of the different approaches.</p><p>The first dimension to consider is how you will serialize your deep learning models. If you are using TensorFlow or PyTorch, you can serialize models using the formats that these libraries provide, such as the SavedModel format in TensorFlow. Another option is to use a library agnostic format for serializing your models, such as <a href="https://onnx.ai/">ONNX</a>, which is supported by several programming languages.</p><p>The second dimension to consider is if you want to serve the model directly (embedded) in the production system, or make a call to a separate endpoint responsible for model inference. I consider the latter to be more common, but it doesn’t work for all deployment situations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*hD9bgyBwK7IeuusMsB2PkA.png" /><figcaption>Serving options across two dimensions (<em>Image by Author</em>)</figcaption></figure><p>The figure above shows some of the tools available for serving deep learning models for AdTech plotted along these two considerations. While there is a hard split between the embedded and endpoint approaches for model serving, the line is blurred along the serialization dimension, because tools like Vertex AI support both TensorFlow and ONNX models. We’ll cover each of these quadrants in more detail.</p><p><strong>Embedded Serving with a Library Format<br></strong>Perhaps the most direct way to serve deep learning models is by running the models in-process on the production servers. If you are using TensorFlow models and the runtime is Rust, you can use <a href="https://github.com/tensorflow/rust">TensorFlow Rust</a>, and if the runtime is Golang you can use <a href="https://github.com/galeone/tfgo">tfgo</a>. If you are using PyTorch and serving with Rust, you can use <a href="https://github.com/LaurentMazare/tch-rs">tch-rs</a>. Most of these libraries provide bindings to the C++ APIs of the frameworks, while support and maintenance for these libraries varies by project.</p><p>This approach can work well when the model sizes are small and everything can fit in memory, and batch sizes are small making CPU inference viable. It can work for smaller propensity and ranking models, but limits the complexity of the models that you can serve. The production servers are unlikely to have GPUs, so in practice it has limited application for large models.</p><p>One of the key disadvantages of this approach is that adding a deep learning framework to your production servers may add significant dependencies to processes that are otherwise lightweight. TensorFlow is complex, because of all of the functionality that it provides, but many of these features may be unnecessary for your serving needs. <a href="https://github.com/huggingface/candle">Candle</a> by Hugging Face strives to provide a lightweight runtime for deep learning models, while offering compatibility with PyTorch models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/522/1*EeOcYCVMOIh7e6HbcwiNyw.png" /><figcaption>Serving an embedded model (<em>Image by Author</em>)</figcaption></figure><p><strong>Embedded Serving with a Portable Format<br></strong>As multiple deep learning frameworks gained popularity, there was a growing desire for a library agnostic approach for serving models. One of the proposals for achieving this goal is ONNX (Open Neural Network Exchange), which provides a standardized runtime for models trained from different machine learning libraries. The runtime is lightweight, because it is focused primarily on model serving, alleviating one of the main issues with the library-based approach to model serving.</p><p>The portable format for model serving provides interoperability between Python and other programming languages by defining a new model serialization format, that specifies how to run the computation graph for a deep learning model. One deployment option, as shown in the figure above, is to use Python and Tensorflow to train a TensorFlow model, use <a href="https://github.com/onnx/tensorflow-onnx">tf2onnx</a> to convert the model to ONNX format, and then use the <a href="https://ort.pyke.io/">ort</a> crate in Rust to serve the TF model in-process. The ort library is a wrapper for the <a href="https://onnxruntime.ai/">ONNX Runtime</a> and provides efficient model serving. ONNX provides multiple <a href="https://onnxruntime.ai/docs/execution-providers/">execution providers</a> that enable the framework to use hardware acceleration if available.</p><p>This approach has many of the same limitations as the library format approach, where it is most relevant for smaller models that can run efficiently on CPUs. In addition to ONNX, some tools I would place into this quadrant include <a href="https://tvm.apache.org/">Apache TVM</a>, which provides a custom compiler for deep learning models, and <a href="https://github.com/deeplearning4j/deeplearning4j">DL4J</a> (Deeplearning4J) which can import TensorFlow, PyTorch, and ONNX models.</p><p><strong>Endpoint Serving with a Library Format<br></strong>One of the most common approaches for enabling interoperability between system languages like Rust and models trained in Python, is to serve the models with Python and provide an endpoint for making requests to the model. This is such a common pattern that both TensorFlow and PyTorch offer this functionality as part of their ecosystems, through <a href="https://www.tensorflow.org/tfx/guide/serving">TF Serving</a> and <a href="https://docs.pytorch.org/serve/">TorchServe</a>. With this approach, you train and save models in the library-based format, and serve the models typically via a REST API. In addition to the options provided by these frameworks, you can use libraries such as <a href="https://fastapi.tiangolo.com/">FastAPI</a> to deploy your deep learning models as endpoints.</p><p>A key advantage to this approach is that the production servers do not need to fit the deep learning models into memory, since they route the model requests to different servers. The production servers are responsible for reshaping the input features into the tensor format required by the model endpoint, which is a <a href="https://www.tensorflow.org/tfx/serving/api_rest">JSON format</a> for TF Serving. Another advantage is that the model endpoint servers can be different instance types from the production servers, and may also include GPUs for hardware acceleration.</p><p>One of the main tradeoffs with this approach is that it introduces additional latency to model serving, because the production servers need to make a network request to the model endpoints, rather than evaluating the model in process. This latency is often minimal, if the model endpoints are co-located with the production servers. This tradeoff is usually worth it, because it allows for more flexibility in instance configuration, decouples model deployment from production server deployments, and enables more separation of concerns between DevOps and MLOps teams.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/1*9R5Z2a41Wm7hEjrUn33Wog.png" /><figcaption>Serving via model endpoint (<em>Image by Author</em>)</figcaption></figure><p><strong>Endpoint Serving with a Portable Format<br></strong>As the model endpoint approach for serving deep learning models gained popularity, public cloud providers started offering managed services for these capabilities including <a href="https://cloud.google.com/vertex-ai?hl=en">Vertex AI</a> on GCP and <a href="https://aws.amazon.com/sagemaker">SageMaker</a> on AWS. Additional tools in this quadrant include <a href="https://developer.nvidia.com/dynamo">NVidia Triton</a> and <a href="https://github.com/SeldonIO/seldon-core">seldon-core</a>. ONNX is one of the popular portable formats for serving with endpoints, while formats like <a href="https://developer.nvidia.com/tensorrt">TensorRT</a> focuses on high-performance inference. You can also host models yourself using libraries like FastAPI to wrap a model served with the ONNX runtime.</p><p>When using this approach, you create an ONNX model the same way as the embedded approach. The main difference is that the production servers no longer load the model into memory, and instead make a prediction request to a model endpoint, as shown in the figure above.</p><p>This approach can be a great option for getting the best throughput for your models. For example, Vertex AI provides an <a href="https://cloud.google.com/vertex-ai/docs/start/tensorflow#optimized_tensorflow_runtime">optimized TensorFlow runtime</a>, batching for prediction requests, and a gRPC interface for reducing latency and request payload sizes. But managed platforms also come with a cost, which can be prohibitive at AdTech scale.</p><p><strong>Conclusion<br></strong>There’s several tools for serving deep learning models that can be applied to the domain of AdTech. For simple models, the best approach may be serving embedding models using a deep learning framework directly, but if you need to serve larger models or support multiple platforms, then a different approach may be better suited for your workload. The endpoint approach is the most flexible, and there’s both self-hosted and managed offerings for this approach, but it does add another network hop to your system and can add more latency to your system. Whatever approach that you use for serving deep learning models, it’s good to focus on MLOps practices around observability to make sure that you can quickly capture and correct issues in production.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0ff9467ed946" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/serving-deep-learning-in-adtech-0ff9467ed946">Serving Deep Learning in AdTech</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Machine Learning Incidents in AdTech]]></title>
            <link>https://medium.com/data-science/machine-learning-incidents-in-adtech-e3f48d858054?source=rss-a80e1f69e782------2</link>
            <guid isPermaLink="false">https://medium.com/p/e3f48d858054</guid>
            <category><![CDATA[incident-response]]></category>
            <category><![CDATA[advertising]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Ben Weber]]></dc:creator>
            <pubDate>Wed, 29 Jan 2025 15:02:06 GMT</pubDate>
            <atom:updated>2025-01-29T15:02:06.549Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uR7otJsxKuscClVtI3kO4A.png" /><figcaption>Source: <a href="https://unsplash.com/photos/a-couple-of-signs-that-are-on-a-fence-xXbQIrWH2_A">https://unsplash.com/photos/a-couple-of-signs-that-are-on-a-fence-xXbQIrWH2_A</a></figcaption></figure><h4>Challenges with deep learning in production</h4><p>One of the biggest challenges I encountered in my career as a data scientist was migrating the core algorithms in a mobile AdTech platform from classic machine learning models to deep learning. I worked on a Demand Side Platform (DSP) for user acquisition, where the role of the ML models is to predict if showing ad impressions to a device will result in the user clicking on the ad and installing a mobile app. For a quick hands-on overview of the click prediction problem, please check out my <a href="https://medium.com/towards-data-science/deep-learning-for-click-prediction-in-mobile-adtech-9739fe3f52de">past post</a>.</p><p>While we were able to quickly get to a state where the offline metrics for the deep learning models were competitive with logistic regression models, it took awhile to get the deep learning models working smoothly in production, and we encountered many incidents along the way. We were able to start with small-scale tests using Keras for model training and Vertex AI for managed TensorFlow serving, and ran experiments to compare iterations of our deep learning models with our champion logistic regression models. We were eventually able to get the deep learning models to outperform the classic ML models in production, and modernize our ML platform for user acquisition.</p><p>When working with machine learning models at the core of a complex system, there are going to be situations where things are going to go off the rails and it’s important to be able to quickly recover and learn from these incidents. During my time at Twitch, we used the <a href="https://www.form3.tech/blog/engineering/five-ws-incident-mngmnt">Five W’s approach</a> to writing postmortems for incidents. The idea is to identify “what” went wrong, “when” and “where” it occurred, “who” was involved, and “why” a problem resulted. The follow up is to then establish how to avoid this type of incident in the future and to set up guardrails to prevent similar issues. The goal is to build a more and more robust system over time.</p><p>In one of my past roles in AdTech, we ran into several issues when migrating from classic ML models to deep learning. We eventually got to a state where we had a robust pipeline for training, validating, and deploying models that improved upon our classic models, but we ran into incidents during this process. In this post we’ll cover 8 of the incidents that occurred and describe the following steps we took for incident management:</p><ol><li>What was the issue?</li><li>How was it found?</li><li>How was it fixed?</li><li>What did we learn?</li></ol><p>We identified a variety of root causes, but often aligned on similar solutions when making our model pipelines more robust. I hope sharing details about these incidents provides some guidance on what can go wrong when using deep learning in production.</p><h4>Incident 1: Untrained Embeddings</h4><p><em>What was the issue?<br></em>We found that many of the models that we deployed, such as predicting click and install conversion, were poorly calibrated. This meant that the predicted value of conversion by the model was much higher than the actual conversion that we saw for impressions that we served. After drilling down further, we found that the miscalibration was worse on categorical features where we had sparse training data. Eventually we discovered that we had embedding layers in our install model where we had no training data available for some of the vocabulary entries. What this meant is that when fitting the model, we weren’t making any updates to these entries, and the coefficients remained set to their randomized initialized weights. We called this incident “Untrained Embeddings”, because we had embedding layers where some of the layer weights never changed during model training.</p><p><em>How was it found?<br></em>We mostly discovered this issue through intuition after reviewing our models and data sets. We used the same vocabularies for categorical features across two models, and the install model data set was smaller than the click model data set. This meant that some of the vocabulary entries that were fine to use for the click model were problematic for the install model, because some of the vocab entries did not have training examples in the smaller data set. We confirmed that this was the issue by comparing the weights in the embedding layers before and after training, and finding that a subset of the weights were unchanged after fitting the model. Because we randomly initialized the weights in our Keras models, this led to issues with the model calibration on live data.</p><p><em>How was it fixed?<br></em>We first limited the size of our vocabularies used for categorical features to reduce the likelihood of this issue occurring. The second change we made was setting the weights to 0 for any embedding layers entries where the weights were unchanged during training. Longer term, we moved away from reusing vocabularies across different prediction tasks.</p><p><em>What did we learn?<br></em>We discovered that this was one of the issues that was leading to model instability, where models with similar performance on offline metrics would have noticeably different performance when deployed to production. We ended up building more tooling to compare model weights across training runs as part of our model validation pipeline.</p><h4>Incident 2: Padding Issue with Batching for TensorFlow Serving</h4><p><em>What was the issue?<br></em>We migrated from Vertex AI for model serving to an in-house deployment of TensorFlow serving, to deal with some of the tail-end latency issues that we were encountering with Vertex at the time. When making this change, we ran into an issue with how to deal with sparse tensors when enabling batching for TensorFlow serving. Our models contained sparse tensors for features, such as the list of known apps installed on a device, that could be empty. When we enabled <a href="https://www.tensorflow.org/tfx/serving/serving_config#batching_configuration">batching</a> when serving on Vertex AI, we were able to use empty arrays without issue, but for our in-house model serving we got error responses when using batching and passing empty arrays. We ended up passing “[0]” values instead of “[ ]” tensor values to avoid this issue, but this again resulted in poorly calibrated models. The core issue is that “0” referred to a specific app rather than being used for out-of-vocab (OOV). We were introducing a feature parity issue to our models, because we only made this change for model serving and not for model training.</p><p><em>How was it found?<br></em>Once we identified the change that had been made, it was straightforward to demonstrate that this padding approach was problematic. We took records with an empty tensor and changed the value from “[]” to “[0]” while keeping all of the other tensor values constant, and showed that this change resulted in different prediction values. This made sense, because we were changing the tensor data to claim that an app was installed on the device where that was not actually the case.</p><p><em>How was it fixed?<br></em>Our initial fix was to change the model training pipeline to perform the same logic that we performed for model serving, where we replace empty arrays with “[0]”, but this didn’t completely address this issue. We later changed the vocab range from [0, n-1] to [0, n], where 0 had no meaning and was added to every tensor. This meant that every sparse tensor had at least 1 value and we were able to use batching with our sparse tensor setup.</p><p><em>What did we learn?<br></em>This issue mostly came up due to different threads of work on the model training and model serving pipelines, and lack of coordination. Once we identified the differences between the training and serving pipelines, it was obvious that this discrepancy could cause issues. We worked to improve on this incident by including data scientists as reviewers on pull requests on the production pipeline to help identify these types of issues.</p><h4>Incident 3: Untrained Model Deployment</h4><p><em>What was the issue?<br></em>Early on in our migration to deep learning models, we didn’t have many guardrails in place for model deployments. For each model variant we were testing we would retrain and automatically redeploy the model daily, to make sure that the models were trained on recent data. During one of the training runs, the model training resulted in a model that always predicted a 25% click rate regardless of the input data and the ROC AUC metric on the validation data set was 0.5. We had essentially deployed a model to production that always predicted a 25% click rate regardless of any of the feature inputs.</p><p><em>How was it found?<br></em>We first identified the issue using our system monitoring metrics in Datadog. We logged our click predictions (<em>p_ctr</em>) as a histogram metric, and Datadog provides p50 and p99 aggregations. When the model was deployed, we saw the p50 and p99 values for the model converge to the same value of ~25%, indicating that something had gone wrong with the click prediction model. We also reviewed the model training logs and saw that the metrics from the validation data set indicated a training error.</p><p><em>How was it fixed?<br></em>In this case, we were able to rollback to the click model from yesterday to resolve the issue, but it did take some time for the incident to be discovered and our rollback approach at the time was somewhat manual.</p><p><em>What did we learn?<br></em>We found that this issue with bad model training occurred around 2% of the time and needed to set up guardrails against deploying these models. We added a model validation module to our training pipeline that checked for thresholds on the validation metrics, and also compared the outputs of the new and prior runs on the model on the same data set. We also set up alerts on Datadog to flag large changes in the p50 <em>p_ctr</em> metric and worked on automating our model rollback process.</p><h4>Incident 4: Bad Warmup Data for TensorFlow Serving</h4><p><em>What was the issue?<br></em>We used <a href="https://www.tensorflow.org/tfx/serving/saved_model_warmup">warmup files</a> for TensorFlow serving to improve the rollout time of new model deployments and to help with serving latency. We ran into an issue where the tensors defined in the warmup file did not correspond to the tensors defined in the TensorFlow model, resulting in failed model deployments.</p><p><em>How was it found?<br></em>In an early version of our in-house serving, this mismatch between warmup files and model tensor definitions would cause all model serving to come to a halt and require a model rollback to stabilize the system. This is another incident that was initially captured by our operational metrics on Datadog, since we saw a large spike in model serving error requests. We confirmed that there was an issue with the newly deployed model by deploying it to Vertex AI and confirming that the warmup files were the root cause of the issue.</p><p><em>How was it fixed?<br></em>We updated our model deployment module to confirm that the model tensors and warmup files were compatible by launching a local instance of TensorFlow serving in the model training pipeline and sending sample requests using the warmup file data. We also did additional manual testing with Vertex AI when launching new types of models with noticeably different tensor shapes.</p><p><em>What did we learn?<br></em>We learned that we needed to have different environments for testing TensorFlow model deployments before pushing them to production. We were able to do some testing with Vertex AI, but eventually set up a staging environment for our in-house version of TensorFlow serving to provide a proper CI/CD environment for model deployment.</p><h4>Incident 5: Problematic Time-Based Features</h4><p><em>What was the issue?<br></em>We explored some time-based features in our models, such as <em>weeks_ago</em>, to capture changes in behavior over time. For the training pipeline, this feature was calculated as <em>floor(date_diff(today, day_of_impression)/7)</em>. It was a highly ranked feature in some of our models, but it also added unintended bias to our models. During model serving, this value is always set to 0, since we are making model predictions in real time, and today is the same as <em>day_of_impression</em>. The key issue is that the model training pipeline was finding patterns in the training data that may create bias issues when applying the model on live data.</p><p><em>How was it found?<br></em>This was another incident that we found mostly through intuition and later confirmed to be a problem by comparing the implementation logic across the training and model serving pipelines. We found that the model serving pipeline always set the value to 0 while the training pipeline used a wide range of values given that we often use months old examples for training.</p><p><em>How was it fixed?<br></em>We created a variant of the model with all of the relative time-based features removed and did an A/B test to compare the performance of the variants. The model that included the time based features performed better on the holdout metrics during offline testing, but the model with the features removed worked better in the A/B test and we ended up removing the features from all of the models.</p><p><em>What did we learn?<br></em>We found that we had introduced bias into our models in an unintended way. The features were compelling to explore, because user behavior does change over time, and introducing these features did result in better offline metrics for our models. Eventually we decided to categorize these as problematic under the feature parity category, where we see differences in values between the model training and serving pipelines.</p><h4>Incident 6: Feedback Features</h4><p><em>What was the issue?<br></em>We had a feature called <em>clearing_price</em> that logged how high we were willing to bid on an impression for a device during the last time that we served an ad impression for the device. This was a useful feature, because it helped us to bid on devices with a high bid floor, where the model needs high confidence that a conversion event will occur. This feature on its own generally wasn’t problematic, but it did become a problem during an incident where we introduced bad labels into our training data set. We ran an experiment that resulted in false positives in our training data set, and we started to see a feedback issue where the model bias became an issue.</p><p><em>How was it found?<br></em>This was a very challenging incident to identify the root cause of, because the experiment that generated the false positive labels was run on a small cohort of traffic, so we did not see a sudden change in operational metrics like we did with some of the other incidents in Datadog. Once we identified which devices and impressions were impacted by this test, we looked at the feature drift of our data set and found that the average value of the <em>clearning_price</em> feature was increasing steadily since the rollout of the experiment. The false positives in the label data were the root cause of the incident, and the drift in this feature was a secondary issue that was causing the models to make bad predictions.</p><p><em>How was it fixed?<br></em>The first step was to rollback to a best-known model before the problematic experiment was launched. We then cleaned up the data set and removed the false positives that we could identify from the training data set. We continued to see issues and also made the call to remove the problematic feature from our models, similar to the time-based features, to prevent this feature from creating future feedback loops in the future.</p><p><em>What did we learn?<br></em>We learned that some features are helpful for making the model more confident in predicting user conversions, but are not worth the risk because they can introduce a tail-spin effect where the models quickly deteriorate in performance and create incidents. To replace the clearing price feature, we introduced new features using the minimum bid to win values from <a href="https://developers.applovin.com/en/max/demand-partners/demand-side-platforms/applovin-ortb-specification/bid-responses/">auction callbacks</a>.</p><h4>Incident 7: Bad Feature Encoding</h4><p><em>What was the issue?<br></em>We explored a few features that were numeric and computed as ratios, such as the average click rate of a device, computed as the number of clicks over the number of impressions served to the device. We ran into a feature parity issue where we handled divide by zero in different ways between the training and serving model pipelines.</p><p><em>How was it found?<br></em>We have a feature parity check where we log the tensors created during model inference for a subset of impressions and run the training pipeline on these impressions and compare the values generated in the training pipeline with the logged value at serving time. We noticed a large discrepancy for the ratio based features and found that we encoded divide by zero as -1 in the training pipeline and 0 in the serving pipeline.</p><p><em>How was it fixed?<br></em>We updated the serving pipeline to match the logic in the training pipeline, where we set the value to -1 when a divide by zero occurs for the ratio based features.</p><p><em>What did we learn?<br></em>Our pipeline for detecting feature parity issues allowed us to quickly identify the root cause of this issue once the model was deployed to production, but it’s also a situation we want to avoid before a model is deployed. We applied the same learning from incident 2, where we included data scientists on pull request reviews to help identify potential issues between our training and serving model pipelines.</p><h4>Incident 8: String Parsing</h4><p><em>What was the issue?<br></em>We used a 1-hot encoding approach where we choose the top <em>k</em> values, which are assigned indices from 1 to <em>k</em>, and use 0 as an out-of-vocab (OOV) value. We ran into a problem with the encoding from strings to integers when dealing with categorical features such as app bundle, which often has additional characters. For example, the vocabulary may map the bundle com.dreamgames.royalmatch to index 3, but in the training pipeline the bundle is set to com.dreamgames.royalmatch$hl=en_US and the value gets encoded to 0, because it is considered OOV. The core issue we ran into was different logic for sanitizing string values between the training and serving pipelines before applying vocabularies.</p><p><em>How was it found?<br></em>This was another incident that we discovered with our feature parity checker. We found several examples where one pipeline encoded the values as OOV while the other pipeline assigned non-zero values. We then compared the feature values prior to encoding and noticed discrepancies between how we did string parsing in the training and serving pipelines.</p><p><em>How was it fixed?<br></em>Our short term fix was to update the training pipeline to perform the same string parsing logic as the serving pipeline. Longer term we focused on truncating the app bundle names at the data ingestion step, to reduce the need for manual parsing steps in the different pipelines.</p><p><em>What did we learn?<br></em>We learned that dealing with problematic strings at data ingestion provided the most consistent results when dealing with string values. We also ran into issues with unicode characters showing up in app bundle names and worked to correctly parse these during ingestion. We also found it necessary to occasionally inspect the vocabulary entries that are generated by the system to make sure specific characters were not showing up in entries.</p><h4>Takeaways</h4><p>While it may be tempting to use deep learning in production for model serving, there’s a lot of potential issues that you can encounter with live model serving. It’s important to have robust plans in place for incident management when working with machine learning models, so that you can quickly recover when model performance becomes problematic and learn from these missteps. In this post we covered 8 different incidents I encountered when using deep learning to predict click and install conversion in a mobile AdTech platform. Here’s are the key takeaways I learned from these machine learning incidents:</p><ul><li>It’s important to log feature values, encoded values, tensor values, and model predictions during model serving, to ensure that you do not have feature parity or model parity issues in your model pipelines.</li><li>Model validation is a necessary step in model deployment and test environments can help reduce incidents.</li><li>Beware of the features that you include in your model, they may be introducing bias or causing unintended feedback.</li><li>If you have different pipelines for model training and model serving, the team members working on the pipelines should be reviewing each other’s pull requests for ML feature implementations.</li></ul><p>Machine learning is a discipline that can learn a lot from DevOps to reduce the occurrence of incidents, and MLOps should include processes for efficiently responding to issues with ML models in production.</p><p>Ben Weber is a principal machine learning engineer at Aarki. He’s worked in the gaming and AdTech industries for over a decade at Microsoft, Amazon, Electronic Arts, and Zynga.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e3f48d858054" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/machine-learning-incidents-in-adtech-e3f48d858054">Machine Learning Incidents in AdTech</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deep Learning for Click Prediction in Mobile AdTech]]></title>
            <link>https://medium.com/data-science/deep-learning-for-click-prediction-in-mobile-adtech-9739fe3f52de?source=rss-a80e1f69e782------2</link>
            <guid isPermaLink="false">https://medium.com/p/9739fe3f52de</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[advertising]]></category>
            <category><![CDATA[mobile-app-development]]></category>
            <category><![CDATA[hands-on-tutorials]]></category>
            <dc:creator><![CDATA[Ben Weber]]></dc:creator>
            <pubDate>Thu, 23 Jan 2025 04:38:07 GMT</pubDate>
            <atom:updated>2025-01-24T17:58:40.426Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K3RRU2maaQdGOIvT9tMFIw.png" /><figcaption>Source: <a href="https://pixabay.com/illustrations/rays-stars-light-explosion-galaxy-9350519/">https://pixabay.com/illustrations/rays-stars-light-explosion-galaxy-9350519/</a></figcaption></figure><h4>Machine Learning for Real-Time Bidding</h4><p>The past few years were a revolution for the mobile advertising and gaming industries, with the broad adoption of neural networks for advertising tasks, including click prediction. This migration occurred prior to the success of Large Language Models (LLMs) and other AI innovations, but is building on the momentum of this wave. The mobile gaming industry is spending billions on User Acquisition every year, and top players in this space such as Applovin have market caps of over $100B. In this post, we’ll discuss a conventional ML approach for click prediction, offer motivations for the migration to deep learning for this task, provide a hands-on example of the benefits of this approach using a data set from <a href="https://www.kaggle.com/">Kaggle</a>, and detail some of the enhancements that this approach provides.</p><p>Most large tech companies in the AdTech space are likely using deep learning for predicting user behavior. Social Media platforms have embraced the migration from classic machine learning (ML) to deep learning, as indicated by this <a href="https://www.reddit.com/r/RedditEng/comments/16bovfo/our_journey_to_developing_a_deep_neural_network/">Reddit post</a> and this <a href="https://www.linkedin.com/blog/engineering/machine-learning/challenges-and-practical-lessons-from-building-a-deep-learning-b">LinkedIn post</a>. In the mobile gaming space <a href="https://www.moloco.com/blog/dr-sechan-oh-explains-molocos-machine-learning">Moloco</a>, <a href="https://liftoff.io/blog/cortex-ml-platform-announcement/">Liftoff</a>, and <a href="https://cloud.google.com/blog/products/compute/applovin-builds-ai-ad-platform-on-google-cloud-g2-vms">Applovin </a>have all shared details on their migration to deep learning or hardware acceleration to improve their user acquisition platforms. Most Demand Side Platforms (DSPs) are now looking to leverage neural networks to improve the value that their platforms provide for mobile user acquisition.</p><p>We’ll start by discussing logistic regression as an industry standard for predicting user actions, discuss some of the shortfalls of this approach, and then showcase deep learning as a solution for click prediction. We’ll provide a deep dive on implementations for both a <a href="https://alumni.soe.ucsc.edu/~bweber/ClickLogit.html">classic ML notebook</a> and <a href="https://alumni.soe.ucsc.edu/~bweber/ClickDNN.html">deep learning notebook</a> for the task of predicting if a user is going to click on an ad. We won’t dive into the state of the art, but we will highlight where deep learning provides many benefits.</p><p>All images in this post, with the exception of the header image, were created from by the author in the notebooks linked above. The Kaggle data set that we explore in this post has the <a href="https://creativecommons.org/publicdomain/zero/1.0/">CC0: Public Domain</a> license.</p><h3>Cost Per Click Modeling</h3><p>One of the goal types that DSPs typically provide for user acquisition is a cost per click model, where the advertiser is charged each time that the platform serves an impression on a mobile device and the user clicks. We’ll focus on this goal type to keep things simple, but most advertisers prefer goal types focused on driving installs or acquiring users that will spend money in their app.</p><p>In programmatic bidding, a DSP is integrated with one or more ad exchanges, which provide inventory for the platform to bid on. Most exchanges use a version of the <a href="https://www.iab.com/wp-content/uploads/2016/03/OpenRTB-API-Specification-Version-2-5-FINAL.pdf">OpenRTB specification</a> to send bid requests to DSPs and get back responses in a standardized format. For each ad request from a Supply Side Platform (SSP), the exchange runs an auction and the DSP that responds with the highest price wins. The exchange then provides the winning bid response to the SSP, which may result in an ad impression on a mobile device.</p><p>In order for a DSP to integrate with an ad exchange, there is an onboarding process to make sure that the DSP can meet the technical requirements of an exchange, which typically requires DSPs to respond to bid requests within 120 milliseconds. What makes this a huge challenge is that some exchanges provide over 1 million bid requests per second, and DSPs are usually integrating with several exchanges. For example, <a href="https://www.moloco.com/r-d-blog/challenges-in-building-a-scalable-demand-side-platform-dsp-service">Moloco responds</a> to over 5 million requests per second (QPS) during peak capacity. Because of the latency requirements and massive scale of requests, it’s challenging to use machine learning for user acquisition within a DSP, but it’s also a requirement in order to meet advertiser goals.</p><p>In order to make money as a DSP you need to be able to deliver ad impressions that meet your advertiser goals, while also generating net revenue. To accomplish this, a DSP needs to bid less than the expected value that an impression will deliver, while also bidding high enough to exceed the bid floor of a request and win in auctions against other DSPs. A demand-side platform is billed per impression shown, which corresponds to a CPM (cost per impression) model. If the advertiser goal is a target cost per click (CPC), then the DSP needs to translate the CPC value to a CPM value for bidding. We can do this using machine learning and predicting the likelihood of a user to click on an impression, which we call <em>p_ctr</em>. We can this calculate a bid price as follows:</p><pre>cpm = target_cpc * p_ctr<br>bid_price = cpm * bid_shade</pre><p>We use the likelihood of a click event to convert from cost per click to cost per impression and then apply a bid shade with a value of less than 1.0 to make sure that we are delivering more value for advertisers than we are paying to the ad exchange for serving the impression.</p><p>In order for a click prediction model to perform well for programmatic user acquisition, we want a model that has the following properties:</p><ol><li><strong>Large Bias<br></strong>We want a click model that is highly discriminative and able to differentiate between impressions unlikely to result in a click and ones that are highly likely to result in a click. If a model does not have sufficient bias, it won’t be able to compete with other DSPs in auctions.</li><li><strong>Well Calibrated<br></strong>We want the predicted and actual conversion rates of the model to align well for the ad impressions the DSP purchases. This means we have a preference for models where the output can be interpreted as a probability of a conversion occurring. Poor calibration will result in inefficient spending. A sample calibration plot is shown below.</li><li><strong>Fast Evaluation<br></strong>We want to reduce our compute cost when bidding on millions of requests per second and have models that are fast to inference.</li><li><strong>Parallel Evaluation</strong><br>Ideally, we want to be able to run model inference in parallel to improve throughput. For a single bid request, a DSP may be considering hundreds of campaigns to bid for, and each one needs a <em>p_ctr</em> value.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*beCmyaZdfGTMVz2GAY92Xw.png" /><figcaption>A model calibration plot (Created by the author in the <a href="https://alumni.soe.ucsc.edu/~bweber/ClickLogit.html">ClickLogit Notebook</a>)</figcaption></figure><p>Many ad tech platforms started with logistic regression for click prediction, because they work well for the first 3 desired properties. Over time, it was discovered that deep learning models could perform better than logistic regression on the bias goal, with neural networks being better at discriminating between click and no-click impressions. Additionally, neural networks can use batch evaluation and align will with the fourth property of parallel evaluation.</p><p>DSPs were able to push logistic regression models pretty far, which is what we’ll cover in the next section, but they do have some boundaries in their application to user acquisition. Deep neural networks (DNN) can overcome some of these issues, but present new challenges of their own.</p><h3>The Big Logistic Era</h3><p>Ad Tech companies have been using logistic regression for more than a decade for click prediction. For example, Facebook presented using logit in combination with other models at <a href="https://research.facebook.com/publications/practical-lessons-from-predicting-clicks-on-ads-at-facebook/">ADKDD 2014</a>. There are many different ways of using logistic regression for click prediction, but I’ll focus on a single approach I worked on in the past called Big Logistic. The general idea was to turn all of your features into tokens, create combinations of tokens to represent crosses or feature interactions, and then create a list of tokens that you use to convert your input features into a sparse vector representation. It’s an approach where every feature is 1-hot encoded and all of the features are binary, which helps simplify hyperparameter tuning for the click model. It’s an approach that can support numeric, categorical, and many-hot features as inputs.</p><p>To determine what this approach looks like in practice, we’ll provide a hands-on example of training a click prediction model using the <a href="https://www.kaggle.com/datasets/arashnic/ctr-in-advertisement">CTR In Advertisement</a> Kaggle data set. The full notebook for feature encoding, model training and evaluation is <a href="https://alumni.soe.ucsc.edu/~bweber/ClickLogit.html">available here</a>. I used Databricks, PySpark, and MLlib for this pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8ydYYLsyCeM_zsM66ZP5EQ.png" /><figcaption>Sample Data from the Kaggle Training Data Set</figcaption></figure><p>The dataset provides a training data set with labels and a test data set without labels. For this exercise we’ll split the training file into train and test groups, so that we have labels available for all records. We create a 90/10% split where the train set has 414k records and test has 46k records. The data set has 15 columns, which includes a label, 2 columns that we’ll ignore (session_id and user_id) and 12 categorical values that we’ll use as features in our model. A few sample records are shown in the table above.</p><p>The first step we’ll perform is tokenizing the data set, which is a form of 1-hot encoding. We convert each column to a string value by concatenating the feature name and feature value. For example, we would create the following tokens for the first row in the above table:</p><p><em>[“product_c”, “campaign_id_359520”, “webpage_id_13787”, ..]</em></p><p>For null values, we use “null” as the value, e.g. “product_null”. We also create all combinations of two features, which generates additional tokens:</p><p><em>[“product_c*campaign_id_359520”, “”, “product_c*webpage_id_13787”, “campaign_id_359520*webpage_id_13787”,..]</em></p><p>We use a UDF on the PySpark dataframe to convert the 12 columns into a vector of strings. The resulting dataframe includes the token list and label, as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DrOviK1E8E3taYh68Hhwtg.png" /><figcaption>The Tokenized Data Set</figcaption></figure><p>We then create a top tokens list, assign an index to each token in this list, and use the mapping of token name to token index to encode the data. We limited our token list to values where we have at least 1000 examples, which resulted in roughly 2,500 tokens.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/522/1*SC-Q8P6IjpLrO3XaXrCojA.png" /></figure><p>We then apply this token list to each record in the data set to convert from the token list to a sparse vector representation. If a record includes the token for an index, the value is set to 1, and if the token is missing the value is set to 0. This results in a data set that we can use with MLlib to train a logistic regression model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7SzMSdMX0UN4nDzvenM68A.png" /><figcaption>The encoded data set, ready for MLlib</figcaption></figure><p>We split the dataset into train and test groups, fit the model on the train data set, and then transform the test data set to get predictions.</p><pre>classifier = LogisticRegression(featuresCol = &#39;features&#39;,\<br>    labelCol = &#39;label&#39;, maxIter = 50, regParam = 0.01, elasticNetParam = 0)<br>lr_model = classifier.fit(train_df)<br>pred_df = lr_model.transform(test_df).cache()</pre><p>This process resulted in the following offline metrics, which we’ll compare to a deep learning model in the next section.</p><pre>Actual Conv: 0.06890<br>Predicted Conv: 0.06770<br>Log Loss: 0.24795<br>ROC AUC: 0.58808<br>PR AUC: 0.09054</pre><p>The AUC metrics don’t look great, but there isn’t much signal in the data set with the features that we explored, and other participants in the Kaggle competition generally had lower ROC metrics. One other limitation of the data set is that the categorical values are low cardinality, with only a small number of distinct values. This resulted in a low parameter count, with only 2,500 features, which limited the bias of the model.</p><p>Logistic regression works great for click prediction, but where we run into challenges is when dealing with high cardinality features. In mobile ad tech, the publisher app, where the ad is rendered, is a high cardinality feature, because there are millions of potential mobile apps that may render an ad. If we want to include the publisher app as a feature in our model, and are using 1-hot encoding, we are going to end up with a large parameter count. This is especially the case when we perform feature crosses between the publisher app and other high cardinality features, such as the device model.</p><p>I’ve worked with logistic regression click models that have more than 50 million parameters. At this scale, MLlib’s implementation of logistic regression runs into training issues, because it densifies the vectors in its training loop. To avoid this bottleneck, I used the <a href="https://github.com/TalkingData/Fregata">Fregata library</a>, which performs gradient descent using the sparse vector directly in a model averaging strategy.</p><p>The other issue with large click models is model inference. If you include too many parameters in your logit model, it may be slow to evaluate, significantly increasing your model serving costs.</p><h3>The Deep Learning Era</h3><p>Deep learning is a good solution for click models, because it provides methods for working efficiently with very sparse features with high cardinality. One of the key layers that we’ll use in our deep learning model is an embedding layer, which takes a categorical feature as an input and a dense vector as an output. With an embedding layer, we learn a vector for each of the entries in our vocabulary for a categorical feature, and the number of parameters is the size of the vocabulary times the output dense vector size, which we can control. Neural networks can reduce the parameter count by creating interactions between the dense layers output of embeddings, rather than making crosses between the sparse 1-hot encoded approach used in logistic regression.</p><p>Embedding layers are just one way that neural networks can provide improvements over logistic regression models, because deep learning frameworks provide a variety of layer types and architectures. We’ll focus on embeddings for our sample model to keep things simplistic. We’ll create a pipeline for encoding the data set into TensorFlow Records and then train a model using embeddings and cross layers to perform click prediction. The full notebook for data preparation, model training and evaluation is <a href="https://alumni.soe.ucsc.edu/~bweber/ClickDNN.html">available here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/1*ZPo6bKjg3yG6q6TH0WAZNg.png" /><figcaption>Vocabulary for the Product Feature</figcaption></figure><p>The first step that we perform is generating a vocabulary for each of the features that we want to encode. For each feature, we find all values with more than 100 instances, and everything else is grouped into an out-of-vocab (OOV) value. We then encode all of the categorical features and combine them into a single tensor named <em>int</em>, as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/534/1*SWCU68o2Dj_aU7AIib5ASw.png" /><figcaption>Features Reshaped into a Tensor</figcaption></figure><p>We then save the Spark dataframe as TensorFlow records to cloud storage.</p><pre>output_path = &quot;dbfs:/mnt/ben/kaggle/train/&quot;<br>train_df.write.format(&quot;tfrecords&quot;).mode(&quot;overwrite&quot;).save(output_path)</pre><p>We then copy the files to the driver node and create TensorFlow data sets for training and evaluating the model.</p><pre>def getRecords(paths):<br>    features = {<br>        &#39;int&#39;: FixedLenFeature([len(vocab_sizes)], tf.int64),<br>        &#39;label&#39;: FixedLenFeature([1], tf.int64)<br>    }<br> <br>    @tf.function<br>    def _parse_example(x):<br>        f = tf.io.parse_example(x, features)<br>        return f, f.pop(&quot;label&quot;)<br> <br>    dataset = tf.data.TFRecordDataset(paths)<br>    dataset = dataset.batch(10000)<br>    dataset = dataset.map(_parse_example)<br>    return dataset<br><br>training_data = getRecords(train_paths)<br>test_data = getRecords(test_paths)</pre><p>We then create a Keras model, where the input layer is an embedding layer per categorical feature, we have two hidden cross layers, and a final output layer that is a sigmoid activation for the propensity prediction.</p><pre>cat_input = tf.keras.Input(shape=(len(vocab_sizes)),\<br>    name = &quot;int&quot;, dtype=&#39;int64&#39;)<br>input_layers = [cat_input]<br><br>cross_inputs = []<br>for attribute in categories_index:<br>    index = categories_index[attribute]<br>    size = vocab_sizes[attribute]<br><br>    category_input = cat_input[:,(index):(index+1)]<br>    embedding = keras.layers.Flatten()\<br>        (keras.layers.Embedding(size, 5)(category_input))<br>    cross_inputs.append(embedding)<br><br>cross_input = keras.layers.Concatenate()(cross_inputs)<br>cross_layer = tfrs.layers.dcn.Cross()<br>crossed_ouput = cross_layer(cross_input, cross_input)<br><br>cross_layer = tfrs.layers.dcn.Cross()<br>crossed_ouput = cross_layer(cross_input, crossed_ouput)<br><br>sigmoid_output=tf.keras.layers.Dense(1,activation=&quot;sigmoid&quot;)(crossed_ouput)<br>model = tf.keras.Model(inputs=input_layers, outputs = [ sigmoid_output ])<br>model.summary()</pre><p>The resulting model has 7,951 parameters, which is about 3 times the size of our logistic regression model. If the categories had larger cardinalities, then we would expect the parameter count of the logit model to be higher. We train the model for 40 epochs:</p><pre>metrics=[tf.keras.metrics.AUC(), tf.keras.metrics.AUC(curve=&quot;PR&quot;)]<br><br>model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=1e-3),\<br>    loss=tf.keras.losses.BinaryCrossentropy(), metrics=metrics)<br>history = model.fit(x = training_data, epochs = 40,\<br>    validation_data = test_data, verbose=0)</pre><p>We can now compare the offline metrics between our logistic regression and DNN models:</p><pre>                Logit    DNN <br>Actual Conv:    0.06890  0.06890<br>Predicted Conv: 0.06770  0.06574<br>Log Loss:       0.24795  0.24758<br>ROC AUC:        0.58808  0.59284<br>PR AUC:         0.09054  0.09249<br></pre><p>We do see improvements to the log loss metric where lower is better and the AUC metrics where higher is better. The main improvement is to the precision-recall (PR) AUC metric, which may help the model perform better in auctions. One of the issues with the DNN model is that the model calibration is worse, and the DNN average predicted value is further off than the logistic regression model. We would need to do a bit more model tuning to improve the calibration of the model.</p><h3><strong>What’s Next?</strong></h3><p>We are now in the era of deep learning for ad tech and companies are using a variety of architectures to deliver advertiser goals for user acquisition. In this post, we showed how migrating from logistic regression to a simple neural network with embedding layers can provide better offline metrics for a click prediction model. Here are some additional ways we could leverage deep learning to improve click prediction:</p><ol><li><strong>Use Embeddings from Pre-trained Models<br></strong>We can use models such as <a href="https://en.wikipedia.org/wiki/BERT_(language_model)">BERT</a> to convert app store descriptions into vectors that we can use as input to the click model.</li><li><strong>Explore New Architectures <br></strong>We could explore the <a href="https://arxiv.org/abs/1708.05123">DCN</a> and <a href="https://arxiv.org/abs/2012.06678">TabTransformer</a> architectures.</li><li><strong>Add Non-Tabular Data<br></strong>We could use <a href="https://github.com/christiansafka/img2vec">img2vec</a> to create input embeddings from creative assets.</li></ol><p>Thanks for reading!</p><p>Ben Weber is a machine learning engineer with over a decade of experience in gaming and ad tech with prior roles at Zynga, Microsoft, Amazon, and Electronic Arts.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9739fe3f52de" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/deep-learning-for-click-prediction-in-mobile-adtech-9739fe3f52de">Deep Learning for Click Prediction in Mobile AdTech</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>