P(beat)-Based Machine Learning vs. Meta Ads Top Creative: When Should You Replace the #1 Ad?

Branding & marketing stories from Soulpapamarketing(Soulpapa Marketing).

Brands with small ad budgets often make a common mistake when choosing Meta ad creatives. They put the creative with the highest ROAS over the last two weeks straight into the #1 slot. Since the number is high, it looks like a sound call. But the smaller the budget, the more that number swings with luck rather than creative quality.

When we judge creatives each week, we don’t use the ROAS ranking as-is. We calculate a separate number, which we internally call P(beat). This post explains what P(beat) is, how it’s calculated, and where we set the replacement threshold, using our actual operating code and backtest figures as they are.

Why the ROAS #1 creative is an illusion in low-budget Meta ads

A creative spending KRW 20,000–30,000 a day gets either 0 or 1 purchase on a given day. Its daily ROAS is either 0% or several hundred percent. Stack up two weeks of those days, and a creative that landed one big order on a single day can end up ranked first on the two-week total. Even if you remove that one order, it’s an ordinary creative.

Picking the highest number out of several creatives also creates overestimation. When each creative’s performance has noise mixed in, the one that comes out on top is usually one whose noise happened to skew upward. Even if each measurement is accurate, the selected result looks better than reality. Research on recommendation and ad systems also finds that selecting the item with the highest predicted value creates the same bias (arXiv 2205.09809, ICLR 2023). The more creatives there are and the smaller the samples, the bigger this bias gets.

Meta’s machine learning has the same problem. Within an ad set, Meta concentrates delivery on the creative it predicts will have the lowest cost per conversion, and those predictions are erratic for creatives with short histories. Low-budget accounts often have to make a human judgment before those predictions settle. P(beat) doesn’t take Meta’s concentrated results at face value. It runs them through actual daily performance once more as a filter.

So for low-budget accounts, we don’t ask how high a creative’s ROAS is. We ask whether this creative actually beats the current #1.

Related readingThe problem with chasing winning creatives blindly in low-budget Meta adsWhen you split campaigns and ad sets on a small budget, auction overlap keeps the budget from being spent and the machine learning phase never completes.

What P(beat) means and where the name comes from

P stands for probability, and beat means ‘to win’. Google Optimize, the A/B testing tool Google used to run, showed ‘probability to beat baseline’ on its experiment results screen, meaning the chance that a variant beats the original. P(beat) takes its name from that concept.

There are two differences. The comparison baseline is not the original version but the current campaign’s #1 creative. And the calculation uses a bootstrap method that resamples daily data, not Optimize’s Bayesian model. Put simply, P(beat) is the rate at which the challenger beats the #1 creative when the date composition is reshuffled. The higher it is, the harder it is to dismiss the win as luck. A 95% doesn’t mean it’s right 95% of the time. How often it was actually right is shown in the backtest below.

How P(beat) is calculated: resampling 14 days of data 2,000 times

The actual two weeks happened only once. So we reshuffle those 14 days to create many two-week windows that could have unfolded differently. We’ll call these simulated two-week periods. The calculation has four steps.

1. Pull the daily ad spend and purchase revenue for the last 14 days for both the #1 creative and the challenger.
2. Randomly draw 14 days from those 14 (the same day may be drawn more than once, and some days may not be drawn at all) to build one simulated two-week period. Repeat this 2,000 times to create 2,000 simulated two-week periods.
3. For each simulated two-week period, calculate each creative’s ROAS separately. Use the same set of drawn dates for both creatives.
4. P(beat) is the share of the 2,000 simulated two-week periods in which the challenger’s ROAS was higher than the #1’s.

Using the same dates for both creatives in step 3 is for a fair comparison. If there are days when the whole market fluctuates, such as the Chuseok holiday, payday, or a competitor’s promotion, both creatives need to be exposed to those effects equally for the comparison to be fair. If one creative only gets drawn on paydays and the other only on weekdays, the comparison is meaningless.

A simulated two-week period’s ROAS is calculated as total revenue ÷ total ad spend, not as the average of 14 daily ROAS values. Days with low ad spend produce extreme ROAS values like 0% or 900%, and averaging them lets those days distort the number. Dividing totals means days with more spend carry proportionally more weight. If there are fewer than 5 days of data, we don’t make a judgment, because the probability isn’t reliable enough.

The logic behind this method filtering out luck is simple. A day with a big windfall is left out of many of the 2,000 simulated two-week periods. A creative that won because of that day mostly loses in the periods where that day was excluded. A creative that sold a little every day stays about the same no matter which days are excluded.

Example P(beat): similar 2-week ROAS, opposite results

This is sample data made for illustration. Three creatives run in a campaign selling a product with an average order value of KRW 39,000. These are the results of running the calculation above as-is.

Looking at ROAS alone, A (363%) and B (418%) both look far better than the #1 (277%). But A’s ROAS falls to around 113% if you remove the one day when orders concentrated. A P(beat) of 60% is only slightly better than a coin flip. B sold steadily at one order a day, and its P(beat) is 99%. Filtering by P(beat), A is eliminated and only B remains.

Even so, in this example we don’t change the #1. Only B passed 95%, and B’s expected ROAS of 259% (418% × 0.62) is still below 332%, which is 1.2× the #1’s 277%. The switching conditions and expected ROAS are explained below. B stays on the watch list as a candidate for next week.

How the 95% P(beat) threshold was set: backtest of 5,482 pairs

The 95% threshold was chosen through a backtest. Using daily Meta ad data from accounts we’ve operated since May 2023, we varied the threshold from 50% to 95% and measured the hit rate. Since the hit rate rose as the threshold went up, we use 95%, the strictest of the options tested.

• We went back to each Monday in the past and calculated P(beat) using only the 14 days of data available up to that point.
• The #1 was defined as the creative with the largest ad spend in the campaign over the prior 7 days.
• We scored the outcome by whether the challenger’s actual ROAS over the following 4 weeks beat the #1’s. We only counted pairs where both creatives spent at least KRW 20,000 in the periods before and after.
• We set the threshold using the first 70% of the period and checked whether it held up in the last 30%.

In July 2026, when we first set the threshold, 84% of creatives with P(beat) at or above 95% among 4,224 pairs actually beat the #1 in the following 4 weeks. While writing this post, we added data through September and reran it on 5,482 pairs. The results are below.

The first line is 61% because we defined the #1 by ad spend. A creative with the biggest spend is more likely to see its ROAS decline, so it’s not surprising that some challengers beat it on ROAS. What P(beat) does is pick out, from among those, the ones that will actually win, raising the hit rate from 61% to 81%.

When we measured only the four brands with sufficient sample sizes, the range widened from 68% to 90%. So brands with enough data use their own figures, while brands with small samples use the pooled figure, which is labeled as such. Looking only at the last 30% of the period, the rate was 76%.

Even above 95%, one in four or five calls is still wrong. This backtest has limitations too. Creatives that were switched off within the following 4 weeks couldn’t be scored, so they were excluded and the calculation skews toward creatives that survived. And because pairs were built by shifting the reference date each week, the same creative appears multiple times. That’s why we don’t change the #1 based on P(beat) alone.

One more point: not all accounts in this backtest were spending at the level of tens of thousands of KRW a day. Challengers often ran on small budgets, but some #1 creatives had larger budgets. If an entire account runs at tens of thousands of KRW a day, the hit rate may differ. So rather than using the 95% and the 0.62 below as they are, we recommend re-measuring them with your own account records.

When to replace the Meta ad #1 creative: when three conditions hold at once

P(beat) is shown for every challenger each week, but a challenger is only put forward as a replacement candidate when all three conditions below are met at the same time.

1. At least 2 challengers have P(beat) above the threshold (95%)
2. The median expected ROAS of those creatives is at least 1.2× the #1’s 2-week ROAS (the #1’s ROAS is used without any discount)
3. At least 3 challengers that can be evaluated are running

Condition 1 filters out cases where a single creative passed 95% by chance. Condition 2 reflects the judgment that switching for a small difference only costs money. Condition 3 checks whether there are enough creatives to split budget among after the switch. A newly launched creative is still in the learning phase, so its P(beat) is displayed but not used as a basis for replacement.

In accounts spending tens of thousands of KRW a day, condition 3 is often the one that isn’t met. If there are fewer than 3 challengers, we don’t change the #1. Instead, we first add more challengers to compare. Each creative needs to accumulate a meaningful amount of spend over 14 days before its P(beat) can be calculated.

How much ROAS falls when budget increases: expected ROAS calculation

A creative that ran on a small budget often sees its ROAS drop when it becomes the #1 and receives more budget. That’s why condition 2 uses expected ROAS instead of the 2-week ROAS. Pinning down exactly why it drops is hard because there are several causes.

We also measured how much it drops, using the same data (July 2026 baseline). For creatives that had P(beat) at or above 90% in the past, we took the median of the share of their prior 2-week ROAS that remained in actual ROAS, grouped by how much their spend later increased.

Even when spend stayed the same, ROAS fell to about 83% of the prior two weeks. This is the same direction as the overestimation discussed earlier. We assume that when a creative becomes #1, its spend rises 2–4×, and we use the 62% from that band in replacement decisions.

Without this adjustment, we’d end up overturning the #1 every time a creative with a high 2-week ROAS appeared. After the switch the number drops, and then the cycle repeats with another creative that looks higher.

How we keep the judgment criteria calibrated

Each week, when we judge, we record the P(beat) and expected ROAS for every challenger. As data accumulates, we check those records against actual outcomes and recalculate the 95% threshold and the adjustment factor. The 5,482-pair results in this post were also obtained by rerunning it this way. Changed values are applied only after a person has reviewed them.

The calculation runs automatically in the weekly creative efficiency report every Monday. We receive the results as a single message together with the list of creatives to stop. That lets us make the #1 replacement decision together with other decisions.

Related readingWhy your ads hit an efficiency ceiling and how to revive low-ROAS creativesWe walk through how to revive high-spend creatives with low ROAS, using real ad manager screens, so the brand can grow steadily.

Why you shouldn’t swap the #1 lightly in low-budget accounts

Accounts with generous budgets can reverse a wrong switch quickly because data piles up fast. Accounts spending tens of thousands of KRW a day are different. If budget floods into a wrong #1, you lose money for two weeks, and in the meantime the other creatives don’t accumulate enough data, which weakens the basis for the next decision.

So the smaller the budget, the stricter the criteria. The ROAS ranking is treated as reference only, and the basis for changing the #1 is the probability that filters out luck, plus the estimate adjusted for budget increases. If you’re changing the #1 often in your account, check each time whether that number came from a single day or accumulated over all 14 days.

Frequently asked questions

Insights from Soulpapa Marketing — Korea’s digital marketing agency.
Original Korean article: https://soulpapa.co.kr/2026/10/06/meta-ads-creative-pbeat/