When to A/B Test vs Just Fix: A Simple Decision Rule

A rule for the team that tests everything and ships nothing

How Is a Fix Different From an A/B Test?

Not everything needs a test.

A fix is a change that removes something objectively broken or unclear on a page. An A/B test compares two working versions when you are changing a judgement call. Knowing when to A/B test vs just fix comes down to 13 named triggers: 7 for fixes and 6 for tests, from WebMedic's CRO cadence.

Most in-house teams call themselves data-driven. Then every change goes into a test queue, and nothing ships.

Meanwhile the add-to-cart button on the mobile product page blends into the background. Everyone knows. It is "in the backlog."

That is not discipline. It is paralysis with a spreadsheet.

This post gives you permission to ship. It also gives you a rule for when that permission runs out.

Why Does a Page Only Get One Test at a Time?

Test slots are scarce.

A page can run only one A/B test at a time, because overlapping tests on the same page muddy each other's results. On WebMedic's cadence, one 25-day page test or two 2-week element tests fill a month. So each page has 1 to 2 test slots monthly.

Run a headline test and a hero image test on the same page together, and you cannot tell which change moved the number. So one test per page, start to finish.

That limit is the whole argument. Every slot spent testing something broken is a slot not spent on something that could move revenue.

Test type	Typical length	Slots per page per month
Page-level test	25 days	1
Element test (headline, UGC placement, section order)	2 weeks	2

Source: WebMedic testing cadence with Shopify clients.

Now run the math on your own store. Say three pages carry your revenue: the homepage, your best-selling product page and the cart. That gives you 3 to 6 test slots a month. This is an illustration, not a client result.

Seven obvious fixes queued as tests would outlast a full month of slots. And you would learn nothing new from any of them.

So which problems skip the queue?

Which Problems Should You Ship as a Fix Without a Test?

Some problems are not hypotheses.

Ship as a fix when the page is objectively broken or unclear. WebMedic's rule names 7 fix triggers: heavy images, a buried add-to-cart button, a weak checkout button, extra cart clicks, unclear shipping or discount messaging, missing product education, and popups that block cart or checkout intent.

None of these needs a theory about shopper psychology. A hidden button is a bug with a design file.

#	Fix trigger	What it looks like on the page
1	Slow load from obvious image weight	Full-size camera photos on the product page
2	Add-to-cart blends into the page	Button color matches the background, or variant clutter pushes it down on mobile
3	Checkout button weaker than secondary options	"Continue shopping" is bolder than "Checkout"
4	Cart needs unnecessary extra clicks	Extra page loads between cart and payment
5	Shipping or discount messaging objectively unclear	Shipping cost first appears on the last step
6	Product education missing for a known objection	Buyers keep asking about fit or ingredients and the page never answers
7	Popup interrupts cart or checkout intent	An email popup fires over the cart drawer

Source: WebMedic fix-versus-test rule.

Two of these have outside evidence behind them. Google rates a Largest Contentful Paint of 2.5 seconds or less as good (web.dev). If your hero image alone blows past that, compress it. Do not test it.

Baymard Institute looked at US online shoppers who abandoned a cart for a reason other than browsing. 40% cited extra costs like shipping, tax and fees. 17% cited a too long or complicated checkout (Baymard Institute). You do not need 25 days to confirm that a late shipping cost or an extra click hurts.

In our 30-day rollout, week 2 is fix week. We compress oversized imagery, defer slow scripts, make add-to-cart high contrast, cut clutter above it on mobile, add missing product education and put trust proof near decision points.

We do not move on to offer and cart work until the first mobile view, CTA visibility, page speed and basic product education stop being obvious blockers. These are the items that fill a growing backlog. Ship them and the backlog shrinks.

But not everything is a bug. Some changes are bets.

Which Changes Deserve an A/B Test?

Judgement calls need proof.

Run an A/B test when the current page works and you are changing a judgement call. WebMedic's rule names 6 test triggers: offer framing, price or discount display, urgency, the default purchase option, the hero product visual and a prominent upsell. Each can lift conversion or hurt it.

#	Test trigger	Why you cannot just ship it
1	Changing offer framing	The same deal can read as a bargain or as a gimmick
2	Changing price display, bundle economics or discount framing	A bundle price can feel like savings or like a bigger bill
3	Introducing urgency	Urgency can push a buyer or erode trust
4	Changing the default purchase option	Pre-selecting subscribe-and-save can help or backfire
5	Changing the hero product visual	Lifestyle shot versus product on white is taste until you test it
6	Adding a prominent upsell	It can lift order value and still cut conversion rate

Source: WebMedic fix-versus-test rule.

The current page works in every one of these cases. You are betting that a different version works better. That bet earns a slot.

Write the bet down before it starts. One row per test: the leak, the hypothesis, the change, the primary metric, the guardrail metric and the traffic segment. No row, no slot.

You will still have more ideas than slots. Score them with the A/B test prioritization framework before they enter the queue. Need ideas? See 49 things to A/B test on your Shopify store. Ready to run one? Start with the ecommerce A/B testing guide.

Does this sound like your store? Find out where you're leaking revenue. Take the free Revenue Score. 3 minutes. Free. No pitch.

How Do You Decide When to A/B Test vs Just Fix?

One question sorts everything.

Ask one question: is the current page objectively broken, or is it functional and you are changing a judgement call? Broken ships as a fix. Functional earns a test. WebMedic applies this split to every backlog item, using 13 named triggers across the two lists above.

Say it out loud in the weekly meeting. "Is this broken, or are we betting?"

Two quick checks back it up:

• Can you point to the defect? A hidden button, a missing size chart, a shipping cost that appears on the last step. Ship it.
• Can you name two reasonable versions? If a smart colleague could argue for either one, test it.

Try it on three items. A missing size guide is a fix. A countdown timer is a test. A new bundle price is a test.

If a shopper on a phone would call the page broken, fix it. If they would call it fine and you are hoping for better, test it.

This is the split we run inside our conversion rate optimization programs for Shopify brands. The weekly meeting where these calls get made is laid out in our CRO operating cadence.

What About Changes That Are Both a Fix and a Judgement Call?

Grey cases are common.

When a change is both, default to the fix. Ship the objective repair now, then test the refined version later. An element test takes 2 weeks on WebMedic's cadence, so waiting to test a repair first leaves a leaking page live for at least 14 days.

Here is an example. Your add-to-cart button is invisible on mobile. Making it visible is a fix.

Deciding whether it should be a sticky bar or an inline button in a new color is a judgement call.

Ship the visible button this week. Then test sticky versus inline against the repaired page.

The fix also gives you a cleaner control. Test against a broken page and you only learn how much better than broken you are. Test against a working page and you learn what is best.

Why Is Testing a Bug a Waste of a Test Slot?

Testing a bug teaches you nothing new.

Testing a bug teaches you one thing: the bug was a bug. In a standard 50/50 split, half your shoppers keep seeing the broken version for the whole test. On WebMedic's 25-day page cadence, that is 25 days of paid traffic hitting a defect you already knew about.

The cost stacks up three ways:

1. The slot. You burn one of your 1 to 2 monthly slots on that page.
2. The revenue. Half your visitors see the defect while the test runs.
3. The trust. When "variant wins" comes back, the team learns that testing confirms the obvious. Then people stop taking results seriously.

A fix is also easier to defend. A test asks the team to wait. A fix asks the team to look at the page.

What Should Every Change Pass Before It Goes Live?

Fixes are not exempt from discipline.

Every change, fix or test, passes WebMedic's 11-item pre-launch check before it goes live: one funnel stage, one primary metric, one guardrail metric, then mobile, desktop, add-to-cart, checkout, discounts, tracking, popups and CTA position. Shipping without a test does not mean shipping without QA.

Score every item before launch:

• The owner has named the funnel stage being improved.
• The owner has named one primary metric.
• The owner has named one guardrail metric.
• Mobile view has been checked.
• Desktop view has been checked.
• Add-to-cart still works.
• Checkout still works.
• Discounts still apply correctly.
• Tracking still fires where relevant.
• No popup blocks checkout intent.
• No new section pushes the primary CTA materially farther down on mobile.

Notice the last item. A new section that pushes the button down the mobile screen is a fix that breaks another fix.

The guardrail matters as much as the target. If your primary metric is add-to-cart rate, watch conversion rate and average order value too. A change that lifts one and drops the other is not a win.

After launch, check orders, conversion rate, checkout errors and support complaints daily while the change is fresh. Then decide weekly: keep, iterate, roll back or expand.

For the wider picture, see our Shopify conversion optimization breakdown.

Frequently Asked Questions

When should you A/B test vs just fix a problem?

Fix anything objectively broken or unclear, like a hidden add-to-cart button or oversized images. Test judgement calls like offer framing, price display or urgency. WebMedic's rule uses 7 fix triggers and 6 test triggers. The deciding question is whether the current page is broken or merely functional.

Do you need to A/B test every change on your Shopify store?

No. A page runs one test at a time, so each revenue page gets 1 to 2 test slots a month on WebMedic's cadence. Spending them on obvious bugs wastes them. Ship objective fixes directly, run the 11-item pre-launch check on each, and save tests for judgement calls.

Should you A/B test a slow-loading page?

No. Compress the images and trim heavy scripts first. Google's Core Web Vitals rate a Largest Contentful Paint of 2.5 seconds or less as good, so a slower page has a measurable defect, not a hypothesis. Save your test slot for a genuine judgement call.

What if a change is both a fix and a judgement call?

Default to the fix. Ship the objective repair now, such as making a hidden add-to-cart button visible, then test the refined version later, such as a sticky bar versus an inline button. An element test takes 2 weeks on WebMedic's cadence, so waiting costs at least 14 days of sales.

Keep Reading

• A/B Test Prioritization Framework: Score Before You Test
• Conversion Rate Optimization Strategy: A Weekly Cadence
• The Shopify Founder's Backlog: Why It Keeps Growing and How to Clear It