Minimum Detectable Effect for A/B Test Planning

Use this guide with the A/B Test Sample Size & Duration Planner to connect a positive binary-outcome improvement to a specified power, sample requirement, and traffic scenario.

Treat the effect as a design target, not a forecast

Minimum detectable effect is the improvement used to size a test at a chosen statistical power and significance level. It is not a forecast of how much the variant will improve performance. In this planner, you enter the effect along with a baseline and other assumptions, and the calculation estimates the people required for that target. The result asks what the procedure could detect with the selected probability if the specified alternative were true. It does not estimate whether that alternative is plausible, whether a campaign will attract participants, or whether a creative change is strong enough to produce the planned difference.

The word minimum can also mislead. A test does not become blind immediately below its planning effect or certain to detect every difference above it. Detection probability changes across possible true effects, and even at the chosen target it remains below certainty when power is below one hundred percent. A smaller difference can sometimes produce a statistically significant result, while a target-sized difference can sometimes fail to do so. Keep that variability in mind when describing the plan to colleagues. Say that the experiment is designed for a stated sensitivity under assumptions, rather than promising that it will find every commercially useful improvement.

Start with a practical threshold: what is the smallest improvement that would make the team seriously consider a different action? Then compare the sample and time required to study it. The smallest worthwhile effect and the effect your current traffic can detect may not coincide. That mismatch is a planning finding, not a reason to redefine value after seeing which input returns a short schedule. Write both concepts down when they differ. An experiment can be affordable but too insensitive for the decision, or sensitive enough in theory but impossible to complete while the offer and target population remain stable.

Convert relative lift and percentage points explicitly

Relative lift expresses the increase as a proportion of the baseline. If the starting rate is twelve percent and the selected lift is ten percent, multiply twelve by 1.10 to obtain a 13.20 percent target. The absolute difference is 1.20 percentage points. Percentage points instead describe direct subtraction between two percentage rates. Entering 1.20 in Percentage points mode produces the same target in this example. Entering ten in that mode would produce twenty-two percent, which is a very different alternative. Always read the derived target and both effect expressions before accepting the resulting sample count.

The same one-point improvement has different relative meanings at different baselines. Moving from four to five percent is twenty-five percent relative lift; moving from eight to nine percent is twelve-and-a-half percent relative lift. Neither description is more truthful, but each highlights a different scale. Relative language can make a small absolute movement sound dramatic when the baseline is low. Absolute language can obscure how substantial the change is compared with existing performance. Include both in a planning discussion so stakeholders agree on the actual target rate rather than nodding at the same word while imagining different numbers.

This planner preserves the numeric entry when you change effect modes, so the same typed number acquires a different meaning. Results are invalidated and need rebuilding after the change. A five percent baseline with twenty percent relative lift gives six percent, while adding twenty percentage points gives twenty-five percent. The target must stay above the baseline and below one hundred percent; an impossible target cannot be rescued by a larger sample. Reductions, equivalence, and noninferiority are outside the supported planning direction. Define the intended success outcome carefully instead of assuming that swapping labels automatically produces the right design for a different statistical question.

Connect the smallest useful effect to a real decision

Consider a fictional resource-library team deciding whether to replace an existing registration explanation with a clearer preview of what members receive. Its primary outcome is completed signup within a fixed observation window, once per eligible person. The team uses a twelve percent baseline and regards a 1.20 percentage-point improvement as worth investigating. Among ten thousand comparable eligible people, that rate difference represents one hundred twenty additional signups in expectation. This is an arithmetic interpretation of an assumed effect, not a prediction of delivered signups or a separate sample recommendation. Whether that improvement is useful depends on the team’s actual costs and objectives.

Account for the work required to implement and maintain the variant, the quality of the resulting signups, and the cost of a wrong decision. A registration improvement can be attractive while creating more support work or attracting people who are poorly matched to the resource. Those concerns can justify separate guardrails and a broader decision protocol. This calculator does not power every guardrail automatically and does not plan mean revenue or customer lifetime value. Use the binary signup target for the statistical question it actually represents, then evaluate economic assumptions elsewhere instead of treating a higher signup rate as a complete business case.

Agree on the decision threshold before exposing people. If the team would adopt a cheap wording improvement for a smaller gain than it would require from an expensive new workflow, those interventions may deserve different planning targets. Avoid borrowing the effect from another campaign merely because its percentage sounds familiar. The same numerical lift can have different value when fulfillment costs, participant quality, or implementation risk change. Record the reason for the chosen threshold in the notes without adding confidential financial detail unnecessarily. That explanation helps a later reviewer distinguish a purposeful sensitivity choice from a number selected only to make the calculator return an affordable test.

Reproduce the resource-library planning example

Enter a twelve percent Baseline rate and choose Relative lift of ten percent, giving a 13.20 percent target. Set confidence to ninety-five percent and power to eighty percent, with fifty percent of eligible people assigned to B. Supply eight hundred Daily eligible people combined, fourteen Minimum enrollment days, five Outcome maturity days, and a Start date of November 16, 2026. Use a completed-signup outcome and an eligibility rule describing new eligible people assigned once before the experience. A platform label can identify the campaign context, but it does not change this calculation or establish the required external randomization and identity arrangements.

The computed requirement is 12,004 people per arm, totaling 24,008. Approximate achieved power is 80.0011 percent at the rounded allocation. With four hundred modeled daily entrants in each arm, the traffic-based requirement is thirty-one days, which exceeds the fourteen-day minimum. Enrollment starting November 16 ends December 16, 2026. The additional five-day maturity period makes December 21 the review date and produces thirty-six total calendar days. These figures come from the stated planning inputs. They do not establish that the new explanation will produce a ten percent relative improvement or that eight hundred unique eligible people will arrive each day.

This example has two independent commitments: the statistical commitment to resolve a target-sized improvement with the chosen power, and the operational commitment to recruit and observe people long enough. A team with only a short promotion window cannot erase the first commitment by shortening the second. Conversely, a longer enrollment period does not make the assumed effect more believable. Save the complete plan when discussing feasibility so reviewers see the baseline, outcome, allocation, and timing together. A lone total sample figure is easy to reuse under a different denominator or outcome, which would make it look authoritative while stripping away the assumptions that gave it meaning.

Use sensitivity to see the price of finer resolution

The sensitivity comparison changes only the entered effect by fixed factors: one-half, one, one-and-a-half, and two. In the resource-library example, half the effect is five percent relative lift, giving a 12.60 percent target and a 0.60 percentage-point difference. The planner returns 47,036 people per arm, totaling 94,072. At the same eight hundred combined daily eligible people, enrollment takes 118 days and total time with maturity is 123 days. Starting November 16 puts enrollment end on March 13, 2027 and review on March 18. A small change in the target rate therefore creates a substantial change in the recruitment burden.

At one-and-a-half times the original effect, the target is 13.80 percent, equivalent to fifteen percent relative lift and 1.80 percentage points. That scenario requires 5,443 people per arm, totaling 10,886. At twice the effect, the target is 14.40 percent and the returned requirement is 3,122 per arm, or 6,244 combined. Both scenarios have nineteen total days under these settings because the fourteen-day minimum enrollment controls the schedule, followed by five maturity days. Their modeled enrollment end is November 29 and their review date is December 4, 2026. The equal calendar length does not mean their sample requirements are equal.

For modest effects under otherwise similar conditions, needing much more traffic to detect a smaller difference is the expected pattern, but do not replace the actual calculation with a universal exact multiplier. Planned rates, variances, allocation, whole-person rounding, and minimum calendar periods all matter. Compare the returned counts and dates directly. These rows are alternative design scenarios, not probabilities for which effect the intervention will produce and not a recommendation to run four tests simultaneously. If a scaled target reaches one hundred percent or the sample exceeds the planning limit, that row is unavailable rather than zero. Preserve that distinction when summarizing the comparison for a colleague.

Review baseline uncertainty before adjusting sensitivity

The baseline is an assumption about the same eligible people and event window that the future control will use. A historical rate among all visitors may be inappropriate for a test limited to new campaign arrivals, and a rate recorded after one day may not match a five-day signup outcome. Improving the wording of the effect cannot repair that denominator mismatch. Check the source population first. If the evidence is incomplete, save a small set of plausible baseline plans and explain why those values are being considered. The sensitivity table varies the effect, not the baseline, so separate baseline scenarios require separate calculations.

Holding relative lift fixed while changing the baseline also changes the absolute improvement. A ten percent relative lift means 0.60 percentage points at a six percent baseline but 1.20 points at a twelve percent baseline. If the business decision is really about a fixed absolute gain, Percentage points mode may make the comparison easier to explain. If the decision is proportional to existing performance, relative mode may be more natural. Neither mode is automatically superior. Choose the scale that reflects the actual decision and then inspect both displayed effect expressions so the alternate scale remains visible rather than hidden inside a convenient input convention.

Very rare outcomes can raise a separate approximation concern. The planner calculates expected successes and non-successes in each arm and warns when any planned cell is below ten. This is a diagnostic for a potentially weak normal approximation, not a certificate of validity above the threshold or a proof that no suitable sparse-outcome method exists below it. A rare qualified lead may need a different method review even when the traffic estimate appears affordable. Do not broaden the outcome to include unqualified actions simply to make the expected counts look healthier. A statistically convenient event that no longer answers the decision is not an improvement in experimental planning.

Keep power, multiplicity, and stopping rules distinct

Power and detectable effect belong together. Saying that a test has eighty percent power without naming the assumed effect leaves out the question that the power calculation answers. The same sample can have different detection probabilities for different true differences. Confidence also has a separate role: ninety-five percent confidence selects a two-sided significance level of five percent in this model, not a ninety-five percent probability that B is best. The method uses a normal approximation with pooled null variance and separate alternative variances, includes both rejection tails, and verifies power after rounding. Its precision should not be mistaken for certainty about the planned alternative.

Changing the number of variants or the number of claims changes the planning problem. A two-arm sample target does not automatically support several treatments, a collection of independently declared successes, or a search for whichever subgroup looks strongest. Reusing the same plan repeatedly until one comparison produces an attractive result also introduces issues beyond this single calculation. Establish a primary outcome and an appropriate broader analysis plan before launch. Additional diagnostics can be useful, but label exploratory findings as exploratory rather than presenting them as if the original sample target had guaranteed a reliable conclusion for every metric the team later decides to inspect.

Fixed-horizon planning also does not authorize significance-based early stopping. Looking at results every day and ending recruitment as soon as a favorable threshold appears changes the procedure. Operational checks for tracking or safety failures are different from repeatedly selecting a winner, but the distinction needs an advance rule. If interim decisions are essential, obtain a suitable sequential design rather than treating this calculator as one. Likewise, reaching the sample target does not eliminate the outcome-maturity requirement. The duration and traffic guide explains why the last entrant’s observation opportunity and the planned review date remain part of a defensible fixed-horizon schedule.

Document the threshold and interpret the eventual result

Save project preserves the editable settings needed to reproduce the planned effect and sample calculation. Include a concise explanation of why the target matters and where the baseline came from, while keeping personal or confidential information out of notes when possible. The report CSV is a calculation record, not a participant-data importer, and the project JSON is not a protocol executed automatically by another product. Copy, CSV, and PDF all include supplied context and notes; the PDF also attaches the project JSON and report CSV. Prepare a separate cleaned plan for wider sharing rather than assuming the visible report pages are a summary-only export.

When the experiment finishes, interpret its estimates and uncertainty using the actual observations and the agreed method. A nonsignificant result does not establish that the two experiences are equivalent or that every useful effect has been ruled out. A significant result does not by itself establish that the improvement is large enough to justify implementation. Compare the estimated difference and relevant uncertainty with the decision threshold, implementation quality, and outcome definition. The separate A/B Test Results Analyzer can review supplied binary counts within its scope, but it cannot recover randomization evidence or repair a stopping rule that changed after the results became visible.

Use the sample-size planner to revise future assumptions rather than retroactively claiming that an earlier design targeted a different effect. The Instagram guide and TikTok guide show how the same reasoning applies to distinct externally designed destination tests. For subsequent account work, SMTasker’s supported workflows and Skye assistance can help organize available workspace operations, not certify experimental conclusions or directly import this plan. Browse all free tools when the next need is a different planning or review task. Keep the evidence and the operational decision connected without confusing their roles.

Questions about Minimum Detectable Effect

Continue with another planning guide

Plan your two-arm binary-outcome test in the A/B Test Sample Size & Duration Planner, or choose another guide below. Find more practical resources in all free tools.