A/B Test Sample Size & Duration Planner

Define the decision and the people it concerns

Start with the decision that a future comparison should inform. You might be choosing between two landing-page explanations for a campaign, with completed registration as the shared outcome. State what would change if the alternative performed meaningfully better, then identify the people eligible to enter that comparison. The planner estimates sample requirements for two independent groups of people, each contributing one yes-or-no outcome under the same definition. It does not estimate the audience needed for a general awareness campaign, the views needed to make a video popular, or the number of posts required to discover a reliable creative formula.

Give the plan a recognizable Title, describe the Outcome, and write an Eligibility rule that someone else could apply before assignment. A useful rule might include new visitors from a named campaign who can complete the same registration process, with each eligible person assigned once. Explain any exclusions before the experiment begins rather than removing inconvenient participants after seeing results. Random assignment, non-overlapping groups, and reliable identity handling must be implemented elsewhere. Selecting the design declaration records your intended assumptions; it neither audits those arrangements nor turns ordinary campaign totals into an experiment. An undeclared plan remains a draft estimate with an explicit assumptions warning.

Choose Instagram, TikTok, YouTube, Facebook, LinkedIn, Threads, X, Snapchat, or Reddit only to identify the campaign context. The same statistical model applies to all nine labels. A visitor arriving from a social platform can enter an externally randomized destination test, but that does not make the originating posts randomized. If assignment occurs only after arrival, the test concerns the destination experience, not the causal effect of the earlier creative on clicks. Keep that boundary visible in the title and notes. Teams often gain more from answering a narrow question honestly than from attaching a broad platform-performance claim to unrelated observations.

Translate the baseline and improvement into rates

Baseline rate is the expected proportion of eligible people in control group A who will complete the defined outcome. Use a comparable historical cohort or a documented pilot with the same eligibility, outcome window, and measurement rules. A purchase rate among product-page visitors is not automatically the baseline for all people exposed to a social ad. Historical evidence can be uncertain, so record its period and scope in your supporting records and compare several plausible baseline assumptions in separate plans. The planner accepts your supplied rate; it does not look up an industry benchmark, estimate account quality, or verify that your historical denominator matches the future experiment.

Minimum detectable improvement describes the positive difference you want the plan to have the selected power to detect. In Relative lift mode, the variant rate equals baseline multiplied by one plus the entered lift divided by one hundred. A five percent baseline with twenty percent relative lift therefore becomes six percent, not twenty-five percent. In Percentage points mode, the entered amount is added directly to the baseline percentage. Adding one percentage point to five percent also gives six percent. Inspect the displayed target rate and both effect expressions after choosing a mode, because retaining the same numeric entry while changing modes changes its meaning.

Choose the smallest improvement worth making a decision about, then examine whether the traffic requirement is feasible. Picking a large effect simply because it returns a short test does not create that effect or make smaller useful improvements unimportant. The supported planning direction is an increase: the target must be greater than the baseline and remain below one hundred percent. This is not a reduction, equivalence, or noninferiority planner. If fewer failures is your business objective, do not casually reverse the labels without defining an appropriate binary success outcome and reviewing whether the resulting experiment answers the original question. Keep the statistical target tied to the operational decision.

Read confidence, power, and approximation correctly

Confidence selects the complement of the test significance level: ninety-five percent confidence corresponds to alpha of five percent in this two-sided planning model. It is not the probability that B will win, the chance that your campaign succeeds, or a guarantee that a particular result is correct. Power is different. An eighty percent power target asks for a design that would reject equal rates with about that probability under the specified alternative and model assumptions. Real outcomes still vary. Neither setting supplies evidence that the assumed improvement exists; both settings describe how the planned statistical procedure should behave across hypothetical repetitions.

The calculation uses a normal approximation for a comparison of two independent proportions, pooling the planned rates under the null and using separate variances under the alternative. It accounts for both rejection tails and does not apply a continuity correction. The solver finds a continuous control-arm requirement at the requested allocation, rounds the arm counts upward, and checks approximate power again using their actual rounded ratio. Achieved power reports that last calculation. It is not observed power from collected campaign results, and the returned integer pair is not a claim to be the globally smallest possible allocation among every combination of whole-number arm sizes.

Higher confidence or higher power generally requires more people when the other assumptions stay fixed. Choose these settings deliberately before recruitment and keep them consistent with the analysis you intend to use later. A plan for independent binary outcomes does not cover mean revenue, average watch duration, repeated view counts, paired observations, or assignment by creator, household, school, or community. Those designs have different uncertainty structures. Likewise, adding several variants or testing many outcomes introduces questions not handled by this two-arm calculation. A precise-looking sample count cannot compensate for a mismatch between the model and the unit that was actually randomized and measured.

Reproduce the fictional signup example

Load example uses a five percent baseline and twenty percent Relative lift, producing a six percent target for B and an absolute improvement of one percentage point. Confidence is ninety-five percent, power is eighty percent, and Variant B traffic share is fifty percent. Daily eligible people is five hundred across both arms combined. Minimum enrollment is seven days and outcome maturity is seven days, with a start date of November 11, 2026. The project describes new eligible visitors assigned once to A or B and a completed-signup outcome. These are fictional planning assumptions, not observed conversion rates or recommended benchmarks for any platform.

For those exact inputs, the planner returns 8,158 people in A and 8,158 in B, totaling 16,316. Approximate achieved power is 80.0014 percent after rounding. With five hundred combined eligible people arriving every calendar day at an even split, each arm receives a modeled two hundred fifty people per day. Both arm targets require thirty-three enrollment days, so the seven-day minimum does not extend this example. Starting on November 11 makes December 13, 2026 the inclusive enrollment end. Adding seven maturity days gives December 20 as the review date and forty total calendar days when the start day counts as day one.

Interpret the answer as a requirement under a scenario, not a prediction that 16,316 people will arrive or that the variant will achieve six percent. The expected success counts are fractional planning quantities because they multiply assumed rates by target people; they are not promised registrations. If your actual eligible traffic is lower, recruitment takes longer. If your outcome needs more time to mature, review must wait longer even after enrollment is complete. Save this example, replace the context and assumptions with your own, and build the plan again. Every input edit invalidates the old results so a previously calculated count should not be reused for a changed question.

Let allocation and the slower arm determine duration

Variant B traffic share allocates the combined stream of eligible people between the two groups. Fifty percent means an even split, while forty percent gives B four out of every ten eligible people in the modeled assignment. This setting affects both the sample calculation and the speed at which each arm reaches its own target. It is not a budget split, an impression share, or a setting that the planner applies to an advertising account. If you need unequal exposure for an operational reason, compare the resulting arm requirements instead of assuming that the same total sample remains sufficient under every allocation.

Duration uses the larger of A target divided by daily A traffic and B target divided by daily B traffic, rounding upward to whole days. Dividing only the combined target by combined traffic can conceal an arm that fills more slowly. Enrollment days then becomes the larger of that traffic-based requirement and your Minimum enrollment days. The estimate assumes the entered number of unique eligible people arrives every calendar day at the requested allocation. It does not model weekends, launch spikes, auctions, audience saturation, or changing campaign spend. If daily traffic is blank or zero, sample requirements remain available but duration and calendar dates are unavailable.

Outcome maturity days is a waiting period after enrollment, not extra recruitment and not additional sample. Use a window long enough for the latest assigned people to receive the same opportunity to complete the outcome as earlier entrants, with any agreed processing delay considered separately in your operational plan. The calendar uses an inclusive enrollment end and adds maturity days to obtain the review date. A zero maturity entry is allowed but needs a defensible immediate-outcome definition. Calendar outputs beyond the supported year range are withheld while integer day counts remain available. Read the duration and traffic guide when the schedule, rather than the statistical settings, is the binding constraint.

Use sensitivity to expose fragile assumptions

The sensitivity comparison holds the rest of the plan fixed and evaluates half, the same, one-and-a-half times, and twice your entered improvement. These are alternate planning effects, not forecasts of likely campaign performance or four experiments to run together. In the fictional signup example, halving the relative improvement from twenty to ten percent changes the target from six to five-and-a-half percent and requires 31,234 people per arm. That is 62,468 people combined, with 125 enrollment days and 132 days including maturity under the same traffic assumption. A modest-looking difference in target rates can therefore imply a very different recruitment commitment.

Use the comparison to ask whether your decision would still be useful if the true improvement were smaller than hoped. If only the most optimistic row fits your calendar, the limitation belongs in the planning discussion. You might choose a more substantial intervention, improve eligible recruitment without changing the population, or decide that a confirmatory experiment is not currently practical. Do not call the largest row the recommended effect merely because it is affordable. The minimum detectable effect guide explains how to connect that choice to a smallest useful improvement rather than a desired headline.

Unavailable scenarios remain unavailable. A scaled effect that reaches or exceeds one hundred percent is not a feasible target, and an arm requirement beyond the planning limit is not replaced by a capped number labeled sufficient. The main plan can still be valid when a sensitivity row is unavailable. Also inspect the expected outcome and non-outcome cells. If any planned cell is below ten, a warning highlights that the normal approximation may be weak. Passing that threshold does not certify the experiment, and failing it does not prove that no valid method exists. Sparse outcomes require a method review instead of uncritical reliance on a reassuring status label.

Keep an editable plan and a careful handoff

Save project downloads the editable JSON settings needed to reopen the plan. Open project accepts that project format rather than a table of historical observations. This tool has no raw-results CSV importer because it plans from a small collection of assumptions, not participant records. Keep the project file when you need to revisit the exact baseline, effect mode, allocation, dates, and notes. A report CSV is a record of the calculation, not a substitute input template. Loading a saved project restores supplied assumptions; it does not reconnect a campaign, recover a tracking system, or verify that the assumptions still describe your current audience.

Copy results, Download CSV, and Download PDF provide ways to review the plan with colleagues. They include the supplied title, outcome, eligibility, and notes, so treat the text fields as potentially sensitive. The PDF also contains the project JSON and report CSV as attachments. A report that looks concise on its visible pages can therefore carry the full plan in attached files. This is not a summary-only or redacted export. Before sharing, remove private campaign names, identifying context, or confidential notes from a deliberately prepared copy, then rebuild and inspect the resulting report and its attachments. Keep the original working version separately when needed.

Calculations and local file actions happen in your browser without a social login or a campaign-data upload workflow. The planner does not autosave, publish, send reports, or monitor future enrollment. Save deliberately before closing the page, and retain a dated version when assumptions change. In a team handoff, identify the owner of random assignment, the owner of identity and outcome tracking, and the person authorized to decide whether the recruitment horizon remains feasible. The exported plan supports that conversation, but it is not an instruction that automatically configures another product. The distinction makes the document useful without allowing a file download to masquerade as an implemented experiment.

Separate planning, execution, and final interpretation

Before launch, compare this plan with the actual experimental procedure. Confirm that assignment happens before the experience being compared, that people cannot enter both arms, and that outcome measurement works the same way in both groups. Define how interrupted enrollment, tracking failures, and delayed outcomes will be handled before looking at a performance difference. A minimum calendar period can help cover an intended operating cycle, but it does not remove seasonality or prove representative traffic. If the assignment unit is a group or people repeatedly contribute observations, stop and use a design that accounts for that dependence rather than inflating the apparent sample with repeated events.

A fixed-horizon sample calculation is not a sequential stopping rule. Checking performance every day and stopping as soon as a result looks favorable changes the statistical procedure. Operational monitoring for broken pages or invalid tracking remains important, but it should not be confused with repeatedly declaring a winner. Specify the intended stopping and analysis rules in advance, and obtain a suitable method when interim decisions are essential. Once the planned comparison has finished and outcomes have matured, the separate A/B Test Results Analyzer can review supplied binary-outcome counts within its own scope. It does not retrospectively validate how the people were assigned.

Use the planning guides to adapt the reasoning to a particular campaign while keeping the common statistical requirements intact. Return to all free tools for adjacent preparation and review tasks. When the next need is routine account operation rather than experimental inference, SMTasker offers supported publishing and account workflows, with Skye and permissioned MCP assistance for available workspace actions. Those capabilities do not randomize this experiment, import its plan directly, or certify a causal result. Keep the experiment protocol and collected evidence in their appropriate systems, then use the resulting decision to guide the social work that your team is actually equipped to perform.

Questions about A/B test planning