TikTok Split Test Planning for Qualified Outcomes

The A/B Test Sample Size & Duration Planner models supplied assumptions in your browser. It does not create TikTok Ads Manager tests, allocate audiences, or predict video performance.

Choose the decision behind the creative contrast

A video can attract attention without producing the downstream action that makes a campaign useful. Before asking how much traffic a split test needs, state the decision in operational terms. Are you selecting an introduction that helps suitable people request a workshop place, or are you studying how long people watch? Those questions need different outcomes and may need different models. This planner addresses one shared binary result per independent participant. It does not convert average viewing time, replay counts, or a mixture of engagement actions into a conversion experiment simply because the campaign began on TikTok.

The fictional Benchcraft workshop teaches basic furniture repair. Its team wants to compare two opening videos on an external destination page: a demonstration of a simple repair step and a reveal of a finished restored chair. Both versions introduce the same workshop, price, prerequisites, and request form. New eligible TikTok-referred people are assigned before either destination video appears. The decision is whether to use one of those destination introductions for future arrivals. It is not a test of which TikTok feed video generates more clicks, because the randomized difference occurs after people have already chosen to arrive.

Define a qualified workshop request as the primary binary outcome. In this example, a participant must complete the same prerequisites check and confirm a request within three days of assignment. The exact rule is fictional and should be replaced with a defensible rule for your own offer. An entertaining introduction might generate many low-intent form starts, so do not choose form activity merely because it produces a larger numerator. Make the event correspond to the decision the team is prepared to make. Keep any separate questions about attendance, refunds, or later revenue visible without claiming that this single-outcome plan has powered those analyses too.

Separate attention measures from conversion people

TikTok campaign reporting can contain several layers of activity: exposure, viewing, destination visits, form interaction, and a later qualified action. The same person can contribute to several layers and can repeat some actions. Adding those counts does not produce independent participants, and dividing a qualified outcome by whichever large number is easiest to find does not create a meaningful conversion rate. Identify the population at the point of assignment and keep it consistent from the historical baseline through the future analysis. The planner’s Daily eligible people field refers to that incoming population across both arms, not all people who encountered content upstream.

Benchcraft excludes existing workshop customers and people outside the intended service region before assignment, using an external eligibility process. It does not remove people later because they watched only a few seconds or failed to open the form. Watching and form opening happen after assignment and may be influenced by the introduction. Conditioning the denominator on those behaviors could distort the comparison. The external system must support stable assignment and a defensible identity rule; the calculator cannot deduplicate visitors or certify that browser identifiers represent unique people. If only session-level data is available, describe that limitation and review the model instead of relabeling sessions as people.

Attention metrics can still serve a useful descriptive role. A reviewer may inspect whether an introduction appears confusing or whether a technical failure prevents playback. But a binary threshold such as watched past a chosen point is a different primary outcome from qualified workshop request, and its baseline cannot be substituted without changing the research question. Mean watch duration is outside this planner’s scope altogether. Choose the outcome before seeing variant performance and preserve the same event window in both groups. Otherwise a convenient metric change can make the finished report sound stronger while disconnecting it from the decision the experiment was supposed to inform.

Match your assignment method to the planning model

TikTok Ads Manager has a native split-testing product with its own setup, metrics, and reporting method. TikTok describes splitting audiences into exclusive groups for those tests. That product is separate from this browser-only calculator. A native test may use a metric, allocation, or decision procedure that does not match a fixed-horizon comparison of person-level binary outcomes. Do not assume that entering a confidence setting here reproduces the native engine. Review the method of the actual experiment you intend to run, and use this planner only when its independent-person, two-arm assumptions are appropriate for the population and outcome being planned.

For the Benchcraft example, assignment is controlled externally at the destination before playback. A reliable implementation needs to keep returning participants in their original arm and avoid exposing them to both introductions during the measurement window. It also needs to record the common qualified request consistently. An ordinary campaign parameter can label a traffic source, but it is not a randomization mechanism or a unique-person identifier. The planner’s platform field provides context only. Its design declaration records the user’s intended conditions, so checking that box does not test the destination, inspect exposure logs, or certify that the deployed assignment worked as described.

If your intended contrast is instead between two videos shown inside the feed, the assignment point and denominator must move upstream accordingly. Randomizing the landing page cannot establish which feed creative caused more visits. Likewise, two separately posted videos may differ in distribution, timing, and audience before their performance is compared. That can support descriptive creative learning, but it does not automatically satisfy this model. Keep a short design diagram in the working brief: where eligibility is decided, where assignment occurs, where exposure happens, and where the outcome is measured. A small diagram often reveals a mismatch that a detailed table of sample settings would otherwise obscure.

Calculate the Benchcraft workshop requirement

Use the following fictional settings to reproduce the worked plan. Enter a three percent baseline, choose Percentage points, and enter an improvement of 0.60 points. The resulting target is 3.60 percent, equivalent to twenty percent relative lift. Set confidence to ninety-five percent, power to eighty percent, and Variant B traffic share to forty percent. Supply four hundred combined Daily eligible people, fourteen Minimum enrollment days, and three Outcome maturity days. Start date is November 16, 2026. The project title identifies the workshop destination-video introduction, and the outcome and eligibility fields describe qualified requests and new TikTok-referred people assigned once before exposure.

The computed targets are 17,269 people in A and 11,513 in B, totaling 28,782. Approximate achieved power is 80.0013 percent after the rounded arm sizes are checked. At the entered allocation, the modeled daily stream provides two hundred forty eligible people to A and one hundred sixty to B. Each arm’s requirement must be satisfied, which gives seventy-two traffic-based enrollment days. This is longer than the fourteen-day minimum. Enrollment beginning November 16 therefore ends January 26, 2027, and the three-day maturity period makes January 29 the review date. The complete modeled schedule is seventy-five calendar days.

Those dates are not a prediction of campaign delivery. They follow directly from a constant daily people assumption, fixed allocation, and the selected effect. A workshop promotion with only a few weeks of useful recruitment may not support this plan even if its videos can receive many views. The useful result is the mismatch between the needed qualified comparison and the available operating window. Review whether the same offer, eligibility, and outcome can remain stable through the modeled enrollment period. If they cannot, do not present the calculated count as a feasible launch commitment. Revisit the intervention, participant supply, or type of evidence the team can realistically obtain.

Translate traffic feasibility into an honest budget discussion

A campaign budget buys delivery opportunities; it does not directly buy a guaranteed number of independent eligible participants. Some exposed people will not visit, some visitors will be ineligible, and some apparent new visits may belong to someone already assigned. Benchcraft’s four hundred daily people is therefore a supplied downstream recruitment assumption, not a rate inferred from spend or video views. Build that assumption from comparable measurements and retain its uncertainty. If the only available evidence is a broad click forecast, review how much of that forecast could plausibly meet the actual preassignment eligibility rule before turning it into a calendar estimate.

The planner does not ask for a media budget or cost per participant and does not return a financial forecast. Keep a separate budget worksheet that distinguishes media, creative production, tracking setup, verification effort, and workshop capacity. A verified request can create staff work before anyone attends. A plan that is affordable in media terms may still be impractical if the team cannot process requests consistently throughout the experiment. Conversely, reducing the outcome to a cheap attention event solely to improve sample feasibility changes the question. Record that trade-off explicitly rather than describing a newly affordable metric as if it still measured qualified demand.

Unequal allocation is another commitment to examine. Giving B forty percent of eligible arrivals is not the same as assigning forty percent of ad spend to a campaign, and the sample requirement is recomputed for the entered people split. The arm with the tighter time-to-target requirement governs enrollment, so a combined total divided by combined traffic is not the full duration method. If operational constraints justify limiting B exposure, preserve that reason in the notes and compare alternate allocations in separately saved plans before launch. Do not alter the split midstream in response to favorable performance and assume that the original fixed-allocation planning calculation still describes the experiment.

Choose a useful effect instead of an affordable headline

The entered 0.60 percentage-point improvement is a planning threshold, not a prediction that the finished-chair introduction will achieve 3.60 percent. Benchcraft should decide whether that change would justify replacing the control experience and maintaining the new creative. If a smaller gain would still matter, the primary plan may not be sensitive enough for the actual decision. If a much larger gain is the only outcome worth acting on, that larger effect can be a legitimate planning choice, but the justification should come from the decision rather than the need to fit an attractive launch date. Power describes detection under an assumed effect, not the likelihood that creative produces it.

The sensitivity rows show what changes when the effect changes while the remaining assumptions stay fixed. At twice the original improvement, the target becomes 4.20 percent, a gain of 1.20 percentage points or forty percent relative lift. The core returns 4,666 people in A and 3,111 in B, totaling 7,777. With the same daily traffic and allocation, that scenario needs twenty enrollment days and twenty-three days with maturity. Starting November 16 puts enrollment end on December 5 and review on December 8, 2026. The shorter schedule belongs to a different detectable improvement; it does not make the original smaller-effect question disappear.

A common planning error is to select that shorter row, call the test adequately powered, and later conclude that any smaller positive movement is unimportant. The design was not built to resolve every smaller difference. Read the minimum detectable effect guide before finalizing the threshold. If none of the useful effects fits the available traffic, consider a more substantial intervention or a clearly descriptive learning stage. A null result from a short, low-sensitivity comparison is not evidence of equivalence, and a visually exciting early rate difference is not a reason to replace the preplanned horizon with an improvised stopping rule.

Plan maturity and calendar stability together

The three-day outcome window starts from each person’s assignment, not from the date the campaign manager opens the report. People arriving near the end of enrollment need the same window as those arriving at the beginning. Outcome maturity days adds waiting time after enrollment so that the latest entrants can have that opportunity. It does not add participants, extend every person’s success deadline indefinitely, or guarantee that all verification systems have finished processing. If the team needs extra administrative time to reconcile qualified requests, record that in the operational protocol without silently changing which events fall inside the outcome definition.

Benchcraft’s primary calendar crosses into January, which raises practical questions beyond arithmetic. Can the same workshop offer remain open? Will staff availability and eligibility rules stay consistent? Could seasonal interest change sharply during the period? The planner assumes everyday traffic and does not forecast those conditions. A minimum enrollment setting also does not correct them. Use the returned dates to expose such dependencies before launch, then decide whether the planned population and intervention can remain meaningful. The duration and traffic guide provides a fuller treatment of the slower-arm calculation, minimum periods, and the distinction between enrollment and later review.

Monitor technical health without turning routine monitoring into repeated winner selection. If one video fails to load or qualification tracking breaks, the protocol needs a documented response. If the team simply checks rates every morning and ends the test when B first looks favorable, it has departed from the fixed-horizon procedure used in the power calculation. A sequential approach needs its own design. Decide in advance who reviews tracking problems, what conditions pause recruitment, and how affected observations are handled. Keeping those responsibilities explicit is more useful than treating the planned review date as an automatic permission to announce a result regardless of implementation quality.

Preserve the planning record and separate the next workflow

Save the editable project before changing assumptions or closing the page. The JSON retains the selected baseline, effect mode, allocation, timing, and supplied context so another reviewer can reproduce the calculation. It does not contain the destination videos, the randomization implementation, or participant observations. Open project restores that planning format; Download CSV is a report rather than an importer for TikTok exports. When a colleague proposes a different daily traffic number, keep the original and revised plans with a short reason for the change. This provides a record of the decision process without pretending that the calculator maintains an automatic experiment history.

Treat report sharing as a separate review step. Copy results, CSV, and PDF include the title, outcome, eligibility, and notes. The PDF also attaches the complete project JSON and report CSV, so removing a sensitive phrase only from an email does not remove it from the report files. Workshop locations, private launch names, or internal qualification details may need a cleaned planning copy before wider distribution. The browser-only tool does not send, publish, monitor, or autosave the plan. Downloaded files remain your team’s responsibility, and the presence of a local calculation does not make every later sharing destination private or appropriate.

Once the test is complete, use the actual mature observations with an analysis method matching the design. The separate A/B Test Results Analyzer reviews supplied binary counts within its own limits; it does not certify assignment or replace missing tracking evidence. For routine TikTok account operations after a considered decision, SMTasker provides supported publishing and account workflows, with Skye and permissioned MCP assistance for available actions. Those capabilities do not implement this statistical experiment or directly import its JSON plan. Return to the sample-size planner for another scenario or browse all free tools for a different preparation task.

Questions about TikTok Split Test Planning

Continue with another planning guide

Plan your two-arm binary-outcome test in the A/B Test Sample Size & Duration Planner, or choose another guide below. Find more practical resources in all free tools.