A/B Test Results Analyzer

Define one outcome and the people who could achieve it

Choose an outcome that each eligible person can either achieve or not achieve during the stated window. A completed registration, a qualifying enquiry or a first download can fit this structure when you can identify unique people consistently. A person who downloads three times is still one successful person for a first-download outcome. Decide the definition before comparing variants so the meaning does not move toward whichever version looks stronger.

The denominator is the number of eligible people in that arm, not whichever traffic total is easiest to find. If the question concerns registration among assigned visitors, include assigned visitors under the same eligibility rule in both groups. Do not quietly replace that denominator with people who reached the final form. Reaching the form might itself be affected by the variant, so selecting only those people changes the question.

Write the success rule in ordinary language: one eligible person who completed the workshop registration within seven days of assignment, for example. Then state whether existing registrants, staff and test records were excluded, using rules set independently of the results. The analyzer receives aggregate counts, not the underlying identity records. Your own preparation process must establish that the successful people belong to the eligible group and that no person is counted twice.

Some useful outcomes do not belong in this calculator. Average watch time, purchase amounts and total comments are not unique binary counts. A person may generate many comments, and two average durations do not tell you how many people met a common condition. Keep those measurements in an appropriate analysis rather than treating seconds, money or interactions as people. A clean definition is more valuable than a formally precise answer to the wrong question.

Prepare comparable records before entering the totals

Create a small reconciliation in your own records: arm label, eligible people, successful people, outcome definition and observation window. Retain the source of the totals where your team normally documents its work. If one record system counts accounts and another counts devices, resolve that mismatch first. The tool does not deduplicate identities, connect to social accounts or interpret a native analytics export on your behalf.

A larger success count does not necessarily mean a higher success rate. Suppose A records 60 successes among 600 people and B records 81 among 900. A is 10%; B is 9%. The B minus A difference is -1 percentage point, and B’s relative change against A is -10%. B has 21 more successes because its group is larger, while its observed success rate is lower.

Check the observation window as carefully as the counts. Comparing seven completed days of follow-up for A with one day for B can make delayed registrations look like a variant effect. Use cohorts with equally mature follow-up under the same rule. When an outcome is not yet known, leave the comparison unfinished rather than entering zero to make the calculation run. Unknown and unsuccessful are different states in the underlying evidence.

Keep the project notes concise and useful: identify a delayed data refresh, an eligibility exclusion or a known measurement interruption. Do not paste names, email addresses or row-level customer records into an aggregate review. Choosing a platform context only labels the project; it does not retrieve data, alter the formulas or certify that the numbers match a platform’s reporting definitions. Prepared counts remain your responsibility in every context.

Read the worked example in three complementary ways

Consider a fictional registration comparison. A has 100 successful people out of 1,000 eligible people; B has 130 out of 1,000. Divide successes by eligible people: 100 / 1,000 = 0.10 and 130 / 1,000 = 0.13. Expressed as percentages, the observed rates are 10% and 13%. These describe the supplied groups without needing a claim about random assignment.

Subtract A’s rate from B’s rate to find the absolute difference: 13% – 10% = +3 percentage points. A percentage point is a difference between percentages, not a relative percentage increase. Here B records three additional successful people per hundred eligible people in the observed groups. With equal groups of 1,000, that corresponds to the observed count difference of 30 successful people.

Relative lift uses A as the reference: (13% – 10%) / 10% = 30%. B’s observed rate is therefore 30% higher than A’s, not 30 percentage points higher. Both +3 percentage points and +30% relative lift are correct, but they answer different questions. Report the starting rate with the relative figure so a colleague does not imagine that the registration rate jumped from 10% to 40%.

The non-success counts are also important: A has 900 and B has 870. This example passes the numerical minimum for displaying inference, but the counts alone cannot establish the experiment’s design. A historical campaign and a later campaign could produce exactly these totals without being randomized. Keep the arithmetic unchanged while changing the strength of the interpretation to match the evidence. No unverified p-value or confidence bounds are needed to explain these observed differences.

Use descriptive mode unless the experiment supports inference

Descriptive mode is the default because a pair of rates does not reveal how the groups were created. It shows the observed rates and lift without inferential intervals or a p-value. Use it for appropriately prepared organic comparisons, operational cohorts and other nonrandomized records. Descriptive does not mean useless: it can identify a promising question, reveal a denominator problem or document why a more controlled comparison would be worthwhile.

Randomized mode requires all four explicit design confirmations. Randomized means eligible people were assigned to the alternatives by a genuine random process. Independent means the observations are suitable as separate people in separate arms, without paired outcomes, overlapping membership or unaccounted clustering. Fixed horizon means the stopping rule was set before examining the emerging result. Common outcome means both arms use the same eligibility, success definition and observation opportunity.

There is also a numerical guard: each arm needs at least 10 successes and at least 10 non-successes. The requirement is not 10 people per arm, 10 successes across both arms or a large combined audience. For example, 12 successes among 20 people leaves only eight non-successes, so that arm does not qualify. All four confirmations and every per-arm count check must pass together before inferential outputs appear.

These checks are a conservative display guard, not proof that every assumption holds or that the approximation is accurate in every case. Confirmations are declarations about work done outside the calculator. A checkbox cannot make a convenience audience random, and enough conversions cannot remove dependence between observations. For the full design distinction, use the organic comparison versus randomized test guide before attaching experimental language to the review.

Interpret the fixed statistical methods without a winner claim

When eligible, the analyzer uses a two-sided pooled two-proportion z-test at a fixed 95% confidence level, equivalent to a 0.05 significance threshold. It compares the equal-rate hypothesis with a difference in either direction. The pooled rate combines successful people across the two arms for the test’s uncertainty calculation. The method uses no continuity correction. It does not switch to a one-sided test because B happens to be ahead.

A p-value describes how unusual a result at least as extreme as the observed one would be under the equal-rate hypothesis and the model assumptions. It is not the probability that B is the winner, that the null hypothesis is true, or that the result happened by chance. A small value cannot compensate for an invalid denominator or a nonrandom design. A large value does not establish that the alternatives are equivalent.

The individual rate intervals use the Wilson 95% method. The difference interval uses the Newcombe method combining the two independent Wilson intervals, without continuity correction, and always describes B minus A. Read the difference interval directly rather than deciding significance by whether the separate arm intervals overlap. Confidence refers to the long-run behavior of the interval procedure under its assumptions, not a 95% probability assigned to this particular variant.

The p-value and difference interval come from different methods and can disagree near a boundary. Keep both results visible and explain the uncertainty instead of rounding until they appear to agree. Threshold decisions use unrounded values. The analyzer does not provide a winner badge, equivalence test, sequential monitoring adjustment or correction for testing many alternatives. Its result is one qualified comparison, not permission to search indefinitely for a favorable threshold crossing.

Ask whether the observed lift changes a real decision

Return to the 10% versus 13% registration example. The +3 percentage-point difference means 30 additional observed registrations per 1,000 eligible people when expressed on that common scale. Whether that matters depends on the next decision. A workshop with only five places remaining has a different operational problem from an event with substantial unused capacity. The same measured lift can be useful, inconvenient or too uncertain for the proposed commitment.

Translate a rate difference into workload without presenting it as a forecast. If each additional registration requires six minutes of preparation, 30 additional registrations would require 180 minutes, or three hours, under that stated assumption. This is scenario arithmetic, not evidence that the next 1,000 people will produce 30 additional registrations. The observed audience, future demand and available capacity may differ. Keep those assumptions beside the scenario wherever it is shared.

Check outcome quality separately. A higher registration rate can coexist with more cancellations or lower attendance. Those are distinct outcomes with their own eligible cohorts and follow-up windows; they are not numbers to fold into the original success count after seeing the result. Decide what evidence you need for adoption, including any operational guardrails. The significance and practical lift guide explains how to keep uncertainty and usefulness in the same review.

A reasonable next step can be limited and reversible. You might retain the current version while checking an unresolved measurement issue, or prepare an independent confirmation before a costly change. That decision should reflect the downside of being wrong, not just which rate is larger. This analyzer does not estimate required sample size, prescribe a stopping target or guarantee a winner. Make the operational decision explicitly rather than expecting a statistical label to make it for you.

Separate social performance observations from controlled effects

Organic posts often reach different people under different circumstances. Timing, topic, audience interest and distribution can all change together with the creative. A later post with a higher action rate may be worth studying, but the difference does not isolate the creative’s effect. Calling the posts A and B is a useful labeling convention, not a randomization mechanism. The evidence remains observational unless the actual assignment process supports a controlled comparison.

Native engagement totals also need care. Impressions can include repeated exposure, views can count repeat activity, and one person may like, comment and share. Adding these events does not create a count of unique successful people. Even a reach figure should not be paired automatically with an unrelated outcome total. Establish a common person-level population in your own records before using the tool’s binary rate calculation. Otherwise keep the native metrics in a descriptive report suited to them.

A separate landing-page experiment can answer a narrower question. For example, eligible visitors arriving from a campaign could be randomly assigned to two registration messages using a suitable system outside this tool. The resulting review concerns those assigned visitors and those messages, not every person who saw the social post. Document where assignment occurred so an improvement after the click is not reported as proof that one public post caused more attention.

Platform context can help your team recognize the campaign associated with a project, but all contexts use the same mathematics. It does not connect Instagram, TikTok, YouTube or another account, and it does not reproduce a native experiment interface. Keep a clear boundary between collecting evidence, analyzing prepared aggregates and operating social routines. Browse the free tools directory when another preparation or reporting task needs a different kind of input.

Preserve a review that another person can actually audit

A useful handoff includes both arm labels, the eligible and successful counts, the common outcome, the follow-up window and any material limitations. State the observed rates, B minus A in percentage points, and relative lift against A when defined. Include whether statistical inference was available and why. This lets a colleague distinguish a calculation error from a disagreement about experimental design without reconstructing the original conversation.

Use Copy results for a complete text summary, CSV for recorded fields and calculated values, JSON for an editable project, or PDF for a styled review. The PDF is rasterized; CSV and JSON preserve selectable data values. Exports carry the supplied information and method context rather than replacing your source records. Keep the experiment protocol and any detailed reconciliation in their normal location, because a compact aggregate project cannot contain the full evidence trail.

The comparison runs locally in the browser without social login, experiment execution, audience allocation or tracking of participants. There is no autosave, so download the project if you need to reopen it. Ordinary site analytics are separate from the private counts and notes, which are not submitted for the calculation. The tool’s JSON project is its own format, not a promise that native platform reporting files or saved projects can be imported into SMTasker.

Once the review is complete, turn its decision into a clearly owned task. SMTasker supports nine social platforms through Android phones with a local Windows or macOS client. Choose from your selected platform’s supported publishing or engagement actions, then set schedules and limits for the approved campaign work. This gives the team a repeatable account routine after the decision is made. Those routines remain separate from random assignment and statistical analysis; SMTasker does not run this experiment or import the analysis project.

Questions about A/B test results