Statistical significance means a difference in your data is larger than random sampling error would plausibly produce by chance. In a survey, the usual question is whether two percentages really differ. Enter the two percentages and their sample sizes below. The statistical significance calculator runs a two-proportion z-test and reports the p-value.
By the Wavefield Research team · Published Nov 12, 2026
Use this when different people sit behind each percentage: men and women, customers and prospects, this wave and the last one.
Every survey interviews a sample, and a sample never matches the population exactly. Ask the same question of two random groups of 400 people drawn from the same city and the two results will differ by a few points for no reason other than luck. The statistical significance definition follows from that: a difference is statistically significant when it is too large to be comfortably explained by this sampling luck.
The test works by assuming the boring explanation first. Suppose the two groups truly agree (statisticians call this the null hypothesis). How often would samples of this size show a gap at least as wide as the one you observed? That frequency is the p-value. A p-value of 0.03 says a gap this wide would appear in about 3% of samples if nothing were going on. Small p-values make the boring explanation hard to keep.
The level of statistical significance is the cutoff you set in advance. The convention is 0.05, usually stated the other way round as 95% confidence: you accept that when no real difference exists, about one test in twenty will flag one anyway. The calculator above offers 80, 85, 90, 95 and 99%, the same five levels our crosstab reports use.
One point about the statistical significance meaning that trips up careful people: the p-value is not the probability that your result is due to chance, and it is not the probability that your hypothesis is true. It is a statement about the data under one assumption. The American Statistical Association said so formally in 2016, in a statement of six principles written because the misreading had become so common.
For two percentages from two separate groups, the statistical significance calculation is a two-proportion z-test. It fits on one line:
Here p₁ and p₂ are the two percentages written as proportions, n₁ and n₂ are the two sample sizes, and p is the pooled proportion across both groups. This is how to find statistical significance by hand, in five steps:
The base is the number of people who answered the question in that group, which is often smaller than the group itself. Example: 52% of 400 women and 44% of 380 men say they would buy.
If there were truly no difference, both groups would share one percentage. The best estimate of it is everyone who said yes over everyone asked: (208 + 167) / 780 = 48.1%.
The standard error is how much the gap would wobble from sample to sample by chance alone: the square root of 0.481 x 0.519 x (1/400 + 1/380), which is 0.0358, or 3.6 points.
8 points over 3.58 points gives z = 2.24. The gap is 2.24 standard errors wide.
At 95% confidence the cutoff is 1.96. Since 2.24 is larger, the difference is statistically significant, with a p-value of 0.025. At 99% confidence the cutoff is 2.58, and the same result does not pass.
That is the whole method for how to determine statistical significance between two groups, and it is the test behind the significance letters in a cross-tabulation. The calculator reports one more number worth reading: the interval around the gap. In the example it is 8 points, plus or minus 7.0, so the true difference is plausibly anywhere from 1 point to 15. The difference passes the test, and its size is still quite uncertain.
Whether a gap passes depends on the sample size as much as on the gap. The table takes one group at 50% and a second group 5 or 10 points lower, with equal sample sizes, and shows the p-value at each size.
| Completes per group | 5-point gap (50% vs 45%) | 10-point gap (50% vs 40%) |
|---|---|---|
| 100 | p = 0.479, not significant | p = 0.155, not significant |
| 200 | p = 0.317, not significant | p = 0.044, significant |
| 400 | p = 0.157, not significant | p = 0.004, significant |
| 1,000 | p = 0.025, significant | p = under 0.001, significant |
| 2,500 | p = under 0.001, significant | p = under 0.001, significant |
Read the left column downward. A 5-point gap shows no statistical significance at 100, 200 or 400 per group. The gap is the same in every row, and what grows down the column is the evidence for it. This is why a result with no statistical significance should never be written up as proof that two groups agree. It means the study could not tell, and the honest wording is that no difference was detected at this sample size.
The table also explains most disappointing subgroup analysis. A survey of 1,000 people reads well in total and then gets cut into regions of 150, where only gaps of 11 points or more can pass. If subgroup differences are the point of the study, size the subgroups first. The sample size calculator page has the table of completes per group for each gap.
Most online significance calculators were built for A/B tests, where two separate sets of visitors see two versions of a page. Survey results often pose a different question. One poll of 800 voters puts candidate A at 48% and candidate B at 43%. Is A ahead? One brand tracker asks 800 shoppers for their first choice and gets 48% and 43%. Is the leader really leading?
Here both percentages come from the same respondents, and each person picked only one answer. That makes the two shares move against each other: if the sample happens to hold too many A supporters, it holds too few B supporters. The gap between them swings about twice as much as either share does. The standard error of the difference becomes:
Run the 48 to 43 example both ways. Treated as two separate groups of 800, the test reports p = 0.045 and calls the lead significant. Treated correctly as two answers from one sample, p = 0.138 and the margin on the lead is 6.6 points, so a 5-point lead is inside it. The first answer is wrong, and it is the one a two-group calculator gives.
Pew Research Center makes the same point about election polls: the margin of error for the gap between two candidates is generally about twice the margin reported for each candidate alone. A poll with a 3-point margin needs a lead of about 6 points before the lead is outside sampling error. Use the second mode of the calculator for this case. It is the same arithmetic behind the margin on our NPS calculator, since a Net Promoter Score is also a gap between two shares of one sample. Reading a horse race this way is routine in political polling.
Weighting corrects a sample that has, say, too many older respondents by counting some people as more than one person and others as less. It reduces bias, and it costs precision: a weighted sample of 1,000 carries less information than an unweighted one. The size of the loss is the design effect. Divide the sample size by it and you get the effective sample size, the number that belongs in a significance test.
A design effect of 1.5 turns 1,000 interviews into an effective 667. Go back to the worked example, 52% of 400 against 44% of 380, which passed at 95% with p = 0.025. If those figures come from data weighted with a design effect of 1.5, the bases become 267 and 253, the p-value rises to 0.068, and the difference no longer passes.
Type the design effect into the optional field and the calculator makes the adjustment. Pew, in the article linked above, describes weighting as making the margin of error larger and says a pollster who ignores the design effect is claiming more precision than the survey has. Our own reports follow the same rule: significance letters are tested on effective bases, the topline prints the design effect, and columns with an effective base under 30 are not tested at all. How the weights are built is covered in the survey weighting guide.
A calculator tests one comparison. A survey report tests hundreds. Take a banner with five columns (five age groups, for instance). Five columns make ten pairs, and a questionnaire with 30 answer rows turns that into 300 tests. At 95% confidence each test has a 5% chance of flagging a difference that is not there, so a table where nothing truly differs would still show about 15 significance flags.
Nobody can tell from the table which 15 they are. Three habits keep statistical significance testing honest at that scale:
Reading a banner table column by column is covered in how to analyze survey results.
Statistical significance answers one narrow question: is the difference probably real? It says nothing about whether the difference is big enough to matter. That second question is practical significance, and the two come apart in both directions.
With enough respondents, trivial gaps pass. Two groups of 10,000 at 51% and 49% give p = 0.005, comfortably significant, and it is hard to name a decision that turns on two points. With too few respondents, large gaps fail. Two groups of 60 at 56% and 44% are 12 points apart, a difference most product teams would act on, and the test returns p = 0.19.
The ASA statement cited earlier puts both halves in its principles: a p-value does not measure the size of an effect or the importance of a result, and business decisions should not be based only on whether a p-value passes a threshold. In practice that means reporting three things together: the size of the gap, the interval around it, and whether it passed. A reader who sees only an asterisk has been told the least useful of the three.
The cleanest way to handle practical vs statistical significance is to settle the practical part first. Before fielding, ask what difference would change the decision. If the answer is 10 points, plan for about 389 completes per group and treat anything smaller as noise you were never going to act on.
A significance test and a confidence interval are built from the same standard error, so they nearly always agree: if the 95% interval around the difference excludes zero, the difference is significant at 95%. The interval is the more informative of the two, because it also shows how large or small the real gap could be.
There is one common mistake here. People compare the two separate margins of error and conclude that overlapping intervals mean no significant difference. That rule is too strict. In the worked example, 52% of 400 has a margin of 4.9 points (47.1% to 56.9%) and 44% of 380 has a margin of 5.0 (39.0% to 49.0%). The two intervals overlap by two points, and the difference is still significant at p = 0.025. The margin on a difference is smaller than the two separate margins added together, so always test the difference directly.
The reverse does hold: when two intervals do not overlap at all, the difference is significant.
A mean rating (4.1 against 3.8 on a five-point scale) needs each group's standard deviation and a t-test or Welch test. Two percentages are not enough information.
Before-and-after answers from the same respondents are paired. The right test (McNemar) needs to know how many individuals changed their answer, which the two percentages do not show.
If one respondent can tick both answers, the two shares overlap in a way neither mode here models. You need the respondent-level data.
A p-value covers random sampling error only. If the people who answered differ from the people who did not, the test cannot see it, and a larger sample makes the wrong number more precise.
A result is statistically significant when it is larger than random sampling error would plausibly produce by itself. Every survey talks to a sample, and two samples from the same population will give slightly different numbers by luck. A significance test asks how often a gap as large as yours would appear if there were no real difference. If the answer is rarely (under 5% of the time, by the usual convention), the gap is called statistically significant.
The convention is 95% confidence, which is the same thing as a significance level of 0.05. Market research reports often test at 90% because subgroup bases are small and a missed difference is costly. Large tables with hundreds of comparisons are better read at 99%, because at 95% roughly one comparison in twenty flags by chance. Choose the level before you look at the results, and state it wherever the result is reported.
Use a two-proportion z-test. Pool the two groups to get one overall percentage, compute the standard error as the square root of pooled x (1 - pooled) x (1/n1 + 1/n2), and divide the gap between the two percentages by that standard error. If the result is larger than 1.96, the difference is significant at 95% confidence. The calculator at the top of this page does the arithmetic and also reports the p-value.
It means the sample was too small to tell the difference you saw apart from chance. It does not mean the two groups are the same. A 5-point gap between two groups of 200 is not significant (p = 0.32), yet the same 5-point gap between two groups of 1,000 is (p = 0.025). Report a non-significant result as 'no difference detected at this sample size', and look at the interval around the gap to see how large a real difference could still be.
No. Statistical significance says a difference is probably real. Practical significance says it is large enough to change a decision. With 10,000 respondents per group, 51% against 49% is statistically significant (p = 0.005), and almost no business decision turns on two points. Decide the smallest difference that would change what you do before fielding, then size the sample to detect that difference.
Not as a whole. Significance belongs to a specific comparison: this percentage against that one, at stated sample sizes. A survey of any size can contain some comparisons that pass and others that do not. When people ask whether a survey is statistically significant they usually mean whether the sample is large enough, which is a margin of error question that a sample size calculator answers.
It depends on the size of the difference you need to detect. At 95% confidence and 80% power, a 10-point difference near 50% needs about 389 completes per group, and a 5-point difference needs about 1,566 per group. Halving the difference roughly quadruples the sample. The sample size calculator page has the full table and the formula.
Related: sample size calculator · cross-tabulation, with significance letters · survey weighting · NPS calculator · survey analysis software
Wavefield crosstabs mark significant differences with column letters at the confidence level you pick, tested on effective bases when the data is weighted. Building and previewing is free; fielding starts at $99 per study.
Start a study