What makes a number statistically significant?
Giovanni Petris, associate professor of mathematical sciences in the J. William Fulbright College of Arts and Sciences, replies:
Pretend for a moment that we are back in October 08, during the final rounds of the presidential campaign. Suppose an opinion poll finds that Obama leads on McCain 46 to 42 percent. How can we be sure that Obama is actually leading among all American voters? After all, the poll is based on a sample. Unfortunately, the answer is that we can never be sure that a conclusion drawn from a sample is valid for the entire population. However, under certain circumstances, we can be highly confident that this conclusion does hold for the population. When this is the case, statisticians say that the conclusion from the sample is significant. Note that, in the example above, the significant conclusion would be that Obama is leading over McCain, and not that he has the support of exactly 46 percent of all American voters.
To understand how statisticians decide whether a statement is significant or not, it is convenient to turn to a simpler example. Suppose somebody gives you a quarter, you flip the coin 15 times and you get 14 heads and one tail. You would be right to conclude that the coin is biased. Probably your reasoning is that one tail out of 15 coin flips is something very unlikely for an unbiased coin, unlikely events rarely happen, ergo, the coin is biased. The reasoning is essentially correct, although a statistician would take it one step further, and consider the probability of observing one or zero tails out of the 15 flips. In other words, she would throw into the calculations also the possibility of observing an even more extreme event. The probability of zero or one tails is 0.0005 for a fair coin; since this probability is less than 0.05, or one in twenty, she concludes that the coin is biased -- 0.05 is a magic number for statisticians!
Back to the initial example: to say that the 46 to 42 percent difference is significant, our statistician must have calculated that, if in fact McCain were leading among American voters, then observing Obama leading in the poll by four percentage points or more has less that one chance in twenty of happening. Ergo, she confidently concludes that Obama is actually leading.