Page:Sm all cc.pdf/30

 99% confidence that the true mean is within the interval -0.23 to 0.27. Actually the true mean for this dataset is zero.

Selection of a confidence level (α$95$, α$99$, etc.) usually depends on one’s evaluation of which risk is worse: the risk of incorrectly identifying a variable or effect as significant, or the risk of missing a real effect. Is the penalty for error as minor as having a subsequent researcher correct the error, or could it cause disaster such as an airplane crash? If prior knowledge suggests one outcome for an experiment, then rejection of that outcome needs a higher than ordinary confidence level. For example, no one would take seriously a claim that an experiment demonstrates test-tube cold fusion at the 95% confidence level; a much higher confidence level plus replication was demanded. Most experimenters use either a 95% or 99% confidence level. Tables for calculation of confidence limits other than 95% or 99%, called tables of the t distribution, can be found in any statistics book.

How Many Measurements are Needed? The standard error of the mean σ$\overline{X}$ is also the key to estimating how many measurements to make. The definition σ$\overline{X}$=σN$-0.5$ can be recast as N=σ$2$/σ$2$$\overline{X}$. Suppose we want to make enough measurements to obtain a final mean that is within 2 units of the true mean (i.e., σ$\overline{X}$≤2), and a small pilot study permits us to calculate that our measurement scatter σ≈10. Then our experimental series will need N≥10$2$/2$2$, or N≥25, measurements to obtain the desired accuracy at the 68% confidence level (or 1σ$\overline{X}$). For about 95% confidence, we recall that about 95% of points are within 2σ of the mean and conclude that we would need 2σ$\overline{X}$≤2, so N≥10$2$/1$2$, or N≥100 measurements. Alternatively and more accurately, we can use the t table above to determine how many measurements will be needed to assure that our mean is within 2 units of the true mean at the 95% confidence level (α$95$>≤2): we need for t$95$=α$95$/σ$\overline{X}$=α$95$N$0.5$/σ=2N$0.5$/10=0.2N$0.5$ to be greater than the t$95$ in the table above for that N. By trying a few values of N, we see that N≥100 is needed.

As a rule of thumb, one must quadruple the number of measurements in order to double the precision of the result. This generalization is based on the N$0.5$ relationship of standard deviation to standard error and is strictly true only if our measure of precision is the standard error. If, as is