NAG Library Chapter Introduction

G07 – Univariate Estimation

There are two main aspects of statistical inference: the estimation of the parameters and the testing of hypotheses about the parameters. In the example above, the values of the parameter σ2 may be estimated and the hypothesis that μ≥3 tested. This chapter is mainly concerned with estimation but the test of a hypothesis about a parameter is often closely linked to its estimation. Tests of hypotheses which are not linked closely to estimation are given in the chapter on nonparametric statistics (Chapter G08).

There are two types of estimation to be considered in this chapter: point estimation and interval estimation. Point estimation is when a single value is obtained as the best estimate of the parameter. However, as this estimate will be based on only one of a large number of possible samples, it can be seen that if a different sample were taken, a different estimate would be obtained. The distribution of the estimate across all the possible samples is known as the sampling distribution. The sampling distribution contains information on the performance of the estimator, and enables estimators to be compared. For example, a good estimator would have a sampling distribution with mean equal to the true value of the parameter; that is, it should be an unbiased estimator; also the variance of the sampling distribution should be as small as possible. When considering a parameter estimate it is important to consider its variability as measured by its variance, or more often the square root of the variance, the standard error.

The sampling distribution can be used to find interval estimates or confidence intervals for the parameter. A confidence interval is an interval calculated from the sample so that its distribution, as given by the sampling distribution, is such that it contains the true value of the parameter with a certain probability.

Estimates will be functions of the observed sample and these functions are known as estimators. It is usually more convenient for the estimator to be based on statistics from the sample rather than all the individuals observations. If these statistics contain all the relevant information then they are known as sufficient statistics. There are several ways of obtaining the estimators; these include least-squares, the method of moments, and maximum likelihood. Least-squares estimation requires no knowledge of the distributional form of the error apart from its mean and variance matrix, whereas the method of maximum likelihood is mainly applicable to situations in which the true distribution is known apart from the values of a finite number of unknown parameters. Note that under the assumption of Normality, the least-squares estimation is equivalent to the maximum likelihood estimation. Least squares is often used in regression analysis as described in Chapter G02, and maximum likelihood is described below.

Estimators derived from least-squares or maximum likelihood will often be greatly affected by the presence of extreme or unusual observations. Estimators that are designed to be less affected are known as robust estimators.

Let Xi be a univariate random variable with probability density function

fXixi;θ,

where θ is a vector of length p consisting of the unknown parameters. For example, a Normal distribution with mean θ1 and standard deviation θ2 has probability density function

12πθ2 exp-12 xi-θ1θ2 2 .

The likelihood for a sample of n independent observations is

Like=∏i=1nfXi xi;θ ,

where xi is the observed value of Xi. If each Xi has an identical distribution, this reduces to and the log-likelihood is The maximum likelihood estimates (θ^) of θ are the values of θ that maximize (1) and (2). If the range of X is independent of the parameters, then θ^ can usually be found as the solution to Note that ∂L ∂θj  is known as the efficient score.

The use of maximum likelihood estimation allows the construction of generalized likelihood ratio tests. If λ=2l1-l2, where l1 is the maximized log-likelihood function for a model 1 and l2 is the maximized log-likelihood function for a model 2, then under the hypothesis that model 2 is correct, 2λ is asymptotically distributed as a χ2 variable with p-q degrees of freedom. Consider two models in which model 1 has p parameters and model 2 is a sub-model (nested model) of model 1 with q<p parameters, that is model 1 has an extra p-q parameters. This result provides a useful method for performing hypothesis tests on the parameters. Alternatively, tests exist based on the asymptotic Normality of the estimator and the efficient score; see page 315 of Cox and Hinkley (1974).

Suppose we can find a function, tx,θ, whose distribution depends upon the sample x but not on the unknown parameter θ, and which is a monotonic (say decreasing) function in θ for each x, then we can find t1 such that P t 1 ≤ t x,θ = 1 - α  no matter what θ happens to be. The function tx,θ is known as a pivotal quantity. Since the function is monotonic the statement that t1≤tx,θ may be rewritten as θ≥θ1x see Figure 1. The statistic θ1x will vary from sample to sample and if we assert that θ≥θ1x for any sample values which arise, we will be right in a proportion 1-α of the cases, in the long run or on average. We call θ1x a 1-α upper confidence limit for θ.

We have considered only an upper confidence limit. The above idea may be generalized to a two-sided confidence interval where two quantities, t0 and t1, are found such that for all θ, P t 1 ≤ t x,θ ≤ t 0 = 1 - α . This interval may be rewritten as θ0x≤θ≤θ1x. Thus if we assert that θ lies in the interval [θ0x,θ1x] we will be right on average in 1-α proportion of the times under repeated sampling.

If θ1 is a location parameter, as described above, then equation (3) becomes where ψz=- ddz loggz.

For the Normal distribution ψz=z and χz=z2/2. Thus, the maximum likelihood estimates for θ1 and σ2 are the sample mean and variance with the n divisor respectively. As the latter is biased, (7) can be replaced by where β is a suitable constant, which for the Normal χ function is 12 .

The influence of an observation on the estimates depends on the form of the ψ and χ functions. For a discussion of influence, see Hampel et al. (1986) and Huber (1981). The influence of extreme values can be reduced by bounding the values of the ψ- and χ-functions. One suggestion due to Huber (1981) is Redescending ψ-functions are often considered; these give zero values to ψz for large positive or negative values of z. Hampel et al. (1986) suggested Usually a χ-function based on Huber's ψ-function is used: χ=ψ2/2. Estimators based on such bounded ψ-functions are known as M-estimators, and provide one type of robust estimator.

For a general discussion of robust estimation, see Hampel et al. (1986) and Huber (1981).

In Section 2.2 it was shown how tests of hypotheses can be used to find confidence intervals. That approach uses a parametric test that requires the assumption that the data used in the computation of the confidence has a known distribution. As an alternative, a more robust confidence interval can be found by replacing the parametric test by a nonparametric test. In the case of the confidence interval for the location parameter, a Wilcoxon test statistic can be used, and for the difference in location, computed from two samples, a Mann–Whitney test statistic can be used.