Minimum mean square error

From Wikipedia, the free encyclopedia

In statistics and signal processing, a minimum mean square error (MMSE) estimator describes the approach which minimizes the mean square error (MSE), which is a common measure of estimator quality. The term MMSE specifically refers to estimation in a Bayesian setting, since in the opposing frequentist setting there does not exist a single estimator having minimal MSE.

Contents 1 Definition 2 Properties 3 Example 4 Notes 5 Further reading

[edit] Definition

Let X be an unknown random variable, and let Y be a known random variable (the measurement). An estimator is any function of the measurement Y, and its MSE is given by

where the expectation is taken over both X and Y.

The MMSE estimator is then defined as the estimator achieving minimal MSE.

In many cases, it is not possible to determine a closed form for the MMSE estimator. In these cases, one possibility is to seek the technique minimizing the MSE within a particular class, such as the class of linear estimators. The linear MMSE estimator is the estimator achieving minimum MSE among all estimators of the form AY + b. If the measurement Y is a random vector, A is a matrix and b is a vector. (Such an estimator would more correctly be termed an "affine MMSE" estimator, but the term linear estimator is widely used.)

[edit] Properties

• Under some weak regularity assumptions,^[1] the MMSE estimator is uniquely defined, and is given by

In other words, the MMSE estimator is the conditional expectation of X given the observed value of the measurements.

• If X and Y are jointly Gaussian, then the MMSE estimator is linear. As a consequence, to find the MMSE estimator, it is sufficient to find the linear MMSE estimator. Such a situation occurs in the example presented in the next section.

• The orthogonality principle: An estimator is MMSE if and only if

for all functions f(Y) of the measurements. A different version of the orthogonality principle exists for linear MMSE estimators.

[edit] Example

An example can be shown by using a linear combination of random variable estimates X₁,X₂ and X₃ to estimate another random variable X₄ using If the random variables X = [X₁,X₂,X₃,X₄]^T are real Gaussian random variables with zero mean and covariance given by

we will estimate the vector X₄ and find coefficients a_i such that the estimate is an optimal estimate of X₄. We will use the autocorrelation matrix, R, and the cross correlation matrix, C, to find vector A, which consists of the coefficient values that will minimize the estimate. Autocorrelation matrix R is defined as

Cross correlation matrix C is defined as

In order to find the optimal coefficients by the orthogonality principle we solve the equation RA = C by inverting R and multiplying to get

So we have a₁ = 2.57, a₂ = − .142, and a₃ = .5714 as the optimal coefficients for Computing the minimum mean square error then gives .^[2]

[edit] Notes

1. ^ Lehmann and Casella, Corollary 4.1.2.
2. ^ Moon and Stirling.

[edit] Further reading

• Johnson, D. (22 November 2004). Minimum Mean Squared Error Estimators. Connexions
• Prediction and Improved Estimation in Linear Models, by J. Bibby, H. Toutenburg (Wiley, 1977). This book looks almost exclusively at minimum mean-square error estimation and inference.
• Jaynes, E. T. Probability Theory: The Logic of Science. Cambridge University Press, 2003.
• Lehmann, E. L.; Casella, G. (1998). Theory of Point Estimation. Springer, 2nd ed, ch. 4. ISBN 0-387-98502-6. 
• Kay, S. M. (1993). Fundamentals of Statistical Signal Processing: Estimation Theory. Prentice Hall, 344-350. ISBN 0-13-042268-1. 
• Moon, T.K. and W.C. Stirling. Mathematical Methods and Algorithms for Signal Processing. Prentice Hall. 2000.