Parameter estimation problems arise in many fields. In the radar or sonar area, an active system estimates the range, velocity, and angle of the target. In a passive system, the bearing to the target is estimated. In communication systems, an estimate of the phase and frequency of a sinusoid is usually required. In radio astronomy, an estimate of frequency is a common objective. In order to determine the power density spectrum of a random process, a common technique is to model it parametrically and then estimate the parameters. In navigation systems, one objective is to estimate the current location of the platform. In traffic management systems, the traffic flow may be modeled as a Poisson process and the average rate estimated.

All of these problems are characterized by a model in which we observe some function of a parameter set of interest. These observations are corrupted by noise. In the first category of problems of interest in this book, the parameters are constant during the observation interval. We refer to this as the static parameter estimation problem.

There are two basic models for the static parameter estimation problem. The first model assumes that the parameters are nonrandom unknown quantities. The second model assumes that the parameters are random variables with a priori probability densities.  Parameter estimation based on the first model is referred to non-Bayesian or Fisher estimation.  Parameter estimation based on the second model is referred to as Bayesian estimation. Thomas Bayes was a Presbyterian minister in England (1702-1761). He was also a mathematician who is best known for Bayes rule for computing the a posteriori probability of an event. His results are in a paper entitled, ``Essay Towards Solving a Problem in the Doctrine of Chances'' (1763), which was published posthumously in the Philosophical Transactions of the Royal Society of London. The paper is available from several sources on the Internet.  The focus of this book is on Bayesian estimation, although some relevant results from non-Bayesian estimation are included. It is important to emphasize that the correct model depends on the specific application.

In many applications of interest, the parameters appear in the observations in a nonlinear manner. In these cases, it is generally impossible to analyze the performance of an estimator and simulation is necessary to evaluate performance. An important problem is to develop bounds on the performance of any estimator or class of estimators. Generally, these are lower bounds, but in some cases upper bounds are obtainable. The focus of the first half of this book is on the development of Bayesian bounds on the performance of algorithms for estimating static parameters. We include a discussion of maximum likelihood (ML) and maximum a posteriori (MAP) estimators, but we do not cover efficient computation techniques or sub-optimum estimators. Our goal is to establish tight bounds on performance that can be used as a standard of comparison for any estimator.

The second estimation problem of interest is the estimation of a sample function of a random process. We refer to this as the random process estimation problem. In the radar or sonar area, the system tracks the path of the target. In an analog communication system, the instantaneous value of the message is estimated.

The random process to be estimated may be be modeled in one of two ways: a correlation function approach or a state variable approach.  The first approach is due to Wiener [Wie 49] and Kolmogorov [Kol 41] and, for the linear Gaussian model, the Bayes estimator is the Wiener filter. The second approach is due to Kushner [Kus 64a]and Stratonovich [Str 59a].  For the linear Gaussian model, the Bayes estimator is the Kalman-Bucy filter [KB 61], [Kal 60]. The linear Gaussian model is well understood and discussed in a number of references.

Our focus is on the case in which the process dynamics or the observation equations, or both, are nonlinear. We will generally use a Markov process representation of the process and estimate the state vector. Although we will discuss various estimators, our focus will be on deriving bounds on the mean-square error (MSE) performance of any estimator. Specifically, we will derive recursive Bayesian bounds on the performance of nonlinear estimators of the state of nonlinear dynamic systems.

Our work in the area of Bayesian estimation and Bayesian bounds spans two disjoint time periods, separated by a gap of sixteen years. The first period from 1958-1972 included research and teaching at Massachusetts Institute of Technology (MIT) by the first author. In 1972, I took leave from MIT and held a number of interesting positions in government and industry. In 1988, I joined the faculty at George Mason University (GMU) where Kristine Bell was a graduate student. The second period is from 1988 to the present and includes our joint activities in the Bayesian estimation area.

My interest in nonlinear estimation started when I joined Professor Yuk Wing Lee's research group in the Research Laboratory of Electronics at MIT in 1959. The previous year I had taken graduate courses from Professor Lee and Professor Amar Bose. The opportunity to work with them in the exciting area of statistical signal processing was a prize for a young graduate student.

The previous year, Norbert Wiener's book, Nonlinear Problems in Random Theory, had been published [Wie 58].  The book was actually written by Lee, Bose, and a group of doctoral students, based on a series of lectures by Wiener.  An excellent discussion of the Wiener-Lee interaction is contained in Therrien's article, ``The Lee-Wiener Legacy, A History of the Statistical Theory of Communication'' [The02].  A rite of passage into the group was the study of Wiener's book. It was also a necessary defense mechanism because Professor Wiener would stop into the room where the graduate students worked to try his ideas. One needed to have some understanding to avoid complete embarrassment. Fortunately Wiener's eyesight was failing so he could not identify the source of a foolish response.

One of the topics in the book was the use of Volterra series and various orthogonal functionals to represent nonlinear systems. It contained three lectures on the application of the theory to frequency modulation problems which generated my interest in analog communication theory. Over the years, the research led to a summer course presented at MIT in 1964 and several papers (e.g. [Van 64], [VB 65]).

It turned out that our viewpoint on modulation theory could best be understood by an audience with a clear understanding of modern detection and estimation theory. At that time, there was no suitable text available to cover the material of interest and emphasize the points that I felt were important, so I started writing notes. It was clear that in order to present the material to graduate students in a reasonable amount of time it would be necessary to develop a unified presentation of the three topics: detection, estimation, and modulation theory, and exploit the fundamental ideas that connected them. The final result was a series of three books on Detection, Estimation, and Modulation Theory [Van 68], [Van 71a], [Van 71b], which were reprinted as [Van 01a], [Van 03], and [Van 01b].

After graduating in 1961, my research and teaching interests gradually diverged from Professor Lee's and I started my own research group in 1964. I was fortunate to have a number of outstanding doctoral students: Arthur Baggeroer, Donald Snyder, Lewis Collins, Michael Austin, and Ted Cruise, who worked in the detection and estimation area. A primary research area of interest was the estimation of random processes based on nonlinear observations. Thus, Bayesian estimation, rather than classical Fisher estimation, was of primary interest. A key goal was to find a method to calculate a lower bound on the mean-square error (MSE) performance of any estimator.

Our research led to a sequence of Bayesian bounds. The first bound was the Bayesian version of the classic Cramér-Rao bound (CRB) [Van 64] which is described in Paper 1.1:[Van 68, Van 01a, pp. 66-86]. The Bayesian version of other classical bounds such as the Bhattacharyya bound were also developed. The Bayesian CRB (BCRB) and the Bayesian Bhattacharyya bound are most appropriate in the small error region and are not useful in predicting threshold behavior. Although we worked with the Barankin bound in the nonrandom parameter estimation problem (e.g. [Bag 70] or [Gla 72]), we did not extend it to the Bayesian model.  This would be accomplished later by Bobrovsky and Zakai (Paper 6.6:[BZ 75]).

In order to study the threshold problem, we introduced the method of interval estimation (e.g. Paper 2.2:[Van 68, Van 01a, pp. 272-286] and Paper 2.20:[Van 71b, Van 01b, pp. 275-308]) to approximate the MSE behavior in nonlinear estimators. In the same period, Ziv and Zakai (Paper 2.11:[ZZ 69]}) derived a lower bound which would become the starting point for the Ziv-Zakai family of bounds.

In order to bound the MSE in the estimation of a continuous random process, we used a Karhunen-Loève expansion of the message process and obtained an integral equation for the MSE averaged over the observation interval (e.g. Paper 6.4:[Van 66]). An important interpretation of the result was that the integral equation satisfied by the Bayesian bound was the same as the equation satisfied by the optimum linear filter, except that the actual noise process was replaced by a modified noise process which depended on the derivative of the signal with respect to the message. This interpretation in terms of an equivalent linear system appears repeatedly in subsequent papers (e.g. Paper 6.7:[BZ 76]). The multidimensional case, which includes the estimation of the state vector as a special case, is also included. The result for the realizable filtering case is included but not proved. In Paper 6.5:[SR 72], Snyder and Rhodes used a similar approach and derived the Bayesian bound for a linear Gaussian process model with nonlinear observations.  Good references for our research in this time frame are the MIT Press monographs by Snyder [Sny 69] and Baggeroer [Bag 70] which are based on their doctoral dissertations.

In 1988, I returned to academia as a Professor of Systems and Electrical Engineering at George Mason University. Kristine Bell was a part-time graduate student pursuing her M.S. degree. She had worked for me at M/A-COM Government Systems since 1985. When she finished her M.S. in 1990, I hired her as a Research Instructor in my C3I Center of Excellence and our collaboration began.  Bayesian estimation and, in particular, Bayesian bounds had evolved over the sixteen years. Many of the papers in this book were published during the 1972-88 period.

Bobrovsky and Zakai (Paper 6.6:[BZ 75]) had developed the Bayesian version of the Barankin Bound. Weiss and Weinstein (Paper 2.6:[WW 85] and Paper 2.7:[WW 88]) had developed the covariance inequality family of lower bounds.  The tightest bound, the Weiss-Weinstein bound (WWB), included the Bayesian CRB and the Bobrovsky-Zakai bound as special cases. The WWB provided a good indication of the threshold in many nonlinear estimation problems. The covariance inequality family also included combined bounds (e.g. BCRB and WWB) which could provide a smooth transition from the small error region (BCRB) into the threshold region (WWB).

The original Ziv-Zakai bound had been improved by several authors, Chazan, Zakai, and Ziv (Paper 2.12:[CZZ 75]), Bellini and Tartara (Paper 2.13:[BT 74]), and Weinstein (Paper 2.14:[Wei 88]). The improved bound provided a good indication of the threshold but was only applicable to scalar parameters with uniform a priori probability densities.

Dr. Bell's Ph.D. thesis in 1995 derived an extended Ziv-Zakai lower bound for the estimation of vector parameters with arbitrary a priori probability densities (Papers 2.15:[BSEV 97] and 2.16:[BEV 96]).

After graduation, Dr. Bell became an Assistant Professor in the Department of Statistics at GMU and our collaboration continued. The idea of a reprint book on Bayesian bounds originated in 1998 but there were several obstacles. I had to finish Part IV of the Detection, Estimation, and Modulation series on Optimum Array Processing which was published in 2002 [Van 02] and Kristine had to have the appropriate accomplishments to receive tenure.

Two events caused the project to resurface in 2005. I agreed to give a keynote speech at the Adaptive Sensor Array Processing Workshop at M.I.T. Lincoln Laboratory on 7 June 2005. The topic was Bayesian Bounds and provided motivation to review the current state-of-the-art. Kristine and I also had a research grant from Michael Zatman at the Defense Advanced Research Projects Agency (DARPA) to study tracking for sparse antenna arrays and tracking in multistatic radar systems. Our recent work on Bayesian bounds (Paper 7.12:[VBW 06] and Paper 7.14:[BV 06]) was done under this research grant. In addition, Michael was supportive of our research in the general theory of Bayesian bounds.

Our objective was to provide a comprehensive overview of the evolution of Bayesian bounds from the initial work to the current state-of-the-art. We wrote a long introduction to provide a unifying content for the ninety reprint papers. The reprint papers include both theoretical results and their usage in a variety of applications.

We assume that the reader has had a graduate course in Estimation Theory (e.g. [Van 68], [Van 01a]). We think the book will be useful as a text in an advanced seminar course on Bayesian estimation and as a reference for users. It was the basis for a tutorial at ICASSP 2007 in April 2007.

Harry L. Van Trees
Kristine L. Bell

References

[Bag 70]     A. B. Baggeroer, State Variables and Communications Theory, Research Monograph No. 61, Cambridge, MA and London, England: The MIT Press, 1970.

[BSEV 97]  K. L. Bell, Y. Steinberg, Y. Ephraim, and H. L. Van Trees, ``Extended Ziv-Zakai Lower Bound for Vector Parameter Estimation,'' IEEE Trans. Info. Theory, vol. 43, no. 2, pp. 624-637, March 1997.

[BEV 96]    K. Bell, Y. Ephraim, and H. L. Van Trees, ``Explicit Ziv-Zakai Lower Bounds for Bearing Estimation,'' IEEE Trans. Signal Process., vol. 44, no. 11, pp. 2810-2824, Nov. 1996.

[BV 06]       K. L. Bell and H. L. Van Trees, ``Combined Cramér-Rao/Weiss-Weinstein Bound for Tracking Target Bearing,'' 4th Annual IEEE Workshop on Sensor Array and Multi-channel Processing (SAM 2006), Waltham, MA, pp. 273-277, July 2006.

[BT 74]       S. Bellini and G. Tartara, ``Bounds on Error in Signal Parameter Estimation,'' IEEE Trans. Commun., vol. COM-22, pp. 340-342, Mar. 1974.

[BZ 75]       B. Z. Bobrovsky, and M.Zakai, ``A Lower Bound on the Estimation Error for Markov Processes,'' IEEE Trans. Auto. Control, vol. 20, no. 6, 785-788, 1975.

[BZ 76]       B. Z. Bobrovsky and M. Zakai, ``A Lower Bound on the Estimation Error for Certain Diffusion Processes,'' IEEE Trans. Info. Theory, vol. IT-22, no. 1, pp. 45-52, January 1976.

[CZZ 75]     D. Chazan, M. Zakai, and J. Ziv, ``Improved Lower Bounds on Signal Parameter Estimation,'' IEEE Trans. Info. Theory, vol. IT-21, no. 1, pp. 90-93, Jan. 1975.

[Gla 72]      F. E. Glave, ``A New Look at the Barankin Lower Bound,'' IEEE Trans. Info. Theory, vol. IT-18, no. 3, pp. 349-355, May 1972.

[Kal 60]      R.E. Kalman, ``New Approach to Linear Filtering and Prediction Problems,'' Trans. ASME, J. Basic Eng., vol. 82, pp. 34-45, March 1960.

[KB 61]      R.E. Kalman and R. Bucy, ``New Results in Linear Filtering and Prediction Theory,'' ASME J. Basic Eng., vol. 83, pp. 95-108, March 1961.

[Kol 41]      A. Kolmogorov, ``Interpolation und Extrapolation von stationären zufälligen Folgen,'' Bulletin de l'académie des sciences de U.R.S.S., Ser. Math., vol. 5, pp. 3-14, 1941.

[Kus 64a]   H. J. Kushner, ``On the Differential Equations Satisfied by Conditional Probability Densities of Markov Processes, with Applications,'' J. Siam Control, Ser. A, vol. 2, no. 1, pp. 106-119, 1964.

[Sny 69]     D. L. Snyder, The State-Variable Approach to Continuous Estimation with Applications to Analog Communications Theory, Cambridge, MA: The MIT Press, 1969.

[SR 72]      D. L. Snyder and I. B. Rhodes, ``Filtering and Control Performance Bounds with Implication on Asymptotic Separation,'' Automatica, vol. 8, pp. 747-753, Nov. 1972.

[Str 59a]     R. L. Stratonovich, ``On the Theory of Optimal Nonlinear Filtration of Random Functions," Theory Prob. and Its Appl., vol. 4, pp. 223-225, 1959.

[The 02]      C. W. Therrien, ``The Lee-Wiener Legacy,'' IEEE Signal Processing Magazine, pp. 33-44, Nov. 2002.

[Van 64]     H. L. Van Trees, ``Analog Modulation and Continuous Estimation,'' notes for MIT Summer Session, July 1964.

[VB 65]      H. L. Van Trees and C. J. Boardman, ``Optimum Angle Modulation,'' IEEE Trans. Commun. Tech., vol. COM-13, no. 4, pp. 452-467, Dec. 1965.

[Van 66]     H. L. Van Trees, ``Bounds on the Accuracy Attainable in the Estimation of Continuous Random Processes,'' IEEE Trans. Info. Theory, vol. 12, no. 3, pp. 298-305, July 1966.

[Van 68]     H. L. Van Trees, Detection, Estimation, and Modulation Theory, Part I, New York, NY: John Wiley and Sons, 1968.

[Van 71a]   H. L. Van Trees, Detection, Estimation, and Modulation Theory, Part II: Nonlinear Modulation Theory, New York, NY: John Wiley and Sons, 1971.

[Van 71b]   H. L. Van Trees, Detection, Estimation, and Modulation Theory, Part III: Radar-Sonar Signal Processing and Gaussian Signals in Noise, New York, NY: John Wiley and Sons, 1971.

[Van 01a]   H. L. Van Trees, Detection, Estimation, and Modulation Theory, Part I, New York, NY: John Wiley and Sons, 1968. [Republished by Wiley-Interscience, 2001.]

[Van 01b]   H. L. Van Trees, Detection, Estimation, and Modulation Theory, Part III: Radar-Sonar Signal Processing and Gaussian Signals in Noise, New York, NY: John Wiley and Sons, 1971. [Republished by Wiley-Interscience, 2001.]

[Van 03]     H. L. Van Trees, Detection, Estimation, and Modulation Theory, Part II: Nonlinear Modulation Theory, New York, NY: John Wiley and Sons, 1971. [Republished by Wiley-Interscience, 2003.]

[Van 02]     H. L. Van Trees, Optimum Array Processing, Part IV of Detection, Estimation, and Modulation Theory, New York, NY: John Wiley & Sons, Inc., 2002.

[VBW 06]   H. L. Van Trees, K. L. Bell, and Y. Wang, ``Bayesian Cramér-Rao Bounds for Multistatic Radar,'' IEEE International Waveform Diversity & Design Conference, January 2006.

[Wei 88]     E. Weinstein, ``Relations between Belini-Tartara, Chazan-Zakai-Ziv, and Wax-Ziv lower bounds,'' IEEE Trans. Info. Theory, vol. 34, no. 2, pp. 342-343, Mar. 1988.

[WW 88]    E. Weinstein and A. J. Weiss, ``A General Class of Lower Bounds in Parameter Estimation,'' IEEE Trans. Info. Theory, vol. 34, no. 2, pp. 338-342, Mar. 1988.

[WW 85]     A. Weiss and E. Weinstein, ``A Lower Bound on the Mean Square Error in Random Parameter Estimation,'' IEEE Trans. Info. Theory, vol. 31, no. 5, pp. 680-682, 1985.

[Wie 49]     N. Wiener, The Extrapolation, Interpolation, and Smoothing of Stationary Time Series, John Wiley & Sons, Inc., New York: 1949.

[Wie 58]     N. Wiener, Nonlinear Problems in Random Theory, New York, NY: The Technology Press of MIT and John Wiley & Sons, Inc., 1958.

[ZZ 69]        J. Ziv and M. Zakai, ``Some Lower Bounds on Signal Parameter Estimation,'' IEEE Trans. Info. Theory, vol. IT-15, no. 3, pp. 386-391, May 1969.

Last updated: 08/29/07