The Annals of Statistics

Why Least Squares and Maximum Entropy? An Axiomatic Approach to Inference for Linear Inverse Problems

Imre Csiszar

Full-text: Open access


An attempt is made to determine the logically consistent rules for selecting a vector from any feasible set defined by linear constraints, when either all $n$-vectors or those with positive components or the probability vectors are permissible. Some basic postulates are satisfied if and only if the selection rule is to minimize a certain function which, if a "prior guess" is available, is a measure of distance from the prior guess. Two further natural postulates restrict the permissible distances to the author's $f$-divergences and Bregman's divergences, respectively. As corollaries, axiomatic characterizations of the methods of least squares and minimum discrimination information are arrived at. Alternatively, the latter are also characterized by a postulate of composition consistency. As a special case, a derivation of the method of maximum entropy from a small set of natural axioms is obtained.

Article information

Ann. Statist., Volume 19, Number 4 (1991), 2032-2066.

First available in Project Euclid: 12 April 2007

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier


Primary: 62A99: None of the above, but in this section
Secondary: 68T01: General 94A17: Measures of information, entropy 92C55: Biomedical imaging and signal processing [See also 44A12, 65R10, 94A08, 94A12]

Image reconstruction linear constraitns logically consistent inference minimum discrimination information nonlinear projection nonsymmetric distance selection rules


Csiszar, Imre. Why Least Squares and Maximum Entropy? An Axiomatic Approach to Inference for Linear Inverse Problems. Ann. Statist. 19 (1991), no. 4, 2032--2066. doi:10.1214/aos/1176348385.

Export citation