Fisher information matrix


Given a statistical model {f𝐗(π’™βˆ£πœ½)} of a random vector X, the Fisher information matrixMathworldPlanetmath, I, is the varianceMathworldPlanetmath of the score functionMathworldPlanetmath U. So,

I=Var⁑[U].

If there is only one parameter involved, then I is simply called the Fisher information or information of f𝐗(π’™βˆ£ΞΈ).

Remarks

  • β€’

    If f𝐗(π’™βˆ£πœ½) belongs to the exponential family, I=E⁑[UT⁒U]. Furthermore, with some regularity conditions imposed, we have

    I=-E⁑[βˆ‚β‘Uβˆ‚β‘πœ½].
  • β€’

    As an example, the normal distributionMathworldPlanetmath, N⁒(ΞΌ,Οƒ2), belongs to the exponential family and its log-likelihood functionMathworldPlanetmath β„“(𝜽∣x) is

    -12⁒ln⁑(2⁒π⁒σ2)-(x-ΞΌ)22⁒σ2,

    where 𝜽=(ΞΌ,Οƒ2). Then the score function U⁒(𝜽) is given by

    (βˆ‚β‘β„“βˆ‚β‘ΞΌ,βˆ‚β‘β„“βˆ‚β‘Οƒ2)=(x-ΞΌΟƒ2,(x-ΞΌ)22⁒σ4-12⁒σ2).

    Taking the derivative with respect to 𝜽, we have

    βˆ‚β‘Uβˆ‚β‘πœ½=(βˆ‚β‘U1βˆ‚β‘ΞΌβˆ‚β‘U2βˆ‚β‘ΞΌβˆ‚β‘U1βˆ‚β‘Οƒ2βˆ‚β‘U2βˆ‚β‘Οƒ2)=(-1Οƒ2-x-ΞΌΟƒ4-x-ΞΌΟƒ412⁒σ4-(x-ΞΌ)2Οƒ6).

    Therefore, the Fisher information matrix I is

    -E⁑[βˆ‚β‘Uβˆ‚β‘πœ½]=12⁒σ4⁒(2⁒σ200-1).
  • β€’

    Now, in linear regression model with constant variance Οƒ2, it can be shown that the Fisher information matrix I is

    1Οƒ2⁒𝐗T⁒𝐗,

    where X is the design matrix of the regression model.

  • β€’

    In general, the Fisher information meansures how much β€œinformation” is known about a parameter ΞΈ. If T is an unbiased estimatorMathworldPlanetmath of ΞΈ, it can be shown that

    Var⁑[T⁒(X)]β‰₯1I⁒(ΞΈ)

    This is known as the Cramer-Rao inequality, and the number 1/I⁒(θ) is known as the Cramer-Rao lower bound. The smaller the variance of the estimate of θ, the more information we have on θ. If there is more than one parameter, the above can be generalized by saying that

    Var⁑[T⁒(X)]-I⁒(𝜽)-1

    is positive semidefinite, where I is the Fisher information matrix.

Title Fisher information matrix
Canonical name FisherInformationMatrix
Date of creation 2013-03-22 14:30:15
Last modified on 2013-03-22 14:30:15
Owner CWoo (3771)
Last modified by CWoo (3771)
Numerical id 14
Author CWoo (3771)
Entry type Definition
Classification msc 62H99
Classification msc 62B10
Classification msc 62A01
Synonym information matrix
Defines Fisher information
Defines information
Defines Cramer-Rao inequality
Defines Cramer-Rao lower bound