previous  home  search  LaTeX  -  PostScript  -  PDF  -  Html/Gif   contact    up    next  

Distribution of Mutual Information from Complete and Incomplete Data


Authors: Marcus Hutter and Marco Zaffalon (2002-2005)
Comments: 26 pages, 5 figures, 4 tables
Subj-class: Learning; Artificial Intelligence
ACM-class:  I.2
Reference: Computational Statistics & Data Analysis, 48:3 (2005) 633-657
Report-no: IDSIA-11-02 and cs.LG/0403025
Paper: LaTeX - PostScript - PDF - Html/Gif
Slides: PowerPoint - PDF

Keywords: Mutual information, cross entropy, Dirichlet distribution, second order distribution, expectation and variance of mutual information, feature selection, filters, naive Bayes classifier, Bayesian statistics.

Abstract: Mutual information is widely used, in a descriptive way, to measure the stochastic dependence of categorical random variables. In order to address questions such as the reliability of the descriptive value, one must consider sample-to-population inferential approaches. This paper deals with the posterior distribution of mutual information, as obtained in a Bayesian framework by a second-order Dirichlet prior distribution. The exact analytical expression for the mean, and analytical approximations for the variance, skewness and kurtosis are derived. These approximations have a guaranteed accuracy level of the order O(1/n^3), where n is the sample size. Leading order approximations for the mean and the variance are derived in the case of incomplete samples. The derived analytical expressions allow the distribution of mutual information to be approximated reliably and quickly. In fact, the derived expressions can be computed with the same order of complexity needed for descriptive mutual information. This makes the distribution of mutual information become a concrete alternative to descriptive mutual information in many applications which would benefit from moving to the inductive side. Some of these prospective applications are discussed, and one of them, namely feature selection, is shown to perform significantly better when inductive mutual information is used.

 previous  home  search  LaTeX  -  PostScript  -  PDF  -  Html/Gif   contact    up    next  

BibTeX Entry

@Article{Hutter:04mifs,
  author =       "Marcus Hutter and Marco Zaffalon",
  title =        "Distribution of Mutual Information from Complete and Incomplete Data",
  _number =       "IDSIA-11-02",
  journal =      "Computational Statistics \& Data Analysis",
  volume =       "48",
  number =       "3",
  pages =        "633--657",
  year =         "2005",
  _month =        mar,
  publisher =    "Elsevier Science",
  url =          "http://www.hutter1.net/ai/mifs.htm",
  url2 =         "http://arxiv.org/abs/cs.LG/0403025",
  ftp =          "ftp://ftp.idsia.ch/pub/techrep/IDSIA-11-02.pdf",
  categories =   "I.2.   [Artificial Intelligence]",
  keywords =     "Mutual information, cross entropy, Dirichlet distribution, second
                  order distribution, expectation and variance of mutual
                  information, feature selection, filters, naive Bayes classifier,
                  Bayesian statistics.",
  abstract =     "Mutual information is widely used, in a descriptive way, to measure the
                  stochastic dependence of categorical random variables. In order to address
                  questions such as the reliability of the descriptive value, one must consider
                  sample-to-population inferential approaches. This paper deals with the
                  posterior distribution of mutual information, as obtained in a Bayesian
                  framework by a second-order Dirichlet prior distribution. The exact analytical
                  expression for the mean, and analytical approximations for the variance,
                  skewness and kurtosis are derived. These approximations have a guaranteed
                  accuracy level of the order O(1/n^3), where n is the sample size. Leading order
                  approximations for the mean and the variance are derived in the case of
                  incomplete samples. The derived analytical expressions allow the distribution
                  of mutual information to be approximated reliably and quickly. In fact, the
                  derived expressions can be computed with the same order of complexity needed
                  for descriptive mutual information. This makes the distribution of mutual
                  information become a concrete alternative to descriptive mutual information in
                  many applications which would benefit from moving to the inductive side. Some
                  of these prospective applications are discussed, and one of them, namely
                  feature selection, is shown to perform significantly better when inductive
                  mutual information is used.",
  note =         "to appear",
}
      
 previous  home  search  LaTeX  -  PostScript  -  PDF  -  Html/Gif   contact    up    next