Artificial intelligence

Scientific journal

ISSN 2710-1673

ONLINE: ISSN 2710-1681

Search by:

Year of publication
Author name
Paper title

The optimization issues of density estimation on real data

Gorbachuk V.1, Dunaievskyi M.1, Syrku A.1, Suleimanov S.1
1 VM Glushkov Institute of Cybernetics of NAS of Ukraine

Full text (PDF)

UDC: 519.8
Publication Language: Ukrainian
Stuc. intelekt. 2017; 22(3-4):106-115

Abstract: The maximum entropy approach, equivalent to the maximum likelihood approach, is often applied to estimation of density for a statitistical distribution. But such an approach produces estimate overfitting on small sets of input data. The estimate overfitting can be eiliminated by such smoothing techniques as regularization or reformulation of constraints.

Keywords: maximization, entropy, likelihood, density, distribution, Boltzmann, Giibs.

References:

  1. Gorbachuk V.M. Statisticheskoye raspoznavanie obrazov i stokhasticheskaya optimizatsiya / Raspoznavanie obrazov i analiz izobrazhenij: novye informatsionnye tekhologii. – Minsk: ITK AN BSSR, 1991. – S. 74–76.
  2. Gorbachuk V. Informatyka yak integral’na nauka // Informatsiya ta novi tekhnologii. – 1993. – No. 1. – S. 11–16.
  3. Fischer G., Ermolieva T., Ermoliev Y., van Velthuizen H.T. Sequential downscaling methods for estimation from aggregate data. – Laxenburg, Austria: IIASA, 2006. – IIASA Interim Report IR-06-002. – 12 p.
  4. Atoyev K.L., Golodnikov O.M., Gorbachuk V.M., Ermoliev Yu.M., Ermolieva T.Yu., Kiriljuk V.S., Knopov P.S. The mathematical problems of complex management and effective utilization of food, energy and water resources under increased uncertainties / Integrated management, security, and robustness. A.G .Zagorodny, Yu.M. Ermoliev, V.L. Bogdanov (eds.) – Kyiv: NAS of Ukraine, 2014. – P. 198–227.
  5. Gaivoronski A.A., Gorabchuk V.M. Interaktivnaya sistema dlya resheniya zadach / Modelirovanie planovykh raschetov i dialogovaya optimizatsiya. – K.: Znanie, 1990. – S. 17–18.
  6. Gorbachuk V. Ekonometrychne programuvannya TSP ta Eviews. – Prepr. 96-14. – K.: Institut kibernetiki NAN Ukrainy, 1996. – 24 s.
  7. Chikrii A., Denisova N., Gorbachuk V., Gromaszek K., Krivonos Y., Lytvynenko V., Matychyn I., Osypenko V., Smailova S., Wojcik W. Current problems in information and computational technologies. V. 2. W. Wojcik, J. Sikora (eds.) – Lublin: Politechnika Lubelska, 2012. – 196 p.
  8. Gorbachuk V.M., Kyrylyuk O.P. Modelirovanie ratsional’nogo rybnogo promysla v vodoemakh // Matematicheskoe modelirovanie v problemakh ratsional’nogo propodopol’zovania. – Rostov-na-Donu: RGU, 1991. – S. 11–12.
  9. Gorbachuk V.M., Dunaevskyi M.S. Analiz i zastosuvannya prykladnoi zagal’noi rivnovagy // Naukovi zapysky NaUKMA. – 2010. – T. 107. – S. 96–100.
  10. Gorbachuk V.M., Ermoliev Y.M., Ermolieva T.Y. Dvoetapna model’ ekologo-ekologichnykh rishen’ // Visnyk Odes’kogo natsional’nogo universytetu. Ekonomika. – 2016. – Т. 21. – Вип. 9. – С. 142–147.
  11. Jaynes E.T. Information theory and statistical mechanics // Physical review. – 1957. – 106(4). – P. 620–630.
  12. Jaynes E.T. Information theory and statistical mechanics / Jaynes E.T. Papers on probability, statistics, and statistical physics. R.D. Rosenkrantz (ed.) – Dordrecht, Holland: D. Reidel Publishing Company, 1983. – P. 6–16.
  13. Kapur J.N., Kesavan H.K. Entropy optimization principles with applications. – Academic Press, 1992.
  14. Berger A.L., Della Pietra S.A., Della Pietra V.J. A maximum entropy approach to natural language processing // Computational linguistics. – 1996. – 22(1). – P. 39–71.
  15. Della Pietra S., Della Pietra V., Lafferty J. Inducing features of random fields // IEEE transactions on pattern analysis and machine intelligence. – 1997. – 19(4). – P. 1–13.
  16. Gorbachuk V.M., Koshul’ko A.I., Syrku A.A. Do pryrodnykh obmezhen’ ekonomichnoi diayal’nosti / Rozvytok suchasnykh mizhanorodnykh ekonomichnykh vidnosyn: finansovo-eknomichni ta sotsial’ni chynnyky (23–24 veresnya 2016 r., Odesa). – Odesa: ONU imeni I.I. Mechnikova, 2016. – S. 125–128.
  17. Shannon C.E. A mathematical theory of communication // The Bell system technical journal. – 1948. – 27– P. 379–423, 623–656.
  18. Sergienko I.V., Shylo V.P. Zadachi diskretnoj optimizatsii. Problemy, metody resheniya, issledovaniya. – K.: Nauk.dumka, 2003. – 261 s.
  19. Kullback S. Information theory and statistics. – New York, NY: Wiley, 1959.
  20. Benes V.E. Mathematical theory of connecting networks and telephone traffic. – New York, NY: Academic Press, 1965.
  21. Shore J.E., Johnson R.W. Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy // IEEE transactions on information theory. – 1980. – 26(1). – P. 26–37.
  22. Sanov I.N. O veroyatnosti bol’shikh otklonenij sluchajnykh velichin // Matematicheskij sbornik. – 1957. – 42(84). – No 1. – С. 11–44.
  23. Van Campenhout J.M., Cover T.M. Maximum entropy and conditional probability // IEEE transactions on information theory. – 1981. – 27. – P. 483–489.
  24. Csiszar I. Sanov property, generalized I-projection and a conditional limit theorem // Annals of probability. – 1984. – 12(3). – P. 768–793.
  25. Grünwald P. Strong entropy concentration, game theory, and algorithmic randomness / COLT/EuroCOLT 2001: Proceedings of the 14-th Annual Conference on Computational Learning Theory and 5-th European Conference on Computational Learning Theory. – London: Springer-Verlag, 2001. – P. 320–336.
  26. Csiszar I. Maxent, mathematics, and information theory / Proceedings of the 15-th Internationl Workshop on Maximum Entropy and Bayesian Methods. – Dordrecht: Kluwer Academic Publishers, 1995.
  27. Skilling J. The axioms of maximum entropy / Maximum-entropy and Bayesian methods in science and engineering. Volume 1. G.J. Erickson, C.R.Smith (eds.). – Kluwer Academic Publishers, 1988. – P. 173–187.
  28. Csiszar I. Why least squares and maximum entropy? An axiomatic approach to inference for linear inverse problems // Annals of statistics. – 1991. – 19(4). – P. 2032–2066.
  29. Dudik M. Maximum entropy density estimation and modeling geographic distributions of species. PhD (Computer Science) Dissertation. – Pronceton, NJ: Princeton University, 2007. – 179 p.
  30. Hajek A. Interpretations of probability / The Stanford encyclopedia of philosophy. E.N.Zalta (ed.) – http://plato.stanford.edu/
  31. Topsoe F. Information theoretical optimization techniques // Kybernetika. – 1979. – 15(1). – P. 8–27.
  32. Cover T.M., Thomas J.A. Elements of information theory. – Wiley, 1991.
  33. Jaynes E.T. Where do we stand on maximum entropy? / The maximum entropy formalism. R.D.Levine, M.Tribus (eds.) – Cambridge, MA: MIT Press, 1978. – P. 15–118.
  34. Jaynes E.T. Where do we stand on maximum entropy? / Jaynes E.T. Papers on probability, statistics, and statistical physics. R.D. Rosenkrantz (ed.) – Dordrecht, Holland: D.Reidel Publishing Company, 1983. – P. 210–314.
  35. Lehmann E.L., Casella G. Theory of point estimation. 2-nd ed. – New York, NY: Springer-Verlag, 1998.
  36. Csiszar I. I-divergence geometry of probability distributions and minimization problems // Annals of probability. – 1975. – 3(1). – P. 146–158.
  37. Khudanpur S.P. A method of maximum entropy estimation with relaxed constraints / Proceedings of the Johns Hopkins University Language Modeling Workshop, 1995. – P. 1–17.
  38. Gorbachuk V.M., Syrku A.A., Suleimanov S.-B. Modeli analizu okhoplennya nestandartnykh danykh // Kompyuternaya matematika. – 2017. – No 1. – С. 63–72.
  39. Knopov P.S., Kasitskaya E.J. Empirical estimates in stochastic optimization and identification. – Springer Science & Business Media, 2002. – 250 p.

View full text (PDF)