Journal of Symbolic Logic

Parsimony hierarchies for inductive inference

Andris Ambainis, John Case, Sanjay Jain, and Mandayam Suraj

Full-text: Access denied (no subscription detected)

We're sorry, but we are unable to provide you with the full text of this article because we are not able to identify you as a subscriber. If you have a personal subscription to this journal, then please login. If you are already logged in, then you may need to update your profile to register your subscription. Read more about accessing full-text


Freivalds defined an acceptable programming system independent criterion for learning programs for functions in which the final programs were required to be both correct and “nearly” minimal size, i.e., within a computable function of being purely minimal size. Kinber showed that this parsimony requirement on final programs limits learning power. However, in scientific inference, parsimony is considered highly desirable. A lim-computable function is (by definition) one calculable by a total procedure allowed to change its mind finitely many times about its output. Investigated is the possibility of assuaging somewhat the limitation on learning power resulting from requiring parsimonious final programs by use of criteria which require the final, correct programs to be “not-so-nearly” minimal size, e.g., to be within a lim-computable function of actual minimal size. It is shown that some parsimony in the final program is thereby retained, yet learning power strictly increases. Considered, then, are lim-computable functions as above but for which notations for constructive ordinals are used to bound the number of mind changes allowed regarding the output. This is a variant of an idea introduced by Freivalds and Smith. For this ordinal notation complexity bounded version of lim-computability, the power of the resultant learning criteria form finely graded, infinitely ramifying, infinite hierarchies intermediate between the computable and the lim-computable cases. Some of these hierarchies, for the natural notations determining them, are shown to be optimally tight.

Article information

J. Symbolic Logic, Volume 69, Issue 1 (2004), 287-327.

First available in Project Euclid: 2 April 2004

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 68Q32: Computational learning theory [See also 68T05]

Computational learning theory minimal size program constructive ordinal notations limiting computable function


Ambainis, Andris; Case, John; Jain, Sanjay; Suraj, Mandayam. Parsimony hierarchies for inductive inference. J. Symbolic Logic 69 (2004), no. 1, 287--327. doi:10.2178/jsl/1080938842.

Export citation


  • J. W. Addison The method of alternating chains, Theory of models. $($Proceedings of the 1963 International Symposium$)$ (Berkeley, California) (J. W. Addison, Leon Henkin, and Alfred Tarski, editors), North--Holland, Amsterdam,1965, pp. 1--16.
  • A. Ambainis The power of procrastination in inductive inference: How it depends on used ordinal notations, EuroCOLT'95, Lecture Notes in Computer Science, vol. 904,1995, pp. 99--111.
  • A. Ambainis, R. Freivalds, and C. H. Smith Inductive inference with procrastination: Back to definitions, Fundamenta Informaticae, vol. 40 (1999), pp. 1--16.
  • A. Ambainis, S. Jain, and A. Sharma Ordinal mind change complexity of language identification, Theoretical Computer Science, vol. 220 (1999), no. 2, pp. 323--343.
  • D. Angluin Finding patterns common to a set of strings, Journal of Computer and System Sciences, vol. 21 (1980), pp. 46--62.
  • K. Aps\=\itis Derived sets and inductive inference, Algorithmic learning theory, Proceedings of the 4th International Workshop on Analogical and Inductive Inference (AII'94) and the 5th International Workshop on Algorithmic Learning Theory (ALT'94), October 10--15, 1994 (Reinhardsbrunn Castle, Germany) (Setsuo Arikawa and Klaus P. Jantke, editors), Lecture Notes in Artificial Intelligence, vol. 872, Springer-Verlag,1994, pp. 26--39.
  • S. Arikawa, T. Shinohara, and A. Yamamoto Learning elementary formal systems, Theoretical Computer Science, vol. 95 (1992), pp. 97--113.
  • L. Behounek Ordinal calculator,1997, Web document at: \ttfamily
  • L. Blum and M. Blum Toward a mathematical theory of inductive inference, Information and Control, vol. 28 (1975), pp. 125--155.
  • M. Blum A machine independent theory of the complexity of recursive functions, Journal of the ACM, vol. 14 (1967), pp. 322--336.
  • I. Bratko and S. Muggleton Applications of inductive logic programming, Communications of the ACM, vol. 38 (1995), no. 11, pp. 65--70.
  • A. Brazma, E. Ukkonen, and J. Vilo Discovering unbounded unions of regular pattern languages from positive examples, Proceedings of the Seventh International Symposium on Algorithms and Computation (ISAAC'96) (Osaka, Japan), Lecture Notes in Computer Science, vol. 1178,1996, pp. 95--104.
  • J. Case Periodicity in generations of automata, Mathematical Systems Theory, vol. 8 (1974), pp. 15--32.
  • J. Case, S. Jain, S. Kaufmann, A. Sharma, and F. Stephan Predictive learning models for concept drift, Theoretical Computer Science, vol. 268 (2001), pp. 323--349, Special Issue for ALT'98.
  • J. Case, S. Jain, S. Lange, and T. Zeugmann Incremental concept learning for bounded data mining, Information and Computation, vol. 152 (1999), pp. 74--110.
  • J. Case, S. Jain, and A. Sharma Machine induction without revolutionary changes in hypothesis size, Information and Computation, vol. 128 (1996), pp. 73--86.
  • J. Case, S. Jain, and M. Suraj Not-so-nearly-minimal-size program inference, Algorithmic learning for knowledge-based systems (Klaus P. Jantke and Steffen Lange, editors), Lecture Notes in Artificial Intelligence, vol. 961, Springer-Verlag,1995, pp. 77--96.
  • J. Case and C. Smith Comparison of identification criteria for machine inductive inference, Theoretical Computer Science, vol. 25 (1983), pp. 193--220.
  • J. Case and M. Suraj Characterizing Ershov hierarchies by algorithmic $O$-count down, Working paper,2003.
  • K. Chen Tradeoffs in inductive inference of nearly minimal sized programs, Information and Control, vol. 52 (1982), pp. 68--86.
  • R. L. Epstein, R. Haas, and R. L. Kramer Hierarchies of sets and degrees below $\bf 0'$, Logic year 1979--80, Lecture Notes in Mathematics, vol. 859, Springer--Verlag, Heidelberg,1981, pp. 32--48.
  • Yu. L. Ershov A hierarchy of sets, I, Algebra i Logika, vol. 7 (1968), no. 1, pp. 47--74, In Russian (English translation in Algebra and Logic, vol. 7 (1968), pp. 25--43.
  • R. Freivalds Minimal Gödel numbers and their identification in the limit, Proceedings of the 4th Symposium on Mathematical Foundations of Computer Science (Jirí Becvár, editor), Lecture Notes in Computer Science, vol. 32,1975, pp. 219--225.
  • R. Freivalds and C. Smith On the role of procrastination in machine learning, Information and Computation, vol. 107 (1993), no. 2, pp. 237--271.
  • R. Freivalds and R. Wiehagen Inductive inference with additional information, Electronische Informationverarbeitung und Kybernetik, vol. 15 (1979), pp. 179--195.
  • M. Fulk A study of inductive inference machines, Ph.D. thesis, SUNY at Buffalo,1985.
  • A. Gale and R. Downey On genericity and Ershov's hierarchy, Mathematical Logic Quarterly, vol. 47 (2001), no. 2, pp. 161--182.
  • W. I. Gasarch and C. H. Smith Recursion theoretic models of learning: some results and intuitions, Annals of Mathematics and Artificial Intelligence, vol. 15 (1995), no. 2, pp. 151--166.
  • E. Gold Language identification in the limit, Information and Control, vol. 10 (1967), pp. 447--474.
  • S. Jain, D. Osherson, J. Royer, and A. Sharma Systems that learn: An introduction to learning theory, second ed., MIT Press, Cambridge, Mass.,1999.
  • S. Jain and A. Sharma Elementary formal systems, intrinsic complexity, and procrastination, Information and Computation, vol. 132 (1997), pp. 65--84.
  • C. G. Jockusch, Jr. Review of ``Hierarchies of Sets and Degrees Below $\textbf0'$'' by Richard L. Epstein, Richard Haas, and Richard L. Kramer, Mathematical Reviews,(1982), MR 82k:03073.
  • P. Kilpeläinen, H. Mannila, and E. Ukkonen MDL learning of unions of simple pattern languages from positive examples, Second European Conference on Computational Learning Theory (Paul Vitányi, editor), Lecture Notes in Artificial Intelligence, vol. 904, Springer-Verlag,1995, pp. 252--260.
  • E. Kinber On the synthesis in the limit of almost minimal Gödel numbers, Theory of algorithms and programs, vol. 1, Latvian State University, Riga,1974, pp. 221--223.
  • S. C. Kleene On notation for ordinal numbers, Journal of Symbolic Logic, vol. 3 (1938), pp. 150--155.
  • K. Kuratowski and A. Mostowski Set theory, North-Holland,1967.
  • N. Lavrač and S. Džeroski Inductive logic programming: Techniques and applications, Ellis Horwood,1994.
  • M. Machtey and P. Young An introduction to the general theory of algorithms, North Holland, New York,1978.
  • T. Mitchell Machine learning, McGraw Hill,1997.
  • S. Muggleton and L. De Raedt Inductive logic programming: Theory and methods, Journal of Logic Programming, vol. 19/20 (1994), pp. 669--679.
  • R. Nix Editing by examples, Technical Report 280, Department of Computer Science, Yale University, New Haven, CT, USA,1983.
  • P. Odifreddi Classical recursion theory, vol. II, Elsevier, Amsterdam,1999.
  • H. Putnam Probability and confirmation, Voice of America, Forum on Philosophy of Science, 10, 1963.
  • H. Rogers Gödel numberings of partial recursive functions, Journal of Symbolic Logic, vol. 23 (1958), pp. 331--341.
  • J. Royer A connotational theory of program structure, Lecture Notes in Computer Science, vol. 273, Springer-Verlag, Berlin,1987.
  • G. Sacks Higher recursion theory, Springer-Verlag,1990.
  • A. Salomaa Patterns (The Formal Language Theory Column), The Bulletin for the European Association for Theoretical Computer Science, vol. 54 (1994), pp. 46--62.
  • M. Schaefer A guided tour of minimal indices and shortest descriptions, Archive for Mathematical Logic, vol. 18 (1998), pp. 521--548.
  • V. L. Selivanov On a hierarchy of limiting computations, Sibirskii Mathematicheskii Zhurnal, vol. 25 (1984), no. 5, pp. 146--156, In Russian (English translation in Siberian Mathematical Journal, vol. 25 (1984), pp. 798--806.
  • N. Shapiro Review of ``Limiting recursion'' by E. M. Gold and ``Trial and error predicates and the solution to a problem of Mostowski'' by H. Putnam, Journal of Symbolic Logic, vol. 36 (1971), p. 342.
  • A. Sharma, F. Stephan, and Y. Ventsov Generalized notions of mind change complexity, Proceedings of the Tenth Annual Conference on Computational Learning Theory, ACM Press,1997, pp. 96--108.
  • S. Shimozono, A. Shinohara, T. Shinohara, S. Miyano, S. Kuhara, and S. Arikawa Knowledge acquisition from amino acid sequences by machine learning system BONSAI, Transactions of the Information Processing Society of Japan, vol. 35 (1994), pp. 2009--2018.
  • T. Shinohara Inferring unions of two pattern languages, Bulletin of Informatics and Cybernetics, vol. 20 (1983), pp. 83--88.
  • T. Shinohara and A. Arikawa Pattern inference, Algorithmic learning for knowledge-based systems (Klaus P. Jantke and Steffen Lange, editors), Lecture Notes in Artificial Intelligence, vol. 961, Springer-Verlag,1995, pp. 259--291.
  • W. Sierpinski Cardinal and ordinal numbers, second revised ed., PWN--Polish Scientific Publishers,1965.
  • R. Smullyan Theory of formal systems, Annals of Mathematics Studies, no. 47, Princeton University Press,1961.
  • R. I. Soare Computability and recursion, Bulletin of Symbolic Logic, vol. 2 (1996), no. 3, pp. 284--321.
  • K. Wright Identification of unions of languages drawn from an identifiable class, Proceedings of the Second Annual Workshop on Computational Learning Theory (Santa Cruz, California) (R. Rivest, D. Haussler, and M. Warmuth, editors), Morgan Kaufmann Publishers, Inc.,1989, pp. 328--333.