Inference Theory of Learning (ITL) models are concerned with; how knowledge is interpreted; the validity of knowledge obtained; the use of prior knowledge; what knowledge can be derived at a given point given prior knowledge; how learning goals and their structure influence the learning process. This theory assumes learning is a process where agents are guided by a set goal and use past experience and knowledge to help reach that set of goals. Some alternative theories are, Computation Learning Theory (aka Statistical Learning Theory which is a simpler theory not concerned with multi-strategic learning goals) and Multi-strategy Task-adaptive learning (these models focus on the cognitive powers of the agents). ITL algorithms are at the core of many data mining techniques. However, you must remember we do not know for certain this is how humans learn so be careful when drawing parallels to the model’s behavior and human behavior. For data miners, the goal is not to model humans but to come up with power tools for analyzing patterns in data and the last thing you wish to enter when presenting results is a debate about how humans think. That is the third rail in any data mining discussion. While a data miner does not need to be an expert in theories of learning it is very useful to be at least aware of them when reading papers and evaluating new techniques.
2. Learning Goal
The learning goals are the key to any ITL process. Without a set of goals (which may include only one goal) agents (or leaner) would just be complying information with no selection or inference of that information. How the agent uses the knowledge space, which is the totality of beliefs, to benefit itself is defined in the set of learning goals. Five questions can define a learning goal.
a) What part of prior knowledge relevant?
b) What knowledge to acquire and of what form?
c) How to evaluate the knowledge?
d) When to stop learning?
3. Learning Methodology
New knowledge can enter in three forms:
a) Derived knowledge, (deductive transformation); knowledge generated by deduction from past knowledge inputs.
b) Intrinsic, (intrinsically new) knowledge; knowledge from an external source, such as a teacher or from analysis of past knowledge.
c) Pragmatically new knowledge, (managed knowledge); if the knowledge can not be obtained in space and time defined by knowledge space.
4. ITL algorithms
Examples of ITL algorithms.
Some Monte Carol techniques with a feedback loop.
Support Vector Machines (Statistical Learning Theory)