[SOLVED] Bayesian Philosophical Roots of Machine Learning

Philosophical Roots of Machine Learning

Philosophical Roots of Machine Learning
Induction
Based on John Vickers, The Problem of Induction, The Stanford Encyclopedia of Philosophy

What is Induction?
The original problem of induction can be simply put: It concerns the support or justification of inductive methods; methods that predict or infer, in Humes words, that that instances of which we have had no experience, must resemble those of which we have had experience, and that the course of nature continues always uniformly the same.

Deduction versus Induction I
(examples)
A deductive argument
No cows are grasshoppers
Daisy is a cow
Therefore, Daisy is not a grasshopper[TRUE]
An inductive argument
One million clovers have been identified and catalogued
Each of them has three leaves
Therefore, all clovers have three leaves[NOT NECESSARILY TRUE]

[From 501 things you should have learned about Philosophy]

Deduction versus Induction II
Deductive logic is demonstrably complete. The premises of an argument constructed according to the rules of this logic imply the arguments conclusion.
Not so for induction: There is no comprehensive theory of sound induction, no set of agreed upon rules that license good or sound inductive inference, nor is there a serious prospect of such a theory.
Further, induction differs from deductive proof or demonstration not only in inductions failure to preserve truth (true premises may lead inductively to false conclusions) but also in failing of monotonicity: adding true premises to a sound induction may make it unsound.

Humes Skepticism
Humes view is that the causal relations we discern in the world are uncovered by a process of induction.
Eg.: If a billiard ball hits another one which then moves, we will say that the first ball caused the movement of the second. We do so because weve seen this happen many times.
However, nothing makes these conjunctions logical necessities, which means that there is no contradiction in asserting that a billiard ball will not move the next time it is struck [even if it has moved on every previous occasion].
[From 501 things you should have learned about Philosophy]

Hume and Mill
Hume (~1739) argued that there can be no support (deductive or inductive) for this problem, which he viewed as insoluble.
J. S. Mill (~1843) asked: Why is a single instance, in some cases, sufficient for a complete induction, while in others myriads of concurring instances, without a single exception known or presumed, go such a little way towards establishing an universal proposition?
Compare: (i) Everyone seated on the bus is moving northward. (ii) Everyone seated on the bus was born on a prime numbered day of the month.

Epistemology vs Metaphysics
Two sides or facets of the problem of induction should be distinguished: The epistemological problem is to find a method for distinguishing good or reliable inductive habits from bad or unreliable habits. The second and deeper problem is metaphysical. This is the problem of distinguishing reliable from unreliable inductions.
According to Hume induction is insoluble at both the epistemological and the metaphysical level.
At the epistemological level: there can be no formula or recipe, however complex, for ruling out unreliable inductions.
At the metaphyisical level: there is no objective difference between reliable and unreliable inductions.
The insolubility at the metaphysical level is disturbing and counter-intuitive. It is explained by the fact that Hume took too restrictive a definition of induction (enumerative induction)

Popper (~1959)s views on induction
A theory of induction is superfluous. It has no function in a logic of science.

The best we can say of a hypothesis is that up to now it has been able to show its worth, and that it has been more successful than other hypotheses although, in principle, it can never be justified, verified, or even shown to be probable. This appraisal of the hypothesis relies solely upon deductive consequences (predictions) which may be drawn from the hypothesis: There is no need even to mention induction.

Probability and Induction I
Carnap (~1952) s inductive logic
Probability in Carnaps theory is a metalinguistic operator. In this context the problem of induction is to choose or to design a language appropriate to a given situation and to define a probability on this language that properly codifies inductive inference.
He came up with a taxonomy of the varieties of inductive inference:
Direct inference , Predictive inference , Inference by analogy , Inverse inference, Universal inference.

Carnaps Taxonomy I
Direct inference typically infers the relative frequency of a trait in a sample from its relative frequency in the population from which the sample is drawn.E.g.: If the incidence of lung disease among all cigarette smokers in the U.S. is 0.15, then it is reasonable to predict that the incidence among smokers in California is close to that figure.
Predictive inference is inference from one sample to another sample not overlapping the first. This, according to Carnap, is the most important and fundamental kind of inductive inference E.g.: Inferring the color of the next ball to be drawn from an urn on the basis of the frequency of balls of that color in previous draws with replacement illustrates a common sort of predictive inference.

Carnaps Taxonomy II
Inference by analogy is inference from the traits of one individual to those of another on the basis of traits that they share. E.g.: Humes famous arguments that beasts can reason, love, hate, and be proud or humble are classic instances of analogy.
Inverse inference infers something about a population on the basis of premises about a sample from that population.E.g.: The use of polls to predict election results, of controlled experiments to predict the efficacy of therapies or medications, are common examples.
Universal inference is inference from a sample to a hypothesis of universal form. Simple enumerative induction is the standard sort of universal inference. Karl Poppers objections to induction, mentioned are for the most part directed against universal inference. Popper and Carnap are less opposed than it might seem in this regard: Popper holds that universal inference is never justified. On Carnaps view it is inessential.

Probability and Induction II
Reichenbachs frequentism
On Reichenbach (~1935)s view, the problem of induction is just the problem of ascertaining probability on the basis of evidence . The conclusions of inductions are not asserted, they are posited. A posit is a statement with which we deal as true, though the truth value is unknown
His taxonomy is a bit different from Carnaps:

Reichenbachs Taxonomy
Induction by enumeration, in which an observed initial frequency is posited to hold for the limit of the sequence;
Explanatory inference, in which a theory or hypothesis is inferred from observations;
Cross induction, in which distinct but similar inductions are compared and, perhaps, corrected;
Concatenation or hierarchical assignment of probabilities.

Subjectivism and Bayesian Induction
Bruno de Finetti (19061985) is the founder of modern subjectivism in probability and induction. Subjectivism in probability identifies probability with strength of belief.
Bruno de Finetti announced that PROBABILITY DOES NOT EXIST! in the beginning paragraphs of his Theory of Probability. By this he meant to deny the existence of objective probability and to insist that probability be understood as a set of constraints on partial belief.
In particular, strength of belief is taken to be expressed in betting odds: If you will put up p dollars (where, for example, p = 0.25) to receive one dollar if the event A occurs and nothing (forfeiting the p dollars) if A does not occur, then your strength of belief in A is p.

Bayesian Induction
The central question of induction is, de Finetti says, if a prediction of frequency can be, in a certain sense, confirmed or refuted by experience. [O]ur explanation of inductive reasoning is nothing else, at bottom than the knowledge of the probability of En + 1 evaluated when the result A of [trials] E1, , En is known.
[T]he probability of En+1 evaluated when one comes to know the result A of [trials] E1, , En is not an element of an essentially novel nature (justifying the introduction of a new term, like statistical or a posteriori probability.) This probability is not independent of the a priori probability and does not replace it; it flows in fact from the same a priori judgment, by subtracting, so to speak, the components of doubt associated with the trials whose results have been obtained.

Conclusion
Induction has been and continues to be a much debated issue in Philosophy.
In this course, we will take a very pragmatic approach to induction, judging its validity using sophisticated testing approaches.
We will not return to the questions raised by the various philosophers mentioned here, although, if interested, feel free to follow up on the source on which this introduction was based.

/docProps/thumbnail.jpeg