Bruce Rushing

Mixture of experts is a prediction aggregation method in machine learning that aggregates the predictions of specialized experts. This method often outperforms Bayesian methods despite the Bayesian having stronger inductive guarantees. We argue that this is due to the greater functional capacity of mixture of experts. We prove that in a limiting case, mixture of experts will have greater capacity than equivalent Bayesian methods, which we vouchsafe through experiments on non-limiting cases. Finally, we conclude that mixture of experts is a type of abductive reasoning in the Peircean sense of hypothesis construction.

At the end of Frank Ramsey’s “General Propositions and Causality” ([1929b] 1990), he offers an enigmatic footnote that briefly describes his philosophy of science as a “forecasting theory”. What he means by this and by a “forecast” is unclear. However, elsewhere in his unpublished notes, he uses the term sporadically. An examination of those notes reveals the skeleton of a behavioral theory of mind. Ramsey held that all actions are at root driven by the sum total of a person’s dispositions or habits. These habits operate in an unconscious process that produces psychological expectations about the realization of desires. When those expectations are frustrated, the violation is registered consciously to the individual as a proposition, and the offending habit is identified. Humans can then regulate and change those habits by the conscious application of logic through deliberation. The applicable logic is Ramsey’s decision theory, which aims to make beliefs probabilistically coherent by adopting the laws and chances that signify the habits people might use for guiding behavior. The outcome of this deliberation is to refashion psychological expectations as mathematical expectations on laws and chances. These mathematical expectations are forecasts, and a forecasting theory of science takes scientific theories to provide forecasts.

Ravit Dotan argues that a No Free Lunch theorem from machine learning shows epistemic values are insufficient for deciding the truth of scientific hypotheses. She argues that NFL shows that the best case accuracy of scientific hypotheses is no more than chance. Since accuracy underpins every epistemic value, non-epistemic values are needed to assess the truth of scientific hypotheses. However, NFL cannot be coherently applied to the problem of theory choice. The NFL theorem Dotan’s argument relies upon is a member of a family of theorems in search, optimization, and machine learning. They all claim to show that if no assumptions are made about a search or optimization problem or learning situation, then the best case performance of an algorithm is that of random search or random guessing. A closer inspection shows that these theorems all rely upon assigning uniform probabilities over problems or learning situations, which is just the Principle of Indifference. A counterexample can be crafted that shows that NFL cannot be coherently applied across different descriptions of the same learning situation. To avoid this counterexample, Dotan needs to privilege some description of the learning situation faced by scientists. However, this means that NFL cannot be applied since an important assumption about the problem is being made. So Dotan faces a dilemma: either NFL leads to incoherent best-case partial beliefs or it is inapplicable to the problem of theory choice. This negative result has implications for the larger debate over theory choice.