SRN – evidence and cross-validation

A paper that has intrigued me recently is “On the marginal likelihood and cross-validation” by E. Fong and C.C.Holmes on Biometrika.

Model evidence appears as marginal likelihood or prior predictive in this paper:

As the paper seeks to establish the connection between model evidence and cross-validation (CV), which is based on predictive performance on held-out test sets, it notes that the log evidence relates to log posterior predictive probability

So log evidence can be interpreted as a predictive sequential scoring rule with score function

The paper begins by arguing that model evidence is the unique scoring rule that guarantees coherency (data ordering should not change the result of inference when data are exchangeable).

Then the authors move on to show the equivalence between evidence and cumulative cross-validation scores. When we consider leave--out CV, there are number of held-out test sets and each set has some predictive score when the rest of the data is used for training. The leave--out CV score, denoted by , is the average of these predictive scores. When the log posterior predictive probability is used as the scoring rule, then we must have

So the log evidence is also the sum of leave--out CV scores, for all values of

Author: PhyllisWithData

Statistics PhD student at Harvard University. View all posts by PhyllisWithData