Permutation Tests for Binary Classification Mathematisches Forschungsinstitut Oberwolfach · 2014
Partially supported by a "Research in Pairs" grant from the Mathematical Research Institute Oberwolfach, together with Ender Konukoglu.
Permutation tests are widely used for various statistical hypothesis-testing problems with small sample sizes. In these cases they can provide more accurate hypothesis testing than parametric alternatives. Binary classification problems with small sample sizes, common in neuroimaging and genomic studies, can also benefit from permutation tests.
However, full permutation tests for binary classification can be computationally costly, since the retraining cost of a learning algorithm is often very high for problems with high feature/representation dimensions – so retraining even a couple of thousand times is not desirable for many studies. Furthermore, the relationship between permutation tests and parametric approaches such as Bayesian intervals is not completely understood, making it hard to know the advantages of one test over another.
A theoretical analysis of permutation tests for binary classification was therefore needed – one that would link Bayesian intervals and permutation tests, suggesting which test to use in which circumstance, and yield approximation techniques and bounds for permutation tests to reduce their computational cost.
Publication: "Permutation tests for classification: Revisited", presented at the 2017 International Workshop on Pattern Recognition in Neuroimaging (PRNI).