Minimum Detectable Improvement: Calibrating the Noise Floor of Microarray Cancer Benchmarks

Contributions.
1) We define MDI@95, a per-dataset, per-protocol resolu-
tion threshold with a direct operational reading, with a
bootstrap interval.
2) We report the floor for five widely used cohorts under
three fold counts and four learners, and release the table
so that future work can situate its margins.
3) We show the floor is set overwhelmingly by partition
choice rather than model stochasticity, that it falls with
both cohort size and fold count, and that the 3-fold
CuMiDa reference protocol sits at the worst end of that
range.
4) We demonstrate the consequence: 61 % of pairwise
comparisons among four standard learners have margins
below their own floor, and on one cohort the identity of
the winning method changes with the seed.
5) We identify a saturated cohort on which every learner
is perfect on every seed, and which therefore has no
capacity to discriminate methods.