Pooled and Compromised: A Leakage Audit of a Popular MRI Benchmark

The Kaggle “Brain Tumor MRI Dataset” (Nick- parvar) is a saturated benchmark, with recent work reporting 97–99.9% accuracy under naive, patient-agnostic splits. It pools images from multiple sources without retaining patient identifiers, so near-duplicate slices from the same scan can land on both sides of a random split, inflating accuracy. We attempt to recover true patient identity for the Figshare-derived portion (glioma, meningioma, pituitary) by matching 7,200 pooled Kaggle images against the original 3,064 patient-labeled Figshare images, using a two-stage pipeline: normalized perceptual-hash retrieval with SSIM confirmation, then an ORB feature-matching fallback. This recovers a patient ID for 527 of 5,400 eligible images (9.8% coverage), a 49-fold improvement over a single-stage baseline (0.2%), though coverage remains far below what the dataset’s documented provenance implies, and only 52.2% class- label agreement among recovered images suggests many ORB- fallback matches are incorrect. We combine these identities with near-duplicate clustering over the remaining pool (24.7% multi- image clusters) to build a group-aware, leakage-corrected split covering all 7,200 images. Retraining three CNNs (ResNet50, EfficientNet-B0, DenseNet121) under both protocols shows the naive split overstates test accuracy by 0.93–1.59 points per architecture. A soft-voting ensemble reaches 98.06% accuracy under the corrected protocol, significantly better than the best single architecture, EfficientNet-B0, at 97.26% (McNemar’s test, p = 0.012), reversing a null ensembling result from a weaker, purely clustering-driven correction in an earlier pass. We release the corrected split, code, and all figures/tables for reuse.