Journal of Advanced Robotics, Autonomous Systems and Human-Machine Interaction
Improving Reproducibility in Machine Learning: Overview, Barriers, and Drivers
Abstract
Thulani Myles Mzyece
Reproducibility remains a fundamental issue in machine learning (ML) research; in a replication study of highly cited AI papers, Gundersen et al. found that only half of the evaluable studies were reproducible to any extent [1]. This systematic review examined the multidimensionality of ML reproducibility across computational, statistical, and methodological dimensions. Important barriers as well as facilitators have been identified. The thematic analysis of peer-reviewed articles, combined with the technical documentation, allowed for the categorization of barriers into the following categories: technical barriers of nondeterminism and environmental instability; methodological barriers of poor reporting and the evaluation trap; and cultural barriers related to the incentive structure. The corresponding drivers included technical tooling ecosystems with containerization, experiment-tracking systems, standardization measures with reporting checklists, artifact-evaluation programmes, institutional interventions with educational integration, and policy requirements. These observations were summarized into an effective five-step model: protocol definition, environment specification, experiment tracking, validation protocols, and artifact preparation. The current research has shown that reproducibility enhancement is difficult to achieve without simultaneous technological, methodological, and institutional interventions.