The performance-actionability trade-off in retention prediction at middle school

Predicting students' retention risk is one of the major trends in machine learning applications in education. While early identification of at-risk students allows timely planning and implementation of measures to prevent adverse outcomes, there is a trade-off between the predictive model's performance and the prediction window size, or model performance and its actionability. In this study, we used a dataset of 83,596 unique Portuguese students in grades 5th to 9th to predict retention at or before the end of 9th grade. We explored how different prediction window sizes impact the predictive model's performance, the feature importance, and the models' bias. The models with the shorter prediction window performed better in terms of precision, but the model with the largest prediction window showed a higher lift over the existing rule-based model. Prediction window size impacted the importance of demographic features and model's fairness. Our results contribute to the extant discussion on predicting retention, by adding empirical evidence about the models' added value in performance versus the existing practice, suggesting types of data to collect and use, and discussing education-specific challenges of responsible data science.