How do you select the appropriate evaluation metric for a machine learning model given a specific problem?

In my final year of college, I worked on a project aimed at predicting student success using machine learning. We chose to focus on predicting whether students would pass or fail a course. Given the potential for imbalanced classes, we decided to use precision and recall as our evaluation metrics instead of just accuracy. We found that focusing on precision was crucial since we wanted to minimize false positives, ensuring that we didn’t incorrectly identify students as likely to succeed. By using the F1 score to balance both precision and recall, we could better capture the model's performance in our context, ultimately leading to a model that provided actionable insights for academic advisors.

During my time volunteering for a non-profit, I was involved in developing a model to predict donor contributions. Since the goal was to identify potential high-value donors, we focused on the metric of AUC-ROC (Area Under the Receiver Operating Characteristic Curve). This metric helped us evaluate how well our model distinguished between high and low-value donors across various thresholds. By using AUC-ROC, we ensured that our model was robust and reliable, which was crucial for maximizing fundraising efforts and tailoring our outreach strategies effectively.

In my first job as a data analyst, I was tasked with building a model to predict customer churn for a subscription service. Given the business implications, we opted for recall as our primary metric because retaining customers was more critical than acquiring new ones in this scenario. By prioritizing recall, we aimed to identify as many potential churners as possible to implement retention strategies. This approach led to a successful intervention that reduced churn rates by 15%, demonstrating the impact of selecting an appropriate evaluation metric aligned with business goals.