How would you approach selecting the appropriate evaluation metrics for a classification model?

During my final year project at university, I developed a sentiment analysis model to classify movie reviews as positive or negative. To evaluate the model, I first considered the dataset's balance; since it had more positive reviews, accuracy alone would have been misleading. Instead, I opted to use precision and recall as my key metrics. Precision helped me understand how many of the reviews I classified as positive were actually positive, while recall measured how many actual positive reviews I successfully identified. This dual focus allowed me to fine-tune my model effectively, leading to an F1-score improvement of 15% over the basic accuracy approach.

In a volunteer role at a local nonprofit, I helped analyze donor profiles to classify them into different segments for fundraising campaigns. I realized that some segments had a higher priority for our outreach efforts. Therefore, I chose metrics like recall to ensure we reached as many potential major donors as possible. By tracking the number of correct classifications of high-potential donors, we improved our targeted campaigns significantly, resulting in a 20% increase in donations compared to previous efforts. This experience taught me the importance of aligning evaluation metrics with organizational goals.

In my first job as a data analyst, I worked on a fraud detection model for an e-commerce platform. Given the high cost of false negatives, I prioritized metrics like recall and the F1-score. During the initial phases of the project, I noticed that while our accuracy was high, we were missing a significant number of fraudulent transactions. By focusing on improving recall, we managed to reduce fraud losses by over 30% within the first quarter, demonstrating the effectiveness of selecting metrics that resonate with business priorities.