How would you approach selecting the appropriate evaluation metric for a classification model?

In my final year project, I worked on a sentiment analysis tool that classified tweets as positive, negative, or neutral. Initially, I used accuracy as my evaluation metric. However, after discussing with my professor, I realized that the dataset was imbalanced, with many more positive tweets than negative ones. To improve my evaluation, I shifted to using F1-score, which gave me a better balance between precision and recall. This change helped me fine-tune the model to better capture negative sentiments, ultimately improving its effectiveness in real-world applications.

During my time volunteering for a nonprofit fundraising campaign, I helped analyze the effectiveness of our outreach emails. We initially measured success by the number of clicks, but later, we recognized that clicks alone didn’t reflect our goal of actual donations. We decided to track the conversion rate instead, which measured the percentage of recipients who made a donation after clicking through. This shift in focus allowed us to better understand our outreach effectiveness and led to a more targeted fundraising approach, ultimately increasing our donations significantly.

In my internship at a SaaS company, I was tasked with predicting customer churn. I started by evaluating the model using accuracy, but I quickly learned that our dataset had a high percentage of retained customers, which skewed the results. After discussing this with my mentor, we decided to use ROC-AUC as our evaluation metric instead, as it provided better insight into the model's performance across various thresholds. This approach enabled us to identify at-risk customers more effectively, allowing the marketing team to implement targeted retention strategies.