Do you know the difference between Precision and Recall?

The simple way to think about this is that Precision is about your Prediction, and Recall is about Reality.

If you run a police department and your job was to identify thieves, precision asks the question, of all the people you identified as thieves, how many were really thieves? Recall asks, of all the thieves out there, how many did you correctly identify?

Your high precision model is really good at correctly predicting the modelled class i.e. if the police department has 100% precision, when they identify someone as a thief, don’t pass go, don’t collect 200, because that person is a thief. On the other hand, if the police department has very low precision, then every time they identify a thief, we all roll our eyes and ignore them.

What if the mayor of the city isn’t happy with the crime rate in the city?

A classifier could have high precision, but still be a poor classifier. Let’s say there were 100 people in a room and 10 of them were thieves. If the police department says there is only thief in the room, and that person is really a thief, they would have 100% precision because of all the people they identified as thieves (1 person), all of them (1 person) were thieves. However, their recall would be 10% because they identified 1 in 10 thieves.

So whilst the police department is excellent at identifying the most obvious thieves, there are still a lot of thieves out there wreaking havoc in the city. So what the mayor really needs is a police department that has high precision (catching real thieves) and high recall (catching lots of thieves).

We want classifiers that run the world

A classifier with high precision and high recall is the ultimate dream for every data scientist. But just like I’d like to sing like Beyonce or Taylor Swift, I have to accept another reality.

Low precision and low recall classifiers are pointless, might as well guess randomly.

In reality you are probably dealing with a high precision / low recall classifier OR a low precision / high recall classifier. Depending on your use-case, this may be good enough for what you are trying to achieve.

High Precision / Low Recall Classifier

This classifier is not making a lot of positive predictions, but when it does, it’s predictions are usually correct. This is useful when the risk of getting a prediction wrong greatly outweighs the risk of not identifying all the targets.

An example could be a military system to identify and destroy enemy targets. In this case, you probably care more about precision (correctly identifying an enemy target) and you are willing to accept that you won’t catch all enemy targets because you want to minimize the risk of destroying civilian infrastructure.

Low Precision / High Recall Classifier

This is the better-safe-than-sorry classifier. In the thief example from earlier, this classifier is basically walking around calling everyone a thief. This is the classifier that is most likely going to be a nuisance to the users, so you want to understand at what point it is more annoying than useful.

Real life examples would include fraud detection, it will probably mark a lot of transactions as fraud. Safe for the bank, annoying for customers. Another example could be cancer detection, people who don’t have cancer will end up getting treatment.

Summary

• If you have a high precision / high recall classifier, life is good. Share any tips with us.
• If you have a low precision / low recall classifier, decommission it and perhaps use some business rules to solve the problem.
• If you have either a high precision / low recall OR low precision / high recall classifier, then think about the business problem and decide whether the performance of the classifier is a problem or not. If you still have a problem, then think about generating more relevant features to help your classifier in differentiating between the groups in your data.