What you are looking at
Each dot is one of 200 examples: the 50 that are really positive in the top row, the 150 that are really negative in the bottom row. Across is the score a classifier gave each one, from 0 to 1. The scores are made up for this lab, drawn from two overlapping bell curves: the classes mostly apart, mixed in the middle.
The threshold turns scores into decisions. A score at or above it is predicted positive (the shaded side), below it negative. That gives four outcomes, the four cells of the confusion matrix: true positives (positives flagged), false negatives (positives missed), false positives (negatives flagged: false alarms) and true negatives (negatives rightly passed). A dot is filled when it is flagged and a ring when it isn't, green when the call is right and red when it is wrong.
The two small curves trace every threshold at once, and the indigo dot on each marks the current one.
Precision against recall
Both read off the matrix. Precision looks down the predicted-positive column: of everything flagged, what share is really positive? TP ÷ (TP + FP). Recall looks along the actually-positive row: of the real positives, what share got flagged? TP ÷ (TP + FN). Recall is also called sensitivity, or the true positive rate.
They pull against each other. At 0.50 the classifier catches 40 of the 50 positives (recall 80%) and raises 21 false alarms (precision 65.6%). Slide up to 0.77: precision reaches 95.2%, but recall falls to 40%. Slide down to 0.30: recall reaches 96%, and precision falls to 39.3%. Raising the threshold can only lose positives, so recall never rises with it. Precision usually rises, but not always: moving past a positive loses a true positive while any false alarms scored above it stay.
Accuracy counts every right call, (TP + TN) ÷ everything. The false positive rate, FP ÷ (FP + TN), is the share of negatives wrongly flagged, and the across axis of the ROC curve.
Choosing a threshold
There is no right threshold in general. It depends on what each kind of mistake costs.
- Spam filter: a false positive sends a real email to the spam folder, where nobody looks; a spam that gets through costs a moment's annoyance. So favour precision. Press Spam filter and the threshold rises to 0.77, the lowest at which 95% of what it flags is spam. It lets 60% of the spam through to keep real mail safe.
- Cancer screening: a missed cancer goes untreated; a false alarm costs a second test. So favour recall. Press Cancer screening and the threshold drops to 0.30, which catches 48 of the 50 cases. The price is 74 false alarms: only 39% of the people flagged have cancer.
When you can put a number on each error, choose the threshold with the lowest expected cost: the false positives times what each costs, plus the false negatives times theirs. Choose it on validation data, not on the data the model was trained on.
F1 is the harmonic mean of precision and recall, which works out as 2TP ÷ (2TP + FP + FN). It is high only when both are, which makes it the usual single number when the errors can't be priced. It misleads in three ways. It weighs a false positive exactly like a false negative, which neither case above does (F-beta tilts it: F2 favours recall, F0.5 precision). True negatives appear nowhere in it. And it moves with the share of positives: drag the threshold to 0 and a classifier that flags everything still scores an F1 of 40%, because a quarter of the examples are positive.
Rare positives
Tick Rare positives. Each dot in the bottom row now stands for 33 negatives: 50 positives against 4,950 negatives, 1 in 100, the kind of balance fraud detection and screening deal with. The classifier and its scores are exactly the same.
Drag the threshold to 1.00, where nothing is flagged: accuracy reads 99.0% and recall 0%. A model that never finds a positive looks excellent by accuracy. At 0.90 it scores 99.1% while catching 3 positives in 50. This is the accuracy paradox: when one class dominates, accuracy mostly measures the balance.
Precision falls hardest. At 0.50 the classifier still catches 40 of the 50 positives, but the same 14% false positive rate now means 693 false alarms, so precision drops from 65.6% to 5.5%. F1 falls with it, from 72.1% to 10.2%, though nothing about the classifier changed.
The ROC curve doesn't move, and its AUC stays 0.90: both its axes are rates within one class (recall within the positives, the false positive rate within the negatives), so the balance between classes cancels out. The PR curve sinks, and its average precision falls from 0.80 to 0.32, because precision mixes the two classes. That is why the PR curve is the more honest picture when positives are rare: a ROC AUC of 0.90 can sit on top of a classifier whose flags are mostly wrong.
In an interview
Questions this lab prepares you for:
- Define precision and recall from the confusion matrix. Which matters more for a spam filter, and which for cancer screening?
- How do you choose a classification threshold? (By the cost of each kind of error, or a target such as 95% recall, picked on validation data.)
- What is F1, and when is it a poor choice?
- Your fraud model is 99.5% accurate. Is it good? (Not on that alone. If 0.5% of transactions are fraud, flagging nothing scores the same. Ask for precision and recall.)
- What does ROC AUC measure? (The chance that a randomly chosen positive scores higher than a randomly chosen negative.)
- When would you use a PR curve rather than a ROC curve? (When positives are rare and finding them is the point.)
The code
Precision, recall and F1 at one threshold, and the PR curve over all of them, in NumPy, with y_true holding 1 for a positive and 0 for a negative. pr_curve walks the examples from the highest score down, counting as it goes; last marks the end of each run of equal scores, since no threshold can split a tie. In practice scikit-learn's precision_recall_curve, roc_curve and roc_auc_score do this for you.
import numpy as np def at_threshold(y_true, scores, threshold): predicted = scores >= threshold tp = np.sum(predicted & (y_true == 1)) fp = np.sum(predicted & (y_true == 0)) fn = np.sum(~predicted & (y_true == 1)) precision = tp / (tp + fp) if tp + fp > 0 else float("nan") recall = tp / (tp + fn) f1 = 2 * tp / (2 * tp + fp + fn) return precision, recall, f1 def pr_curve(y_true, scores): order = np.argsort(-scores, kind="stable") scores, y = scores[order], y_true[order] last = np.append(np.diff(scores) != 0, True) tp = np.cumsum(y)[last] fp = np.cumsum(1 - y)[last] return tp / tp[-1], tp / (tp + fp), scores[last]