10-01: Exercises

Question

An IDS was tested against 1,000 labeled network connections (500 actual attacks, 500 actual normal traffic). It produced this confusion matrix:

	Predicted: Attack	Predicted: Normal
Actual: Attack	440 (TP)	60 (FN)
Actual: Normal	30 (FP)	470 (TN)

1. Compute accuracy, precision, recall, and F1-score.
2. Interpret what the 60 false negatives and 30 false positives mean operationally for a security team.
3. If the team could tune the model to trade some precision for higher recall, would that generally be the right call here? Why or why not?

---

Solution

Step 1: The formulas and the arithmetic

Per 10-07: Evaluating Detection Models:

Accuracy = (TP + TN) / Total

= (440 + 470) / 1000
= 910 / 1000
= 0.91  →  91%

Precision = TP / (TP + FP) — "of everything flagged as an attack, how much really was?"

= 440 / (440 + 30)
= 440 / 470
≈ 0.936  →  93.6%

Recall = TP / (TP + FN) — "of all the real attacks, how many did we actually catch?"

= 440 / (440 + 60)
= 440 / 500
= 0.88  →  88%

F1-score = 2 × (Precision × Recall) / (Precision + Recall)

= 2 × (0.936 × 0.88) / (0.936 + 0.88)
= 2 × 0.8237 / 1.816
≈ 0.907  →  ~90.7%

Step 2: Interpreting the errors operationally

• 60 false negatives (actual attacks predicted as normal): these are missed attacks — the most dangerous kind of error for a security system. An attacker's traffic sailed through undetected, and the security team has no idea it happened at all.
• 30 false positives (actual normal traffic predicted as attacks): these are false alarms. An analyst has to investigate each one, confirm it's benign, and move on — wasted time and, at scale, "alert fatigue" that can cause real alerts to get deprioritized or ignored.

Step 3: Should the team trade precision for recall?

In most intrusion-detection contexts, yes, generally — recall usually matters more than precision, because the cost of the two error types is asymmetric:

• A false positive costs an analyst some investigation time — annoying, but bounded and recoverable.
• A false negative can mean a real, ongoing intrusion goes completely undetected, with potentially unbounded damage (data theft, ransomware, lateral movement) before it's ever discovered by other means.

Given that asymmetry, a security team would often accept somewhat more false alarms (lower precision) in exchange for catching more real attacks (higher recall) — as long as the resulting alert volume doesn't overwhelm the analysts' capacity to actually investigate them (a real, practical limit in production). This exact reasoning is why 10-07: Evaluating Detection Models highlights recall as often the priority metric for security-critical detection systems.

---

Final Answer

• Accuracy = 91%, Precision ≈ 93.6%, Recall = 88%, F1 ≈ 90.7%.
• The 60 false negatives are missed attacks (the more dangerous error); the 30 false positives are false alarms costing analyst time.
• Generally yes — trading some precision for higher recall is usually the right call in intrusion detection, since missing a real attack is typically far more costly than investigating an extra false alarm, as long as alert volume stays manageable.