Class-Aware Knowledge Distillation for Detecting Rare Attacks with Tiny IoT Intrusion Detection Models

Abstract

Knowledge distillation (KD) is a common way to shrink an intrusion detection model until it fits on an Internet-of-Things (IoT) edge device. Accuracy is usually the only number reported after distillation, and on heavily imbalanced traffic it is dominated by flooding attacks. We show that this can hide a large loss on rare attack classes. On CIC-IoT2023, Edge-IIoTset and ToN-IoT, a 5–9K-parameter student distilled from a transformer teacher with standard KD loses 11.8, 26.4 and 65.1 points of minority-class recall relative to its teacher. On ToN-IoT, recall on ransomware falls from 69.2% to 0% while overall accuracy stays above 95%. Splitting this gap shows that the capacity gap between teacher and student is about 11 points on every dataset, while the loss caused by distillation itself, measured against the same student trained without a teacher, is 0.3, 14.9 and 54.3 points for KD at the setting we tested. We trace the effect to the teacher’s soft labels, which inherit the class prior, and propose Minority-Aware KD (MA-KD): a distillation term reweighted per class and computed against prior-debiased teacher logits, which we relate formally to Balanced Knowledge Distillation. MA-KD raises minority recall over standard KD by 10.8, 16.6 and 73.5 points and macro-F1 by 2.0, 11.1 and 15.6 points, with 95% intervals that exclude zero on every dataset. An ablation attributes the gain to class reweighting and prior debiasing; a per-class temperature schedule we also tested has no measurable effect. We also report a negative result: a student trained alone with class-weighted cross-entropy is at least as good as MA-KD, with higher macro-F1 on all three datasets and higher minority recall on Edge-IIoTset, so at this student size the teacher repairs the damage done by distillation but adds little beyond a well-configured imbalance-aware loss. The student occupies 21 KB, runs in 14 µs per flow on one cloud CPU thread, and is 41–103 times faster than the teacher. A cross-dataset study on a 13-feature harmonized schema fails for every model, which we report as a limitation of current IoT benchmarks.