Confidence-Aware Retrieval-Augmented Generation with Abstention for Cyber Threat Intelligence

Retrieval-Augmented Generation (RAG) grounds
language models in external knowledge, which makes it a
natural tool for Cyber Threat Intelligence (CTI). It has one
dangerous habit, however: it answers every question, even when
the retrieved evidence does not support an answer. The result
is hallucinated vulnerability details that can mislead a security
analyst. We present a confidence-aware RAG framework for
retrieving CVE-based threat intelligence. The framework adds a
tunable abstention step. When its confidence in the retrieval is too
low, the system declines to answer instead of guessing. We measure
this confidence at the retrieval stage from two simple signals: how
similar the best-matching record is to the query, and how far
ahead it is of the second-best record. The measure needs no access
to the language model’s internals. We evaluate three systems,
Naive RAG, Hybrid RAG, and our Confidence-Aware RAG, on
200 keyword-based queries over the 2023 National Vulnerability
Database (30,932 CVE records). At a confidence threshold of
0.20, our system answers with 81.3% precision, against 70.0%
for a baseline that always answers, and it lowers the Expected
Calibration Error from 0.309 to 0.259. In short, for security-
critical CTI work, a few trustworthy answers are worth more
than many unreliable ones.