 ICACT20260248 Slide.01
[Big Slide]
[YouTube] |
Chrome  Click!! |
 |
Welcome, everyone.
Thank you for taking the time to view this presentation.
This work focuses on adaptive retrieval for cyber threat intelligence summarization, with the goal of improving factual grounding in large language model outputs.
In this presentation, we begin by motivating the problem of summarizing cyber threat intelligence and the limitations of using language models in isolation.
We then introduce our key insight and adaptive retrieval approach, describe the system architecture and inference workflow, and evaluate the method using factuality and latency metrics.
Finally, we conclude with key findings and directions for future work.
|
| ICACT20260248 Slide.02
[Big Slide]
|
Chrome Click!! |
 |
Cyber threat intelligence consists of diverse and continuously evolving sources such as vulnerability databases, attack frameworks, and threat reports.
These sources are critical for security analysts, but their scale and heterogeneity make manual analysis and summarization challenging.
|
| ICACT20260248 Slide.03
[Big Slide]
|
Chrome Click!! |
 |
Large language models are capable of generating fluent and coherent summaries.
However, when applied without access to external evidence, they frequently produce unsupported or hallucinated claims.
In cybersecurity settings, such inaccuracies can lead to incorrect assessments and operational risk.
|
| ICACT20260248 Slide.04
[Big Slide]
|
Chrome Click!! |
 |
Retrieval augmented generation improves factual grounding by supplying relevant external context to the language model.
By conditioning generation on retrieved evidence, summaries become more accurate and aligned with authoritative sources.
|
| ICACT20260248 Slide.05
[Big Slide]
|
Chrome Click!! |
 |
Despite these advantages, most retrieval augmented systems rely on a fixed retrieval depth.
This one-size-fits-all strategy fails to account for variations in query complexity, corpus structure, and runtime constraints, leading to inefficiencies.
|
| ICACT20260248 Slide.06
[Big Slide]
|
Chrome Click!! |
 |
Our key insight is that retrieval requirements vary significantly across queries.
Simple queries typically have clear intent and require minimal supporting context.
In contrast, complex queries are more ambiguous and demand broader evidence.
Therefore, retrieval depth should adapt dynamically on a per-query basis.
|
| ICACT20260248 Slide.07
[Big Slide]
|
Chrome Click!! |
 |
Based on this observation, we treat retrieval as a decision rather than a predefined constant.
The system evaluates whether the current context is sufficient and retrieves additional evidence only when necessary.
|
| ICACT20260248 Slide.08
[Big Slide]
|
Chrome Click!! |
 |
We propose an adaptive retrieval approach guided by factuality feedback.
The system retrieves evidence incrementally using hybrid search, evaluates the factual quality of generated summaries, and stops retrieval when additional context no longer provides meaningful benefit.
|
| ICACT20260248 Slide.09
[Big Slide]
|
Chrome Click!! |
 |
This slide shows the end-to-end architecture of our adaptive retrieval system, split into an offline indexing phase and an online adaptive inference phase.
Starting with the offline phase, we ingest cyber threat intelligence corpora such as NVD and MITRE ATT&CK and build two complementary indices. The first is a sparse TF-IDF index that captures exact lexical matches, which are especially important for vulnerability IDs and precise technical terms. The second is a dense semantic index built using MiniLM embeddings and FAISS, which helps retrieve conceptually relevant passages even when wording differs.
During online inference, the process begins with a user’s threat intelligence query. Instead of retrieving a fixed number of documents, we start with a small retrieval depth, or top-k, using this hybrid retriever.
The retrieved context is passed to the LLM to generate a summary. We then immediately evaluate that generated summary using the Grounded Entailed Claim Score, or GECS, which measures how well each claim in the summary is supported by the retrieved evidence.
GECS acts as a feedback signal to the adaptive controller. If factual grounding improves meaningfully when we increase retrieval depth, the controller allows another retrieval step. If the factual gains saturate or the latency budget is exceeded, retrieval stops.
The key point here is that the LLM itself is unchanged. The intelligence lives in the controller: retrieval depth becomes a dynamic, per-query decision rather than a static configuration, allowing the system to balance factual accuracy and cost in real time.
|
| ICACT20260248 Slide.10
[Big Slide]
|
Chrome Click!! |
 |
In the offline phase, cyber threat intelligence sources are preprocessed and indexed.
Both sparse and dense indices are constructed to support flexible and effective retrieval during online inference.
|
| ICACT20260248 Slide.11
[Big Slide]
|
Chrome Click!! |
 |
During online inference, a user query retrieves candidate context, generates a summary, and evaluates factuality.
This process is executed dynamically for each query, enabling adaptive retrieval behavior.
|
| ICACT20260248 Slide.12
[Big Slide]
|
Chrome Click!! |
 |
In cyber threat intelligence applications, inaccurate summaries can mislead analysts and result in incorrect decisions.
Therefore, factual grounding is essential, and factuality must be measured explicitly to guide adaptive retrieval decisions.
|
| ICACT20260248 Slide.13
[Big Slide]
|
Chrome Click!! |
 |
We introduce the Grounded Entailed Claim Score, or GECS, as a metric for evaluating factuality.
GECS measures whether claims in a generated summary are supported by the retrieved evidence and aggregates this support across the entire summary.
|
| ICACT20260248 Slide.14
[Big Slide]
|
Chrome Click!! |
 |
Beyond evaluation, GECS serves as a control signal for adaptive retrieval.
By observing how factuality changes as more context is retrieved, the system determines whether further retrieval is beneficial.
|
| ICACT20260248 Slide.15
[Big Slide]
|
Chrome Click!! |
 |
The adaptive retrieval controller begins with minimal retrieval depth.
It incrementally increases context when factuality improves and stops retrieval when additional evidence no longer justifies the associated cost.
|
| ICACT20260248 Slide.16
[Big Slide]
|
Chrome Click!! |
 |
We evaluate our approach using cyber threat intelligence datasets, including vulnerability databases and attack frameworks.
Adaptive retrieval is compared against fixed retrieval depths, and summary quality is evaluated using the GECS metric.
|
| ICACT20260248 Slide.17
[Big Slide]
|
Chrome Click!! |
 |
Cyber threat intelligence data varies in structure.
Highly structured vulnerability data often reaches factual saturation with limited context, while semi-structured threat reports benefit from broader evidence.
This difference influences retrieval behavior across datasets.
|
| ICACT20260248 Slide.18
[Big Slide]
[YouTube] |
Chrome Click!! |
 |
This slide illustrates how retrieval depth affects factual consistency, measured using GECS, across different types of CTI data.
On the x-axis, we vary the retrieval depth, or top-k, and on the y-axis we plot the mean GECS score. What we observe is a clear pattern of diminishing returns.
For structured data like NVD vulnerability descriptions, factual gains increase quickly with small amounts of retrieved context, but they saturate very early. Beyond that point, additional retrieval does not improve factual grounding and can even introduce noise.
For semi-structured narrative data like MITRE ATT&CK, the gains are weaker and more variable. These descriptions are more conceptual and less fact-dense, so adding more context does not necessarily translate into stronger entailment.
This behavior highlights why a fixed retrieval depth is suboptimal. A single top-k value either over-retrieves for simple, structured queries or underperforms for more complex ones.
Our adaptive approach leverages this saturation behavior explicitly. By monitoring marginal improvements in GECS, the system learns when additional retrieval is no longer justified and stops early, preserving factual quality without unnecessary computation.
|
| ICACT20260248 Slide.19
[Big Slide]
|
Chrome Click!! |
 |
While the previous slide focused on factuality, this slide shows the other side of the trade-off: latency.
As retrieval depth increases, latency grows monotonically. This is expected because larger contexts increase both retrieval overhead and LLM inference time, with context size becoming the dominant cost factor.
Importantly, latency continues to rise even after factuality has already saturated, especially for structured CTI data. This means that beyond a certain point, we are paying a clear computational cost without gaining any factual benefit.
This mismatch between factual gains and latency growth is the motivation for our cost–utility formulation. Instead of optimizing factuality alone, we optimize a utility function that explicitly subtracts latency from factual gain.
The adaptive retrieval strategy uses this insight to cap retrieval depth early, avoiding expensive over-retrieval. In practice, this results in summaries that are just as factual as fixed-depth approaches, but significantly more efficient.
In real-world analyst workflows—where response time, scalability, and cost matter—this ability to control latency while preserving trustworthiness is critical.
|
| ICACT20260248 Slide.20
[Big Slide]
|
Chrome Click!! |
 |
In conclusion, fixed retrieval strategies are inefficient for cyber threat intelligence summarization.
Our adaptive retrieval approach improves factual grounding while controlling cost, without requiring retraining of language models.
Future work includes multi-hop reasoning and incorporating internal uncertainty signals to further enhance decision-making.
|
| ICACT20260248 Slide.21
[Big Slide]
|
Chrome Click!! |
 |
Thank you for taking the time to review this presentation.
I hope the content was informative and provided useful insights into adaptive retrieval for cyber threat intelligence.
If you have any questions or would like further clarification, I would be happy to address them.
|
