 |
While the previous slide focused on factuality, this slide shows the other side of the trade-off: latency.
As retrieval depth increases, latency grows monotonically. This is expected because larger contexts increase both retrieval overhead and LLM inference time, with context size becoming the dominant cost factor.
Importantly, latency continues to rise even after factuality has already saturated, especially for structured CTI data. This means that beyond a certain point, we are paying a clear computational cost without gaining any factual benefit.
This mismatch between factual gains and latency growth is the motivation for our cost–utility formulation. Instead of optimizing factuality alone, we optimize a utility function that explicitly subtracts latency from factual gain.
The adaptive retrieval strategy uses this insight to cap retrieval depth early, avoiding expensive over-retrieval. In practice, this results in summaries that are just as factual as fixed-depth approaches, but significantly more efficient.
In real-world analyst workflows—where response time, scalability, and cost matter—this ability to control latency while preserving trustworthiness is critical.
|
IEEE/ICACT20260248 Slide.19
[Big Slide]