Ekka: Automated Diagnosis of Silent Errors in LLM Inference
University of Washington researchers developed Ekka, a system that automatically detects silent errors in LLM outputs. This could make AI systems more reliable for everyday users.
Researchers at the University of Washington's SyFi Lab (Systems, Foundations, and Infrastructure) released Ekka, an automated system designed to diagnose silent errors — mistakes that large language models (LLMs) make without any obvious outward signs, such as confidently sounding correct while being factually wrong. Ekka works by systematically comparing an LLM's output against a set of known correct examples and analyzing inconsistencies to catch errors that humans might miss.
This matters because it makes AI more trustworthy. In safety-critical applications like healthcare, finance, or legal advice, a model could give a wrong answer that sounds compelling. Ekka could catch those mistakes before they cause real-world harm, making LLMs safer for important tasks.