Researchers Introduce 'Background Temperature' to Explain Hidden Randomness in LLMs

Researchers from Thinking Machines Lab have introduced the concept of 'background temperature' (T_bg) to characterize hidden randomness in large language models (LLMs). Even when decoding with temperature T=0, LLMs can produce varying outputs for identical inputs due to implementation-level nondeterminism. This includes factors like batch-size variation, kernel non-invariance, and floating-point non-associativity.

The study formalizes this behavior by quantifying the effective temperature induced by these implementation-dependent perturbations. This concept helps explain why LLMs can exhibit unpredictable behavior even under seemingly deterministic conditions. The researchers suggest that understanding and controlling background temperature could lead to more consistent and reliable LLM outputs.

The introduction of background temperature opens up new avenues for research into the determinism of LLMs. Future work could focus on mitigating these implementation-level sources of randomness to improve model consistency. This research also highlights the importance of considering hardware and software implementation details in the development and deployment of LLMs.