DOVE: A New Framework for Evaluating LLM Cultural Value Alignment
Researchers introduce DOVE, a distributional evaluation framework that compares human text distributions with LLM outputs to assess cultural value alignment. This method overcomes the limitations of traditional multiple-choice benchmarks by addressing the C3 challenge of context, composition, and subcultural heterogeneity.
A new research paper from arXiv introduces DOVE, a distributional evaluation framework designed to better assess how Large Language Models align with diverse cultural values. Unlike traditional benchmarks that rely on discriminative, multiple-choice questions, DOVE directly compares the statistical distributions of human-written text against LLM-generated outputs. This approach aims to move beyond simply testing a model's knowledge of values to actually probing its true cultural orientations in open-ended generation scenarios.
The framework addresses the critical C3 challenge—Construct, Composition, and Context—which has long plagued existing evaluation methods. Current benchmarks often fail to capture subcultural heterogeneity and do not match the complexity of real-world interactions where models generate text freely. By shifting from a binary correct/incorrect format to a distributional comparison, DOVE offers a more nuanced view of how well models reflect the diversity of human cultural expression rather than just memorizing predefined value sets.
As global deployment of LLMs accelerates, the ability to accurately measure cultural alignment is becoming a cornerstone of safety and user engagement. The introduction of DOVE suggests a significant shift in how the research community approaches AI safety, moving towards metrics that reflect the fluidity of human culture. Future work will likely focus on refining the value codebooks used in this framework and determining how these distributional insights can be applied to fine-tune models for specific regional or subcultural contexts.