New Benchmark Tests AI's Ability to Understand Mixed-Language Text
Researchers created a new test to evaluate how well AI models handle mixed-language text, like Hindi or Tamil blended with English. This is common in many multilingual communities but often confusing for AI systems.
Researchers introduced Indi-RomCoM, a new benchmark to test how well AI models understand mixed-language text. This type of text, called Romanized Code Mixing (RCM), happens when people blend local languages like Hindi or Tamil with English using the Roman alphabet. For example, someone might write 'Aaj kal mera mood very chill hai' instead of 'Today my mood is very chill.' While AI models perform well with single languages or native scripts, they often struggle with these mixed-language instructions.
This matters because many people around the world communicate this way, especially in multilingual communities. If AI models can't understand or respond accurately to mixed-language text, they won't be as helpful in real-world situations. For instance, someone might ask an AI assistant for directions in a mixed-language format, and the AI might not understand the request correctly. The Indi-RomCoM benchmark aims to improve this by providing a standardized way to test and train AI models on these types of instructions.
The benchmark spans several Indic languages and evaluates models on their ability to follow instructions and reason over RCM-based content. If you're curious about how well current AI models handle mixed-language text, you can try asking a language model like ChatGPT or Claude a question in a mixed-language format. For example, you might ask, 'Kal ka weather kya hoga?' which blends Hindi with English. Observe how well the AI understands and responds to your query. This can give you a practical sense of how far AI has come in understanding mixed-language text.