New benchmark PuMVR reveals script bias in multilingual VLMs

Arxiv CS.CL (NLP/language models)

PuMVR (Punjabi Multimodal Visual Reasoning) tests VLMs on Punjabi text in Gurmukhi and Shahmukhi scripts, revealing significant accuracy drops across scripts for the same language. The benchmark challenges the one-to-one language-script mapping assumption in multilingual VLM evaluation.

pumvrMultimodal

See 1 more source

Not Truly Multilingual: Script Consistency as a Missing Dimension in VLM Evaluation6 days agoPrabhjot Singh, Bhushan Pawar, Madhu Reddiboina, Rajvee Sheth

Arxiv CS.CL (NLP/language models)·Prabhjot SinghBhushan PawarMadhu Reddiboina

23 hours ago