PuMVR benchmark quantifies orthographic bias in multilingual VLMs

The PuMVR (Punjabi Multimodal Visual Reasoning) benchmark evaluates vision-language models on Punjabi text in multiple scripts, revealing that models assume a one-to-one language-to-script mapping. The findings highlight that billions of multi-script language users are overlooked by current multilingual evaluations.