Evaluating language transfer in robot policies by adding Greek to a Cosmos3 vision-language-action system
benchmarks
| Source: HF Papers | Original article
Researchers extend a Cosmos3 vision-language-action robot policy to include Greek using machine‑rephrased instructions, without changing the architecture.
A new study demonstrates that a large‑scale vision‑language‑action (VLA) model can be extended to understand Greek without altering its architecture. Researchers took the Cosmos3 policy—an open‑source robot foundation model trained primarily on English instructions—and fed it machine‑rephrased Greek commands. The experiment shows that the model can process the foreign language, but the authors stress that the real difficulty lies in measuring performance rather than in translation itself.
Current robot foundation models are overwhelmingly English‑centric, and publicly available demonstration datasets for other languages are scarce. When the team evaluated the Greek‑augmented policy on standard robotic benchmarks, they found the usual success‑rate metrics to be largely insensitive to the language of the instruction. Prior work has reported that VLA policies often ignore linguistic cues on such suites, and this new work reproduces that pattern, with success rates hovering around 84 % under Greek prompts. The authors warn that many seemingly plausible evaluation tools can give misleading signals, highlighting a need for benchmarks that truly capture language understanding in embodied tasks.
The findings matter because multilingual robot instruction is a prerequisite for broader deployment of autonomous systems in non‑English‑speaking environments. If evaluation remains blind to language, progress toward genuinely language‑aware robots will be hard to track. The next steps will likely involve designing benchmark suites that tie each scene to multiple, semantically distinct goals, and testing transfer to additional languages. Monitoring how the community responds with new metrics and datasets will be key to turning multilingual capability from a technical curiosity into a reliable feature of future robot assistants.
Sources
Back to AIPULSEN