In brief
- A vision language model, built to work with images and words, controlled real robot arms from instructions written in everyday language.
- The robot completed its first task, putting a carrot in a box, on the first try, in an office and from a camera angle it had never seen.
- It is far from reliable. In a large simulated robot test, the general-purpose language model GPT-6 Astra completed only about one task in five. Yet that was enough to put it in first place, ahead of every system built specifically for robots.
Getting a robot to do a new job usually takes an expert. Someone has to write the software that makes the robot understand its environment and act, set up the robot for that particular workspace and test it. That is one reason why robots are common in big factories but rare for small, varied or one-off jobs.
What if you could simply tell the robot what you want, the way you would ask a colleague? That is what we set out to test.
A surprising result in simulation
In a September 2026 evaluation, the team behind an established robot benchmark called “RoboDojo” asked language models to control a simulated robot arm directly, with no robot-specific training. Two of the models almost never succeeded. GPT-6 Astra completed about 22.5% of attempts, ahead of all 40 robot-specific systems it was compared against (the best of them, DM0.5, reached 19.3%).

A 22.5% success rate means about one attempt in five, so this is not a finished solution. The surprise is that a general-purpose language model beat systems designed for the job.
The RoboDojo benchmark covers 42 robot tasks inside a computer simulation, and the tasks probe five skills: coping with new situations, remembering, being precise, carrying out long multi-step jobs, and following open-ended instructions.
The study also found a clear pattern. The model did well on tasks that mainly require understanding what to do, and poorly on tasks that need fine precision, fast reactions, or two arms working together. The authors describe the result as early evidence, with reliable precision still a limitation. In some recorded runs the model also corrected itself when something went wrong.
All of this happened in simulation. The obvious next question was whether it holds up in the real world.
Our real-world test
We chose to skip the polished demo and build the least forgiving setup we could think of. We used two robot arms in an ordinary office the system had never seen, filmed from a camera angle it had never seen. There was no calibration (the step where the system is taught exactly where its camera, arms, and table are in relation to each other). The language model has access to something called an MCP server that enables it to talk to the arm, it uses tool calls to get images and depth information from the cameras, and it can send movement corrections to the arms and gripper. We worked through AI coding assistants (GitHub Copilot and Claude Code), and a plain-language instruction was the only input.
“Put the carrot in the box.”
First try, unseen office and unseen camera angle.
It worked. We then pushed further, with drawing, copying a person from a video, and a task that needs two arms to cooperate.
The five videos
These are demonstrations of what the approach can do:
Note: The process is somewhat slow, so the videos are played back at a faster speed.
- The carrot: The first attempt. The robot has to see the scene, decide what to do and pick up the carrot, all from one sentence.
- The logo: Drawing the SINTEF logo, Opus uses the two arms to uncap a marker and draws line by line.
- The Block Stack: Instead of describing the task, we show a video of a person stacking blocks and say “do what the person is doing in the video.”
- The Toothpaste: The hardest task. Two arms work together to put toothpaste on a toothbrush, using a soft, squeezable tube.
The Failures: Moments where it goes wrong, kept in on purpose because they are part of the honest picture.
Video scripts
Video 1: The carrot
Prompt: “Put the carrot in the box”
| 0:00 | This is the first attempt. A robot arm, a carrot on an office desk, and an open box. (x4 normal speed). |
| 0:05 | The only instruction is one sentence: put the carrot in the box. |
| 0:10 | No special setup and no hand-written robot program. The system looks at the scene and works out where things are. |
| 0:18 | The arm misses the first grasp since the depth camera was not calibrated and gives 2cm too short a distance – The model recognizes this and correct on the next try. |
| 0:25 | It carries the carrot over to the box and puts it in. |
| 0:29 | Unseen room, unseen camera angle, first try. This is what made us take the approach seriously. |
Video 2: Drawing the SINTEF logo
Prompt: “Draw the SINTEF logo”
| 0:00 | Two arms stand at the table. They collaborate on getting the marker cap off, no one explicitly told them to cooperate. |
| 0:07 | The instruction: draw the SINTEF logo. |
| 0:14 | The arm lowers the pen to the paper and begins tracing the round emblem. |
| 0:24 | Then the lettering, S-I-N-T-E-F, stroke by stroke. |
| 0:34 | Lift, reposition, touch down again. Every move is planned from the picture and the sentence. |
| 0:40 | The result is a recognizable logo, drawn by a robot that was never programmed to draw it. |
Video 3: Doing what the person does
Prompt: “Do what the person is doing in the video”
| 0:00 | First, a person stacks colored blocks on the table while the camera watches. |
| 0:06 | Then the instruction changes. Not a description of the task, just: do what the person is doing in the video. |
| 0:11 | The robot has to work out what the person did: which block, where, and in what order. |
| 0:16 | One arm picks up a block and carries it across the table. |
| 0:21 | It places the block on the stack, copying what it saw. |
Video 4: Toothpaste on the toothbrush
Prompt: “Put toothpaste on the toothbrush”
| 0:00 | The hardest task of the set. One arm holds a tube of toothpaste, and a toothbrush lies on the table. |
| 0:05 | The instruction is a single sentence: put toothpaste on the toothbrush. |
| 0:10 | The two arms have to coordinate. One manages the tube, the other helps position it. |
| 0:16 | The tube is soft and awkward to hold, so small errors show up quickly. |
| 0:22 | Toothpaste on the table shows where the tube squeezed out. Close, but not perfect. |
| 0:25 | An imperfect result that shows how far the approach has come and what remains. |
Why this matters
If a robot can be instructed in everyday language, setting one up for a new task could become much faster and cheaper, which might make automation practical for tasks and organizations that cannot afford a dedicated robot programmer today. A robot cell you can talk to already exists, but it is still at an early stage. Today these systems are too slow and expensive, their success rates are too low, and mainly require access to frontier models which run on proprietary servers outside of our infrastructure. The simulated test shows that most attempts still fail, the bloopers show real hardware going wrong in ordinary ways, and the model is weakest on exactly the things many jobs need: precision and fast, coordinated movement.
What matters here is the direction of travel and how low the cost of getting started has become. A task that once needed specialist setup took one sentence, and the failures we saw are the kind that better models and better tools can plausibly address. We expect these systems to improve with time, though how quickly, and whether they reach the reliability that industrial use demands, remains an open question.
How will the work continue
The work continues on several fronts, under the overall goal of developing general robot systems that can interact intelligently with the environment for a wide range of tasks. We are training robots in simulation using trial and error (reinforcement learning) and learning how to accurately transfer to the real world for industrial tasks. This also entails building detailed 3D copies of our real lab environments using a technique called Gaussian splats, so that the robots can practice in realistic visual twins of the robot’s real environment. In parallel, we are comparing this to training the robots on smaller scale real data from the lab directly.
The language-model experiment is another strand of this effort, a way to find out how far a general-purpose AI can go before specialized training takes over.
About this work
The work was carried out within the physical agentic AI activities, together with the Norwegian Centre for Embodied AI (NCEI).
Read more about our physical AI research at SINTEF.
Language model, defined: A large language model (LLM) is an AI system that has learned from vast amounts of text and can follow instructions written in everyday language. It is the technology behind chatbots such as ChatGPT, Gemini, and Claude. LLMs can write, summarize, translate and produce computer code, and many can also understand images. What they were not built to do is move a robot arm. For years, attempts to use them that way mostly failed.

Comments
No comments yet. Be the first to comment!