PAMI turns a sentence into a 3D person handling an object
Type “lift the suitcase, move it, and put it down”, and PAMI animates a 3D person doing exactly that, with the suitcase in their hands the whole way. It comes from the Tübingen AI Center and the Max Planck Institute for Informatics in Germany. Every motion here was generated from the sentence at the top of the frame.
Older methods let the tripod float away from the hands
Here's why that's hard. Give three methods the same sentence: lift the tripod, move it, and put it down. PAMI keeps the tripod in hand. An earlier method called LIGHT moves the tripod on its own, and the body turns purple when the hand loses contact. With InterAct, the body turns pink where it passes straight through the tripod, and the tripod is never set back down.
Each body part votes on where the object goes
PAMI ties the object to the body. The hips, both wrists and both ankles each predict where the object should be, relative to themselves. A learned weight decides, frame by frame, how much each prediction counts. The idea is borrowed from the Hough transform, a classic way of finding shapes in images by letting points vote.
As the hand picks up the ball, the wrist takes over the vote
In this render, the colored dots are those anchors. While the ball sits on the ground, its position is shared out among them. Once the hand closes around it, the wrist holding it carries almost all of the weight, so the ball goes where the hand goes.
A second pass fixes the grip
The first pass gets the motion roughly right. A second model, the refiner, then looks at the surface of the object right around the hands, and nudges the pose and the object until the fingers actually touch it. Without it, the hand hovers off the tripod and the body turns purple.
On the InterAct benchmark, hands stay on the object more often
Contact recall 0.836 against LIGHT's 0.730, 14.5% higher. Penetration 0.062 against LIGHT's 0.121 and InterAct's 0.119; real motion capture scores 0.060.
InterAct benchmark, hand contact. Li et al., “PAMI”, arXiv 2609.38466, Table 1.
On the InterAct benchmark, PAMI reproduces about 84 percent of the hand contacts in real motion capture, against 73 for the best earlier method, LIGHT. Bodies pass through objects about half as much as with LIGHT or InterAct, close to the level in the real recordings. And its motions match their sentences almost as well as the real recordings do.
One person and one object so far, and the code is coming
Posted to arXiv on 29 September 2026. It needs the object's 3D model up front, and the refiner can't rescue a wrong action.
There are limits. It handles one person and one rigid object at a time, and it needs the object's 3D model up front. The refiner polishes a grip, but it can't rescue a motion that's wrong to begin with. The paper went up on September 29th, and the authors say the code and models will be released. The paper and the project page are linked below.
