New paper: Robot Trajectron V3 brings Bayesian shared control to SE(3) grasping!

Robot Trajectron V3 (RT-V3) [1] extends our probabilistic shared control from navigation to high-DoF manipulation: it assists a user who teleoperates a robot arm through a low-bandwidth interface. 🤖

The core idea is to cast shared control as Bayesian inference over future trajectories, rather than over a fixed set of candidate goals:

✅ Learned intent prior: A transformer-based conditional generative model predicts a distribution over future trajectories, from past robot dynamics and from a scene encoded as point clouds and candidate grasp poses.
✅ Continuous posterior updates: A likelihood derived from the user’s commands refines the intent estimate throughout execution. By contrast, in our experiments, the dense candidate poses of Hindsight Optimization easily trap the robot in an undesired grasp.
✅ Factorized action representation: Separating translation and rotation improves learning efficiency in high-dimensional action spaces.

In a real-world study with 21 novice joystick users, RT-V3 reached 86.9% success, against 78.6% for direct teleoperation and 74.7% for Hindsight Optimization, while substantially reducing physical and mental workload.

Check out the code & videos: https://mousecpn.github.io/RTV3_page/