Grasping Using Tactile And Visual Data Part of the EU project CogX. ← Back to research project list.

To grasp an object, an agent typically first devises a grasping plan from visual data, then it executes this plan, and finally it assesses the success of its action. Planning relies on (1) the extraction of object information from vision, and on (2) the recovery of memories related the current visual context, such as previous attempts to grasp a similar object. Because of the uncertainty inherent to these two processes, designing grasp plans that are guaranteed to work in an open-loop system is difficult. Grasp execution greatly benefits from a closed-loop controller which considers sensory feedback before and while issuing motor commands.

In this project, we study means of monitoring the execution of a grasp plan using vision and touch. By pointing a camera to the robot's workspace, we can track the 6D pose of visible objects in realtime. Touch data are captured by sensors placed on the robot's fingers. These two modalities are complementary, since during grasps objects are partly occluded by the hand, and visual object cues become uncertain. Monitoring the execution of a grasp allows the agent to abort grasps that are unlikely to succeed, thus preventing potential damage to the objects or the robot.

Our robot platform is composed of an industrial arm, a three-finger gripper equipped with tactile sensing arrays, and a camera.

We aim to establish the likelihood of success of a grasp before attempting to lift an object. Our agent learns and memorizes what it feels like to grasp objects from various sides. Tactile data are recorded once the hand is fully closed around the object. As the object often moves while the hand is closing around it, we track the object pose throughout the grasp, and record the pose once the hand is fully closed. The robot lifts up the object and turns it upside-down. If the object stays rigidly bound to the hand during this movement, the grasp is considered successful. During training, the agent encounters both successful and unsuccessful grasps, which provide it with input-output pairs, in the form of tactile imprints and relative object-gripper configurations (input) and success/failure labels (output). These data are used to train a classifier, which is subsequently used to decide whether a grasp feels stable enough to proceed to lifting the object.

Our experiment demonstrates that joint tactile and pose-based perceptions carry valuable grasp-related information, as models trained on both hand poses and tactile parameters perform better than the models trained exclusively on one perceptual input.

Part of the EU project CogX.

Supported by the Belgian National Fund for Scientific Research (FNRS).

Supported by the Swedish Foundation for Strategic Research (SSF).

Main reference:

bekiroglu2011d 

Y. Bekiroglu, R. Detry and D. Kragic, Learning Tactile Characterizations Of Object- And Pose-specific Grasps. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2011.
doi; pdf; show/hide bibtex

@inproceedings{bekiroglu2011d,
	author = {Yasemin Bekiroglu and Renaud Detry and Danica Kragic},
	booktitle = {{IEEE/RSJ} International Conference on Intelligent Robots and Systems},
	doi = {10.1109/IROS.2011.6094878},
	title = {Learning Tactile Characterizations Of Object- And Pose-specific Grasps},
	year = {2011}}

hyttinen2015a 

E. Hyttinen, D. Kragic and R. Detry, Learning the Tactile Signatures of Prototypical Object Parts for Robust Part-based Grasping of Novel Objects. In IEEE International Conference on Robotics and Automation, 2015.
doi; pdf; show/hide bibtex

@inproceedings{hyttinen2015a,
	author = {Emil Hyttinen and Danica Kragic and Renaud Detry},
	booktitle = {{IEEE} International Conference on Robotics and Automation},
	doi = {http://dx.doi.org/10.1109/ICRA.2015.7139883},
	title = {Learning the Tactile Signatures of Prototypical Object Parts for Robust Part-based Grasping of Novel Objects},
	year = {2015}}

Papers covering this topic:

bekiroglu2011c 

Y. Bekiroglu, R. Detry and D. Kragic, Joint Observation of Object Pose and Tactile Imprints for Online Grasp Stability Assessment. In Manipulation Under Uncertainty (Workshop at IEEE ICRA 2011), 2011.
pdf; show/hide bibtex

@inproceedings{bekiroglu2011c,
	author = {Yasemin Bekiroglu and Renaud Detry and Danica Kragic},
	booktitle = {Manipulation Under Uncertainty (Workshop at {IEEE} {ICRA} 2011)},
	title = {Joint Observation of Object Pose and Tactile Imprints for Online Grasp Stability Assessment},
	year = {2011}}

bekiroglu2011d 

Y. Bekiroglu, R. Detry and D. Kragic, Learning Tactile Characterizations Of Object- And Pose-specific Grasps. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2011.
doi; pdf; show/hide bibtex

@inproceedings{bekiroglu2011d,
	author = {Yasemin Bekiroglu and Renaud Detry and Danica Kragic},
	booktitle = {{IEEE/RSJ} International Conference on Intelligent Robots and Systems},
	doi = {10.1109/IROS.2011.6094878},
	title = {Learning Tactile Characterizations Of Object- And Pose-specific Grasps},
	year = {2011}}

hyttinen2015a 

E. Hyttinen, D. Kragic and R. Detry, Learning the Tactile Signatures of Prototypical Object Parts for Robust Part-based Grasping of Novel Objects. In IEEE International Conference on Robotics and Automation, 2015.
doi; pdf; show/hide bibtex

@inproceedings{hyttinen2015a,
	author = {Emil Hyttinen and Danica Kragic and Renaud Detry},
	booktitle = {{IEEE} International Conference on Robotics and Automation},
	doi = {http://dx.doi.org/10.1109/ICRA.2015.7139883},
	title = {Learning the Tactile Signatures of Prototypical Object Parts for Robust Part-based Grasping of Novel Objects},
	year = {2015}}

Many of these publications are copyrighted by their respective publishers. Downloadable versions are not necessarily identical to the published versions. They are made available here for personal use only.