Autonomous Learning of Object Grasp Models Part of the EU project PACO-PLUS. ← Back to research project list.

In this project, we study means of modeling and learning object grasp affordances, i.e., relative object-gripper poses that lead to stable grasps. Affordances are represented probabilistically with grasp densities [detry2011a], which correspond to continuous density functions defined on the space of 6D gripper poses – 3D position and orientation.

Projection of a 6DOF grasp density on a 2D image. Grasp success likelihood is proportional to the intensity of the green mask.

Grasp densities are linked to visual stimuli through registration with a visual model of the object they characterize, which allows the robot to grasp objects lying in arbitrary poses: to grasp an object, the object's model is visually aligned to the correct pose; the aligned grasp density is then combined to reaching constraints to select the maximum-likelihood achievable grasp. Grasp densities are learned and refined through exploration: grasps sampled randomly from a density are performed, and an importance-sampling algorithm learns a refined density from the outcomes of these experiences. Initial grasp densities are computed from the visual model of the object.

We demonstrated that grasp densities can be learned autonomously from experience. Our experiment showed that through learning, the robot becomes increasingly efficient at inferring grasp parameters from visual evidence. The experiment also yielded conclusive results in practical scenarios where the robot needs to repeatedly grasp an object lying in an arbitrary pose, where each pose imposes a specific reaching constraint, and thus forces the robot to make use of the entire grasp density to select the most promising achievable grasp. This work led to publications in the fields of robotics [detry2010a, detry2010b, detry2011a] and developmental learning [detry2009c].

Part of the EU project PACO-PLUS.

Supported by the Belgian National Fund for Scientific Research (FNRS).

Code based on the Nuklei library.

Main reference:

detry2011a 

R. Detry, D. Kraft, O. Kroemer, L. Bodenhagen, J. Peters, N. Krüger and J. Piater, Learning Grasp Affordance Densities. In Paladyn. Journal of Behavioral Robotics, 2 (1): 1–17, 2011.
doi; pdf; show/hide bibtex

@article{detry2011a,
	author = {R. Detry and D. Kraft and O. Kroemer and L. Bodenhagen and J. Peters and N. Krüger and J. Piater},
	doi = {10.2478/s13230-011-0012-x},
	journal = {Paladyn.\ Journal of Behavioral Robotics},
	number = {1},
	pages = {1--17},
	title = {Learning Grasp Affordance Densities},
	volume = {2},
	year = {2011}}

Papers covering this topic:

bodenhagen2011a 

L. Bodenhagen, R. Detry, J. Piater and N. Krüger, What a successful grasp tells about the success chances of grasps in its vicinity. In ICDL-EpiRob, 2011.
pdf; show/hide bibtex

@inproceedings{bodenhagen2011a,
	author = {Leon Bodenhagen and Renaud Detry and Justus Piater and Norbert Krüger},
	booktitle = {ICDL-EpiRob},
	title = {What a successful grasp tells about the success chances of grasps in its vicinity},
	year = {2011}}

detry2009b 

R. Detry, E. Başeski, N. Krüger, M. Popović, Y. Touati and J. Piater, Autonomous Learning of Object-specific Grasp Affordance Densities. In Approaches to Sensorimotor Learning on Humanoid Robots (Workshop at the IEEE International Conference on Robotics and Automation), 2009.
pdf; show/hide bibtex

@inproceedings{detry2009b,
	author = {Renaud Detry and Emre Başeski and Norbert Krüger and Mila Popović and Younes Touati and Justus Piater},
	booktitle = {Approaches to Sensorimotor Learning on Humanoid Robots (Workshop at the IEEE International Conference on Robotics and Automation)},
	title = {Autonomous Learning of Object-specific Grasp Affordance Densities},
	year = {2009}}

detry2009c 

R. Detry, E. Başeski, N. Krüger, M. Popović, Y. Touati, O. Kroemer, J. Peters and J. Piater, Learning Object-specific Grasp Affordance Densities. In IEEE International Conference on Development and Learning, pages 1–7, 2009.
doi; pdf; show/hide bibtex

@inproceedings{detry2009c,
	author = {Renaud Detry and Emre Başeski and Norbert Krüger and Mila Popović and Younes Touati and Oliver Kroemer and Jan Peters and Justus Piater},
	booktitle = {{IEEE} International Conference on Development and Learning},
	doi = {10.1109/DEVLRN.2009.5175520},
	pages = {1--7},
	title = {Learning Object-specific Grasp Affordance Densities},
	year = {2009}}

detry2010a 

R. Detry, E. Başeski, M. Popović, Y. Touati, N. Krüger, O. Kroemer, J. Peters and J. Piater, Learning Continuous Grasp Affordances by Sensorimotor Exploration. In From Motor Learning to Interaction Learning in Robots, pages 451–465, Springer-Verlag, 2010.
doi; pdf; show/hide bibtex

@incollection{detry2010a,
	author = {Renaud Detry and Emre Başeski and Mila Popović and Younes Touati and Norbert Krüger and Oliver Kroemer and Jan Peters and Justus Piater},
	booktitle = {From Motor Learning to Interaction Learning in Robots},
	doi = {10.1007/978-3-642-05181-4_19},
	editor = {Olivier Sigaud and Jan Peters},
	pages = {451--465},
	publisher = {Springer-Verlag},
	title = {Learning Continuous Grasp Affordances by Sensorimotor Exploration},
	year = {2010}}

detry2010b 

R. Detry, D. Kraft, A. G. Buch, N. Krüger and J. Piater, Refining Grasp Affordance Models by Experience. In IEEE International Conference on Robotics and Automation, pages 2287–2293, 2010.
doi; pdf; show/hide bibtex

@inproceedings{detry2010b,
	author = {Renaud Detry and Dirk Kraft and Anders Glent Buch and Norbert Krüger and Justus Piater},
	booktitle = {{IEEE} International Conference on Robotics and Automation},
	doi = {10.1109/ROBOT.2010.5509126},
	pages = {2287--2293},
	title = {Refining Grasp Affordance Models by Experience},
	year = {2010}}

detry2010d 

R. Detry, Learning of Multi-Dimensional, Multi-Modal Features for Robotic Grasping. Ph.D. Thesis, University of Liège, 2010.
pdf; show/hide bibtex

@phdthesis{detry2010d,
	author = {Renaud Detry},
	school = {University of Liège},
	title = {Learning of Multi-Dimensional, Multi-Modal Features for Robotic Grasping},
	year = {2010}}

detry2011a 

R. Detry, D. Kraft, O. Kroemer, L. Bodenhagen, J. Peters, N. Krüger and J. Piater, Learning Grasp Affordance Densities. In Paladyn. Journal of Behavioral Robotics, 2 (1): 1–17, 2011.
doi; pdf; show/hide bibtex

@article{detry2011a,
	author = {R. Detry and D. Kraft and O. Kroemer and L. Bodenhagen and J. Peters and N. Krüger and J. Piater},
	doi = {10.2478/s13230-011-0012-x},
	journal = {Paladyn.\ Journal of Behavioral Robotics},
	number = {1},
	pages = {1--17},
	title = {Learning Grasp Affordance Densities},
	volume = {2},
	year = {2011}}

kraft2009b 

D. Kraft, R. Detry, N. Pugeault, E. Başeski, J. Piater and N. Krüger, Learning Objects and Grasp Affordances through Autonomous Exploration. In International Conference on Computer Vision Systems, pages 235–244, 2009.
doi; pdf; show/hide bibtex

@inproceedings{kraft2009b,
	author = {Dirk Kraft and Renaud Detry and Nicolas Pugeault and Emre Başeski and Justus Piater and Norbert Krüger},
	booktitle = {International Conference on Computer Vision Systems},
	doi = {10.1007/978-3-642-04667-4_24},
	pages = {235--244},
	title = {Learning Objects and Grasp Affordances through Autonomous Exploration},
	volume = {5815},
	year = {2009}}

kraft2010a 

D. Kraft, R. Detry, N. Pugeault, E. Başeski, F. Guerin, J. Piater and N. Krüger, Development of Object and Grasping Knowledge by Robot Exploration. In IEEE Transactions on Autonomous Mental Development, 2 (4): 368–383, 2010.
doi; pdf; show/hide bibtex

@article{kraft2010a,
	author = {Dirk Kraft and Renaud Detry and Nicolas Pugeault and Emre Başeski and Frank Guerin and Justus Piater and Norbert Krüger},
	doi = {10.1109/TAMD.2010.2069098},
	journal = {{IEEE} Transactions on Autonomous Mental Development},
	number = {4},
	pages = {368--383},
	title = {Development of Object and Grasping Knowledge by Robot Exploration},
	volume = {2},
	year = {2010}}

piater2009a 

J. Piater, S. Jodogne, R. Detry, D. Kraft, N. Krüger, O. Kroemer and J. Peters, Learning Visual Representations for Interactive Systems. In International Symposium on Robotics Research, 2009.
doi; pdf; show/hide bibtex

@inproceedings{piater2009a,
	author = {Justus Piater and Sébastien Jodogne and Renaud Detry and Dirk Kraft and Norbert Krüger and Oliver Kroemer and Jan Peters},
	booktitle = {International Symposium on Robotics Research},
	doi = {10.1007/978-3-642-19457-3_24},
	title = {Learning Visual Representations for Interactive Systems},
	year = {2009}}

Many of these publications are copyrighted by their respective publishers. Downloadable versions are not necessarily identical to the published versions. They are made available here for personal use only.