Thursday - 23 May 2019

    8:30am - 12:30pm

    Room: 520d

The lack of performance assessment procedures is a prominent problem in robotic manipulation preventing researchers to compare their results against a common basis. In this workshop, well-developed and experimentally-demonstrated benchmarking protocols will be presented for various manipulation problems including robotic grasping, dexterous manipulation, aerial manipulation, non-prehensile manipulation, end-effector design among many others. The benchmarks explain step-by-step execution of the experimental procedure along with performance evaluation metrics, and baseline results obtained by state-of-the-art methods. These benchmarks are composed via close collaboration of experts in the related sub-fields. The experiences gained in the development process will also be discussed in panels for providing guidelines and inspiration for upcoming benchmarks in robotics. This workshop will be in conjunction with RA-Letters Special Issue on Benchmarking Protocols in Robotic Manipulation. For more details:


The workshop speakers will be chosen from the prospective contributors of the special issue. The topics include but not limited to:

Manipulation planning (e.g. performance of grasp planners)
Mechanism design (e.g. performance of robotic hands)
Machine learning (e.g. learning manipulation abilities)
Cognitive robotics (e.g. task representations)

Please email to bcalli@wpi.edu for any questions or comments.


8:30 - 8:45 INTRODUCTION
8:45 - 8:57 Benchmarking Protocols for Evaluating Grasp Strength of Robotic Hands
Joe Falco, Daniel Hemphill, Kenny Kimble, Elena Messina, Adam Norton, Rafael Ropellato, and Holly Yanco
National Institute of Standards and Technology (NIST) and University of Massachusetts Lowell
8:57 - 9:09 Benchmarking Robotic In-Hand Manipulation
Silvia Cruciani, Kaiyu Hang, Aaron M. Dollar and Danica Kragic, Balakumar Sundaralingam, Tucker Hermans, Vikash Kumar
KTH, University of Utah, Google AI, and Yale University
9:09 - 9:21 Benchmarking Artificial Hands in Dynamic Tasks: Hand and Grasp Resilience
Francesca Negrello, Antonio Bicchi, Manuel Catalano, Giorgio Grioli, Manolo Garabini, Maximo Roa, Oliver Brock
IIT, DLR, Unipi, TU Berlin
9:21 - 9:33 Dual-Arm Cloth Manipulation Benchmarking
Anastasiia Varava, Rika Antonova, Michelle Welle, Hang Yin, Danica Kragic, Julia Borras, Joan Lobo, Guillem Alenya and Carme Torras
KTH, Institut de Robotica i Informatica Industrial
9:33 - 9:45 Benchmarking Robotic Non-prehensile Object Rearrangement on Table-tops
Joshua A. Haustein, Haoran Song, Kaiyu Hang, Johannes Stork, Aaron M. Dollar, and Danica Kragic
Yale University, KTH, University of Science and Technology, Hong Kong, Orebro University
9:45 - 9:57 Benchmarking Manipulation via Simulation
Jack Collins, Juxi Leitner
Queensland University of Technology
10:00 - 10:30 COFFEE BREAK
10:30 - 10:42 Benchmarking Learning Algorithms for Dexterous Multi-Arm Insertion of Semi-Deformable Objects
Konstantinos Chatzilygeroudis, Kunpeng Yao, Bernardo Fichera, Farshad Khadivar, Ilaria Lauzana, Anais Haget, and Aude Billard
10:42 - 10:54 Aerial Manipulation Benchmark
Alejandro Suarez, Victor Vega, Manuel Fernandez, Guillermo Heredia, Anibal Ollero
Escuela Superior de Ingeniería
10:54 - 11:06 Groceries Pick and Place System Assessment Protocol
Pavlos Triantafyllou, Panagiotis Sotiropoulos, Hussein Mnyusiwalla, Maximo A. Roa
Ocado Technology, DLR
11:06 - 11:18 Box and Blocks Benchmark for Robotic Manipulation
Andrew Morgan, Walter Bircher, Aaron Dollar
Yale University
11:18 - 11:30 A Benchmarking Protocol for Data-based Within-hand Manipulation of an Underactuated Adaptive Hand
Avishai Sintov, Andrew Kimmel, Abdeslam Boularias and Kostas E. Bekris
Rutgers University
11:30 - 11:42 Benchmarking the Sub-systems of a robot grasping system
John Morrow, Yi Herng Ong, Ravi Balasubramanian, Cindy Grimm
Oregon state University
11:42 - 11:54 Jigsaw-based Learning Benchmarking for Robotic Manipulation
Xiaobo Liu, Fang Wan, Haoran Sun, Qichen Luo, and Chaoyang Song
Southern University of Science and Technology
11:54 - 12:06 Rubik’s Cube Manipulation – An Accuracy and Dexterity Benchmark for Robotic Manipulation
Boling Yang, Patrick Lancaster, Joshua R. Smith
University of Washington
12:06 - 12:18 Grasping pipeline in Cluttered Tabletop
Fabrizio Bottarel, Giulia Vezzani, Giulia Pasquale, Ugo Pattacini, and Lorenzo Natale


Dr. Berk Calli Worcester Polytechnic Institute (WPI) E-mail: bcalli@wpi.edu
Prof. Aaron Dollar Yale University E-mail: aaron.dollar@yale.edu
Dr. Maximo Roa German Aerospace Center (DLR) E-mail: maximo.roa@dlr.de
Dr. Yu Sun University of South Florida E-mail: yusun@cse.usf.edu