Protocols & Benchmarks

One of the main goals of this website is to serve as a portal for proposition and discussion of new community driven protocols and benchmarks. The following links can guide you through proposing new protocols and benchmarks:

Protocol Template                                               Benchmark Template 

Protocol Template Explanations                              Benchmark Template Explanations

If you would like your protocol/benchmark added to this website, please email to Berk Calli (bcalli@wpi.edu)

A standard set of objects and associated models are a great starting point for common replicable research and benchmarking in manipulation, but protocols and benchmarks that define the experimental procedure and quantification methods are equally important for common replicable research and benchmarking in manipulation. Given the wide range of technical interests, research approaches and applications being examined in the manipulation research community, along with how quickly the field moves, it is quite challenging to provide sufficient task descriptions that will span the range of interests and remain relevant in the long-term. Therefore, we seek to lay the groundwork for those to be driven by the research community and sub-communities.

The proposed protocols and benchmarks by the community will be displayed here. The discussions on these protocols and benchmarks are in the corresponding forum page.

Here are the proposed protocols and benchmarks so far:

NamePublication DateAuthorsPurposeFiles (ID / Version)
The Elliott and Connolly Benchmark: A Test for Evaluating the In-Hand Dexterity of Robot Hands Oct. 2021Ryan Coulson, Chao Li, Carmel Majidi, and Nancy S. Pollardevaluating in- hand dexterity, specifically for humanoid type robot handsProtocol

Benchmark

Videos

Paper

Supplementary
Benchmarking Protocol for Grasp Planning AlgorithmsFeb. 2020Yasemin Bekiroglu, Naresh Marturi, Máximo A. Roa, Komlan Jean Maxime Adjigble, Tommaso Pardi, Cindy Grimm, Ravi Balasubramanian, Kaiyu Hang, Rustam StolkinAssess the performance of a grasp planning algorithm in a physical robot setup, minimizing the influence of key components in the grasping pipelineProtocol

Benchmark

Videos

Paper
Bimanual Manipulation BenchmarkFeb. 2020Konstantinos Chatzilygeroudis, Bernardo Fichera, Ilaria Lauzana, Fanjun Bu, Kunpeng Yao, Farshad Khadivar, and Aude BillardAssess the motion planning and/or learning performance of a multi-arm systemsProtocol

Benchmark

Videos

Paper
Benchmarks for Aerial ManipulationFeb. 2020Alejandro Suarez , Victor M. Vega, Manuel Fernandez, Guillermo Heredia, Anibal OlleroEvaluate and compare the performance of
aerial manipulators
Protocol and Benchmarks

Videos

Paper
Box and Blocks BenchmarkFeb. 2020Andrew S. Morgan, Kaiyu Hang, Walter G. Bircher, Fadi M. Alladkani, Abhinav Gandhi, Berk Calli, Aaron M. DollarAssess unilateral gross manual dexterityProtocol

Benchmark

Videos

Supplementary

Paper
Benchmarking In-Hand ManipulationFeb. 2020Silvia Cruciani, Balakumar Sundaralingam, Kaiyu Hang, Vikash Kumar,
Tucker Hermans, Danica Kragic
Evaluate the planning
and control aspects of robotic in-hand manipulation systems
Protocol

Benchmark

Videos

Paper
GRASPA-benchmarkFeb. 2020Fabrizio Bottarel, Giulia VezzaniAssess the performance of a grasp planning algorithm in a physical robot setup, accounting for the limitations of the test platformProtocol

Benchmark

Paper
Bimanual Cloth Manipulation BenchmarkFeb. 2020Irene Garcia-Camacho*, Martina Lippi*, Michael C. Welle, Hang Yin, Rika Antonova, Anastasiia Varava,Julia Borras, Carme Torras, Alessandro Marino, Guillem Alenya, Danica KragicAssess the ability of a bimanual robotic system to manipulate cloth piecesProtocol

Benchmark

Videos

Paper
Grasp Resilience BenchmarkFeb. 2020F. Negrello, W. Friedl, G. Grioli, M. Garabini, O. Brock, A. Bicchi, M. Roa, M. G. CatalanoCharacterizing the behavior of robotic hands under impulsive loads conditions.Reference Document

Paper
Simulated Robotic Manipulation ProtocolFeb. 2020J. Collins, J. McVicar, D. Wedlock, R. Brown, D. Howard and J. LeitnerAssessing manipulation performance in simulationProtocol

Benchmark

Videos

Paper
Rubik's Cube Manipulation ProtocolFeb. 2020Boling Yang, Patrick Lancaster, Siddhartha S. Srinivasa, Joshua R. SmithEvaluating a robotic system's overall manipulation accuracy, speed, and robustness.Protocol

Benchmark

Videos

Paper
The Anthropomorphic Hand Assessment Protocol (AHAP)Nov. 2019Immaculada Llop-Harillo, Antonio Pérez-González, Julia Starke, Tamim Asfour (Universitat Jaume I, Karlsruhe Institute of Technology (KIT))Quantifying the grasping ability of artificial hands and comparing hand designsProtocol

Benchmark

Paper
Gripper Assessment Protocol and BenchmarkSept. 2015Berk Calli, Aaron Dollar
(Yale University)
Assessing the abilities of a gripper to firmly grasp objects of different sizes and shapes.Protocol

Benchmark

Paper
Table Setting Protocol and BenchmarkSept. 2015Berk Calli (Yale University)To assess the pick and place ability of a robotic system.Protocol

Benchmark

Paper
Block Pick and Place Protocol and BenchmarkSept. 2015Aaron Walsman, Siddhartha Srinivasa (CMU)Assess the dexterity of a robotic manipulator using a pick-and-place task.Protocol

Benchmark

Paper
Peg Insertion Learning Assessment Protocol and BenchmarkSept. 2015Arjun Singh, Pieter Abbeel
(UC Berkeley)
To assess the learning performance of a robot by the task of inserting a single peg to a hole.Protocol

Benchmark

Paper
Pitcher-Mug Protocol and BenchmarkSept. 2015Aaron Walsman, Berk Calli (CMU, Yale University)To assess the ability of a robot to execute a common daily task.Protocol

Benchmark

Paper