Protocols & Benchmarks

One of the main goals of this website is to serve as a portal for proposition and discussion of new community driven protocols and benchmarks. The following links can guide you through proposing new protocols and benchmarks:

Protocol Template                                               Benchmark Template 

Protocol Template Explanations                              Benchmark Template Explanations

Submit a New Protocol or Benchmark

A standard set of objects and associated models are a great starting point for common replicable research and benchmarking in manipulation, but protocols and benchmarks that define the experimental procedure and quantification methods are equally important for common replicable research and benchmarking in manipulation. Given the wide range of technical interests, research approaches and applications being examined in the manipulation research community, along with how quickly the field moves, it is quite challenging to provide sufficient task descriptions that will span the range of interests and remain relevant in the long-term. Therefore, we seek to lay the groundwork for those to be driven by the research community and sub-communities.

The proposed protocols and benchmarks by the community will be displayed here. The discussions on these protocols and benchmarks are in the corresponding forum page.

Here are the proposed protocols and benchmarks so far:

NameAuthorsPurposeFiles (ID / Version)
Gripper Assessment Protocol and BenchmarkBerk Calli, Aaron Dollar
(Yale University)
Assessing the abilities of a gripper to firmly grasp objects of different sizes and shapes.Protocol (P-GA / 0.1)
Benchmark (B-GA / 0.1)
Table Setting Protocol and BenchmarkBerk Calli (Yale University)To assess the pick and place ability of a robotic system.Protocol (P-TS / 0.1)
Benchmark (B-TS / 0.1)
Block Pick and Place Protocol and BenchmarkAaron Walsman, Siddhartha Srinivasa (CMU)Assess the dexterity of a robotic manipulator using a pick-and-place task.Protocol (P-BP / 0.1)
Benchmark (B-BP / 0.1)
Peg Insertion Learning Assessment Protocol and BenchmarkArjun Singh, Pieter Abbeel
(UC Berkeley)
To assess the learning performance of a robot by the task of inserting a single peg to a hole.Protocol (P-PI / 0.1)
Benchmark (B-PI / 0.1)
Pitcher-Mug Protocol and BenchmarkAaron Walsman, Berk Calli (CMU, Yale University)To assess the ability of a robot to execute a common daily task.Protocol (P-PM / 0.1)
Benchmark (B-PM / 0.1)