Sequence-of-Constraints MPC: Reactive Timing-Optimal Control of Sequential Manipulation

Project Page for the IROS 2022 submission

Marc Toussaint, Jason Harris, Jung-Su Ha, Danny Driess, Wolfgang Hönig

Learning & Intelligent Systems Lab, Science of Intelligence Excellence Cluster, TU Berlin


Brief: With this research we build on our previous methods for physical reasoning, but bring them towards highly reactive robotic manipulation, robust to an experimenter interfering.

Abstract: Task and Motion Planning has made great progress in solving hard sequential manipulation problems. However, a gap between such planning formulations and control methods for reactive execution remains. In this paper we propose a model predictive control approach dedicated to robustly execute a single sequence of constraints, which corresponds to a discrete decision sequence of a TAMP plan. We decompose the overall control problem into three sub- problems (solving for sequential waypoints, their timing, and a short receding horizon path) that each is a non-linear program solved online in each MPC cycle. The resulting control strategy can account for long-term interdependencies of constraints and reactively plan for a timing-optimal transition through all constraints. We additionally propose phase backtracking when running constraints are missed, leading to a fluent re-initiation behavior that is robust to perturbations and interferences by an experimenter.

Source Code: The code specific to these experiments is available here: The repository contains the actual code base, including the general SecMPC implementation (in src/BotOp/SecMPC). Please install botop in a folder next to the 22-SecMPC repository.

Funding: The research has been supported by the German Research Foundation (DFG) under Germany’s Excellence Strategy EXC 2002/1–390523135 “Science of Intelligence”.


  • Push-with-Stick with the experimeter interfering: youtube
  • Pick-and-Place with the experimeter interfering: youtube
  • Paper accompanying video (short parts of the above): youtube