For Edward Johns' Google Scholar profile, click here.
For the lab's YouTube channel, click here.

Below are the key peer-reviewed publications from the lab. 
Click on each row to visit the project page, read the paper, and watch the video.

2021 IROS Coarse-to-Fine for Sim-to-Real

Coarse-to-Fine for Sim-to-Real: Sub-Millimetre Precision Across Wide Task Spaces

Eugene Valassakis, Norman Di Palo, and Edward Johns

IROS 2021

Mini Abstract We develop a framework which allows for precise sub-millimetre control for zero-shot sim-to-real transfer, whilst also enabling interaction across a wide range of object poses. Each trajectory involves first a coarse, ICP-based planning stage, followed by a fine, end-to-end visuomotor control stage.


Coarse-to-Fine Imitation Learning: Robot Manipulation from a Single Demonstration

Edward Johns

ICRA 2021

Mini Abstract We propose a method for visual imitation learning, which can learn novel, everyday tasks, from just a single human demonstration. First, sequential state estimation aligns the end-effector with the object, which is trained with self-supervised learning, and second, the robot simply repeats the original demonstration velocities.


DROID: Minimizing the Reality Gap using Single-Shot Human Demonstration

Ya-Yen Tsai, Hui Xu, Zihan Ding, Chong Zhang, Edward Johns, and Bidan Huang

RA-Letters and ICRA 2021

Mini Abstract We introduce a dynamics sim-to-real method, which exploits a single real-world demonstration. Simulation parameters are optimised by attempting to align the simulated and real-world demonstration trajectories. An RL-based policy can then be trained in simulation and applied directly to the real world.


Benchmarking Domain Randomisation for Visual Sim-to-Real Transfer

Raghad Alghonaim and Edward Johns

ICRA 2021

Mini Abstract We benchmark the design choices in domain randomisation for visual sim-to-real. Evaluation is done on a simple pose estimation task. Results show that a small number of high-quality images is better than a large number of low-quality images, and that both random textures and random distractors are effective.


Crossing the Gap: A Deep Dive into Zero-Shot Sim-to-Real Transfer for Dynamics

Eugene Valassakis, Zihan Ding, and Edward Johns

IROS 2020

Mini Abstract We benchmark sim-to-real for tasks with complex dynamics, where no real-world training is available. We show that previous works require significant simulator tuning to achieve transfer. A simple method which just injects random forces, outperforms domain randomisation whilst being significantly easier to tune.


Physics-Based Dexterous Manipulations with Estimated Hand Poses and Residual Reinforcement Learning

Guillermo Garcia-Hernando, Edward Johns, and Tae-Kyun Kim

IROS 2020

Mini Abstract We introduce a method for dexterous object manipulation in a virtual environment, using a hand pose estimator in the real world. Residual reinforcement learning, trained in a physics simulator, learns to correct the noisy pose estimator. Rewards use adversarial imitation learning, to encourage natural motion.


Shape Adaptor: A Learnable Resizing Module

Shikun Liu, Zhe Lin, Yilin Wang, Jianming Zhang, Federico Perazzi, and Edward Johns

ECCV 2020

Mini Abstract We introduce a method to optimise a neural network's shape for a given dataset. Traditional resizing layers, such as max-pooling and striding, use a fixed resizing ratio based on heuristics, whereas we can now train these ratios end-to-end. Results show improvements over hand-engineered network architectures.

Sim-to-Real Transfer for Optical Tactile Sensing

Zihan Ding, Nathan Lepora, and Edward Johns

ICRA 2020

Mini Abstract We train a TacTip optical tactile sensor to detect edge positions and orientations. Training is done with simulated data using a soft-body model, and transfer to the real world is achieved by randomising simulator parameters. Real-world tests show that edge positions can be predicted with an error of less than 1mm. 


Constrained-Space Optimization and Reinforcement Learning for Complex Tasks

Ya-Yen Tsai, Bo Xiao, Edward Johns, and Guang-Zhong Yang

RA-Letters and ICRA 2020

Mini Abstract We train a robot to perform a complex sewing task. Human demonstrations are used to create a constrained space, and reinforcement learning in simulation optimises a trajectory by exploring this space. Real-world experiments show that this optimised trajectory is superior to any one of the individual demonstrations.


Self-supervised Generalisation with Meta Auxiliary Learning

Shikun Liu, Andrew Davison, and Edward Johns

NeurIPS 2019

Mini Abstract We train a CNN for image recognition, using automatically-generated auxiliary labels. A second CNN generates these auxiliary labels using meta learning, by encouraging labels which assist the primary task. Experiments show that the performance on the primary task is as good as using human-defined auxiliary labels.

End-to-End Multi-Task Learning with Attention

Shikun Liu, Edward Johns, and Andrew Davison

CVPR 2019

Mini Abstract We train a multi-task CNN which can share features across different tasks. A common trunk of features is learned, and each task applies a soft attention mask to the common pool, where the attention masks are learned end-to-end. Our method achieves state-of-the-art performance on dense image prediction.


Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task

Stephen James, Andrew J. Davison, and Edward Johns

CoRL 2017

Mini Abstract We train an end-to-end robot controller to grasp a cube from multiple positions, and drop it into a basket. Training is done in simulation with behavioural cloning, and is transferred to the real world with domain randomisation. In real-world experiments, we show robustness to distractor objects and illumination changes.

Deep Learning a Grasp Function for Grasping under Gripper Pose Uncertainty

Edward Johns, Stefan Leutenegger, and Andrew Davison

IROS 2016

Mini Abstract We train a robot to grasp novel objects using depth images. Data is collected in simulation by attempting grasps across a wide range of objects. A CNN is trained to predict the grasp quality across a regular grid of gripper poses, which is combined with the gripper's pose uncertainty to create a robust grasp.


Pairwise Decomposition of Image Sequences for Active Multi-View Recognition

Edward Johns, Stefan Leutenegger, and Andrew Davison

CVPR 2016 (oral)

Mini Abstract We propose a multi-view object recognition pipeline which can recognise objects over arbitrary camera trajectories. Image sequences are decomposed into pairs for pairwise classification with a CNN, and a second CNN is trained to predict the next-best view. State-of-the-art results are achieved on ModelNet.