As robotics enters the “robust perception age”, a major focus of modern robotics research will be the design of perception systems capable of operating over extended periods of time in a broad range of environments. Visual perception in particular holds great promise in this area due to the wealth of information available from standard colour cameras. Indeed, we humans rely heavily on vision for navigating our daily lives. But how can we use vision to build persistent maps and localize against them when the appearance of the world is always changing?

Lee’s research focused on developing ways for robots to reason about more than just the geometry of their environment by incorporating information about illumination and appearance into the mapping and localization problem. In particular, he applied machine learning algorithms to create robust data-driven models of visual appearance, and used these models as an enabler of long-term visual navigation.

Projects he worked on included:

Modelling Appearance Change for Long-term Visual Localization

CAT-Net learns to transform images to correspond to a previously-seen reference appearance.

Learning Matchable Image Transformations for Long-term Metric Visual Localization
Lee Clement, Mona Gridseth, Justin Tomasi and Jonathan Kelly
IEEE RA-L and ICRA 2020. Paris, France.

Matchable Image Transformations for Long-term Metric Visual Localization
Lee Clement, Mona Gridseth, Justin Tomasi and Jonathan Kelly
CVPR Image Matching Workshop 2019. Long Beach, USA.

How to Train a CAT: Learning Canonical Appearance Transformations for Direct Visual Localization Under Illumination Change
Lee Clement and Jonathan Kelly
IEEE RA-L and ICRA 2018. Brisbane, Australia.

Visual Sun Sensing

Sun-BCNN regresses the 3D direction of the sun to improve stereo VO.

Inferring sun direction to improve visual odometry: A deep learning approach
Valentin Peretroukhin, Lee Clement, and Jonathan Kelly
IJRR 2018.
Reducing Drift in Visual Odometry by Inferring Sun Direction using a Bayesian Convolutional Neural Network
Valentin Peretroukhin, Lee Clement, and Jonathan Kelly
ICRA 2017. Singapore.
Improving the Accuracy of Stereo Visual Odometry Using Visual Illumination Estimation
Lee Clement, Valentin Peretroukhin, and Jonathan Kelly
ISER 2016. Tokyo, Japan.

Monocular Visual Teach & Repeat

MonoVT&R is capable of retracing human-taught routes with centimetre accuracy using only a monocular camera.

Robust Monocular Visual Teach and Repeat Aided by Local Ground Planarity and Colour-Constant Imagery
Lee Clement, Jonathan Kelly, and Timothy D. Barfoot
JFR 2017.
Monocular Visual Teach and Repeat Aided by Local Ground Planarity
Lee Clement, Jonathan Kelly, and Timothy D. Barfoot
FSR 2015. Toronto, Canada.

Shui Song visited us from NTU, Singapore, to work on semantic segmentation and mapping for manipulation tasks.

Juraj worked on mm-wave radar as part of ongoing research for his PhD in the LAMoR group at the University of Zagreb.

Brian developed extensions to our deep appearance modelling framework. Now a Machine Learning Engineer at Google, Mountain View.

Richard worked on an advanced SLAM package for the RSX Rover Challenge Team at U of T. Now a Software Engineer at IBM, Markham.

Soroush helped to devise several algorithms for certifiably optimal estimation. Now a B.A.Sc. student in Engineering Science at U of T.

Andrej developed a fast visual pose estimation framework for objects in dense clutter. Now a developer at Fitch Ratings, London, UK.

Ray developed tools to support visual navigation of our tail sitter aerial platform. Now at Uber, San Francisco.

Ziye applied convex optimization to robot mapping problems. Currently a Ph.D. student in Prof. Somayeh Sojoudi’s lab at UC Berkeley.

Tobias completed his Master’s at the STARS lab on exchange from ETH Zurich. He worked on human-robot cooperative manipulation.