Lucas-Kanade Reloaded:
End-to-End Super-Resolution from Raw Image Bursts

Bruno Lecouat1,2   Jean Ponce1,3   Julien Mairal2

1 Inria and DIENS (ENS-PSL, CNRS, Inria), Paris, France  ·  2 Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK, Grenoble, France
3 Center for Data Science, New York University, New York, USA

ICCV 2021

Paper Results BibTeX

Super-resolution results from a raw burst

A burst of raw frames from a handheld camera, merged into a single high-resolution image.

Abstract

Our work addresses the problem of reconstructing a high-resolution image from multiple lower-resolution snapshots captured from slightly different viewpoints in space and time. Key challenges for solving this super-resolution problem include (i) aligning the input pictures with sub-pixel accuracy, (ii) handling raw (noisy) images for maximal faithfulness to native camera data, and (iii) designing/learning an image prior (regularizer) well suited to the task. We address these three challenges with a hybrid algorithm building on the insight from Wronski et al. that aliasing is an ally in this setting, with parameters that can be learned end to end, while retaining the interpretability of classical approaches to inverse problems. The effectiveness of our approach is demonstrated on synthetic and real image bursts, setting a new state of the art on several benchmarks and delivering excellent qualitative results on real raw bursts captured by smartphones and prosumer cameras.

Results on real bursts

Each scene opens two synchronized deep-zoom viewers: the camera JPEG on the left, our reconstruction on the right, typically 16 times more pixels. Drag to pan, scroll to zoom.

Latest results

With chromatic aberration reduction and adaptive sharpening.

Fleurs Fleurs Guernsey Guernsey I Guernsey Guernsey II Spain Spain I Spain Spain II Mountains Mountains Objects Objects Fly Fly Kikki Kikki

Earlier results

With adaptive sharpening only.

Friant FriantPixel 4a New York New YorkPixel 3 Paris ParisPixel 3a Street Street IPixel 3a Street Street IIPixel 3a Cherry tree Cherry treePixel 3 Stairs StairsLumix GX9 Fly FlyLumix GX9 Mountains MountainsLumix GX9 Night NightLumix GX9 Pool PoolGalaxy S3

More visualizations

BibTeX

@inproceedings{lecouat2021lucas,
  title     = {Lucas-Kanade Reloaded: End-to-End Super-Resolution from Raw Image Bursts},
  author    = {Lecouat, Bruno and Ponce, Jean and Mairal, Julien},
  booktitle = {International Conference on Computer Vision (ICCV)},
  year      = {2021}
}

Acknowledgements

We thank Frédéric Guichard for useful discussions and comments, and the authors of Deep Burst Super-Resolution for providing their qualitative and quantitative results. This work was funded in part by the French government under management of Agence Nationale de la Recherche as part of the “Investissements d'avenir” program, reference ANR-19-P3IA-0001 (PRAIRIE 3IA Institute). JM and BL were supported by the ERC grant number 714381 (SOLARIS project) and by ANR 3IA MIAI@Grenoble Alpes (ANR-19-P3IA-0003). JP was supported in part by the Louis Vuitton/ENS chair in artificial intelligence and the Inria/NYU collaboration. This work was granted access to the HPC resources of IDRIS under the allocation 2020-AD011011252 made by GENCI.