Bird's-eye-view (BEV) Perception: A Survey and Collection

Awesome BEV perception papers and toolbox for achieving state-of-the-arts performance.

Table of contents

• Bird's-eye-view Perception: A Survey and Collection
  • Table of contents
  • Introduction
    • Major Features
  • What's New
  • BEV Algorithm Family
  • BEV Toolbox
  • Literature Survey
  • License and Citation

Introduction

This repo is associated with the survey paper "Delving into the Devils of Bird’s-eye-view Perception: A Review, Evaluation and Recipe", which provides an up-to-date literature survey for BEV percption and an open source BEV toolbox based on PyTorch. We also introduce BEV algorithm family, incluidng follow-up work on BEV percepton such as VCD, GAPretrain and FocalDistiller. We hope this repo can not only be a good starting point for new beginners but also help current researchers in the BEV perception community.

If you find some work popular enough to be cited below, shoot us email or simply open a PR!

Major Features

• SOTA BEV Algorithm Family
  We include important follow-up works of BEVFormer/BEVDet/BEVDepth in different aspects, ranging from plug-and-play tricks to knowledge distillation(GAPretrain, FocalDistiller). More detail of each paper can be found in each README.md file under here.
• Convenient BEVPerception Toolbox
  We integrate bag of tricks in the BEV toolbox that help us achieve 1st in the camera-based detection track of the Waymo Open Challenge 2022, which can be used indedependly or as a plug-in for mmdet3d and detectron2. Moreover, we provide a suitable playground for new-beginners in this area, including hands-on tutorial and small-scale dataset (1/5 WOD in kitti format) to validate idea. More detail can be found here.

Bag of Tricks

Multiple View Data Augmentation	BEV encoder	Heads family	Loss	Post-Process
Random Flip Random Multi-scale Resize Grid Mask	BEVFormer encoder Voxel-SPVCNN encoder	Deformable DETR head FreeAnchor head Centerpoint head	Camera-only detection: auxiliary 2D detection loss auxiliary depth supervision LiDAR segmentation: Geo loss Lovasz loss	Test-time Augmentation Weighted Box Fusion Two-stage Ensemble

• Up-to-date Literature Survey for BEV Perception
  We summarize important methods in recent years about BEV perception including different modalities (camera, LIDAR, Fusion) and tasks (Detection, Segmentation, Occupancy). More detail of the survey paper list can be found here.

What's New

[2023/10/26]: A new paper VCD are comming soon with official implementation.

[2023/09/06]: We have a new version of the survey. Check it out!

[2023/04/06]: Two new paper GAPretrain and FocalDistiller are comming soon with official implementation.

[2022/10/13]: v0.1 was released.

• Integrate some practical data augmentation methods for BEV camera-based 3D detection in the toolbox.
• Offer a pipeline to process the Waymo dataset (camera-based 3D detection).
• Release a baseline (with config) for Waymo dataset and also 1/5 Waymo dataset in Kitti format.

Please refer to changelog.md for details and release history of the toolbox code.

BEV Algorithm Family

The BEV algorithm family includes follow-up works of BEVFormer in different aspects, ranging from plug-and-play tricks to pre-training distillation. All paper summary is under nuscenes_playground alongwith official implementation, check it out!

• VCD (NeurIPS 2023)
  Leveraging Vision-Centric Multi-Modal Expertise for 3D Object Detection. More detail can be found in nuScenes_playground/VCD.
• GAPretrain (arXiv)
  Geometric-aware Pretraining for Vision-centric 3D Object Detection. More detail can be found in nuScenes_playground/GAPretrain.md.
• FocalDistiller (CVPR 2023)
  Distilling Focal Knowledge from Imperfect Expert for 3D object Detection. More detail can be found in nuScenes_playground/FocalDistiller.md.

BEV Toolbox

The BEV toolbox provides useful recipe for BEV camera-based 3D object detection, including solid data augmentation strategies, efficient BEV encoder design, loss function family, useful test-time augmentation, ensemble policy, and so on. Please refer to bev_toolbox/README.md for more details.

Literature Survey

The general picture of BEV perception at a glance, where consists of three sub-parts based on the input modality. BEV perception is a general task built on top of a series of fundamental tasks. For better completeness of the whole perception algorithms in autonomous driving, we list other topics as well. More detail can be found in the survey paper.

We have summarized important datasets and methods in recent years about BEV perception in academia and also different roadmaps used in industry.

• Academic Summary of BEV Perception
  • Datasets for BEV Perception
  • BEV Camera
  • BEV Lidar
  • BEV Fusion
• Industrial Roadmap of BEV Perception

We have also summarized some conventional methods for different tasks.

• Conventional Methods Camera 3D Object Detection
• Conventional Methods LiDAR Detection
• Conventional Methods LiDAR Segmentation
• Conventional Methods Sensor Fusion

License and Citation

This project is released under the Apache 2.0 license.

If you find this project useful in your research, please consider cite:

@article{li2022bevsurvey,
  title={Delving into the Devils of Bird's-eye-view Perception: A Review, Evaluation and Recipe},
  author={Li, Hongyang and Sima, Chonghao and Dai, Jifeng and Wang, Wenhai and Lu, Lewei and Wang, Huijie and Zeng, Jia and Li, Zhiqi and Yang, Jiazhi and Deng, Hanming and Tian, Hao and Xie, Enze and Xie, Jiangwei and Chen, Li and Li, Tianyu and Li, Yang and Gao, Yulu and Jia, Xiaosong and Liu, Si and Shi, Jianping and Lin, Dahua and Qiao, Yu},
  journal={arXiv preprint arXiv:2209.05324},
  year={2022}
}

@misc{bevtoolbox2022,
  title={{BEVPerceptionx-Survey-Recipe} toolbox for general BEV perception},
  author={BEV-Toolbox Contributors},
  howpublished={\url{https://github.com/OpenPerceptionX/BEVPerception-Survey-Recipe}},
  year={2022}
}