3D Bounding Box Detection & Tracking on Highway

Описание: Let's suppose that you want to detect all types of vehicles on a highway from a camera located on a bridge. The goal of this project is to detect them while estimating their dimensions and not to detect any type of vehicles (car, truck, van etc).

The main problem to do it with a monocular camera, in comparison to RGB-D camera or LiDAR, is to regress 3D dimensions from 2D image. Moreover, obtaining a groundtruth of a real-life record requires huge means. This is why I decided to use the Carla simulator to build up my dataset where the bounding box dimensions are perfectly known.

Once the training is done, you can track cars individually and generate statistics on it (averaged speed, dimensions distributions...).

My contribution has 4 components:
a camera calibration algorithm to find camera pose above the highway (pitch, yaw, height etc)
a python script to interact with Carla simulator and generate a groundtruth with a similar camera (intrinsic/extrinsic)
a Centernet-based network to regress positions and dimensions (we predict width, height and length divided by the depth)
a tracking algorithm with a Kalman Filter

As you can see performances are much better with Carla than with real life, but it still demonstrates that it is possible to use it for real life scenarios. Imagine how good this could be if I had a real-life record with annotated bounding boxes!

00:00 Multi Object tracking (for real)
01:13 Camera calibration
01:30 Groundtruth generation with Carla simulator
01:57 Multi Object detection (Carla sample)
02:20 Multi Object tracking (Carla sample)
02:46 Multi Object tracking (for real)

Github code : https://github.com/antoinekeller/3d_b...

Like, Share & Subscribe ;)

[Music] Fredji - Happy Life