Pedestrian Intention Prediction

Product Management / Research — Volvo Cars × UC Berkeley × Chalmers University

Model A - Pedestrian Intention Prediction
Model C - Fusion-based Prediction

Overview

Collaborative research project between Volvo Cars, and student teams from UC Berkeley and Chalmers University. The project was aimed at predicting the intention of a pedestrian to cross or not cross the road — a successful attempt to emulate the behavior of a human driver to guess the intention of fellow road users such as pedestrians.

Approach

Our team based the approach on a research paper that had some progress in this line of work. After replicating the paper, our teams decided on experimenting with additional approaches and eventually arrived at Fusion-based Intention Prediction Networks.

The first model consisted of 3 components:

Object Detector (YOLOv3) — dissects video into frames and localizes pedestrians by plotting bounding boxes
Object Tracker (SORT) — tracks detected pedestrians with unique IDs across frames
3D Scene Classifier (DenseNet) — uses pedestrian information to predict crossing intention (green = non-crossing, red = crossing)

The challenge was to predict intention at least 0.5 seconds before the intended action occurred, which from a safety perspective was required for the car to make any necessary decision.

Experimentation

Our teams experimented with alternate components:

Trackers: SORT, DeepSORT, Pose-Estimation based detection and tracking
Classifiers: Recurrent Neural Networks, Random Forest, DenseNets
Feature engineering: applied skeletons using Pose-Estimator and retrained the classifier

Results

Using the new classifier and additional feature engineering, we achieved an improved AP score of 0.89 — surpassing the original research paper. The model successfully predicts pedestrian intent 0.5 seconds before the intended action occurs.

Moraldeep Sachdeo