AI City Competition: ImageNet for Intelligent Transportation Video Analysis

"ImageNet competition in the field of intelligent transportation video analysis"-NVIDIA City Challenge came to an end. The Panasonic Research Institute of Singapore and the Institute of Automation of the Chinese Academy of Sciences proposed a dual-mode dynamic and static joint detection scheme, which won the first place in the traffic anomaly detection competition, achieving high detection accuracy while maintaining extremely low detection time errors. Because the proposed scheme has the advantages of high accuracy, strong robustness, and low sensitivity to data resolution, it can be used as a reliable solution for anomaly detection in real scenarios.

In 2020, there will be one billion cameras worldwide. In the future, intelligent video analysis can solve some city-scale problems such as transportation, public safety, crime prevention, and efficient resource utilization, and improve the user experience of large public and private spaces (such as shopping malls, stadiums, train stations, airports, etc.).

Although there are currently some corpora and benchmarks for video retrieval (for example, NIST TRECVID), for transportation or public safety, there is still a lack of large-scale tagged high-quality video data corpora, and video analysis should be applied in reality. In world issues, such a database is essential.

The NVIDIA AI City Challenge is proposed in this context. Inspired by the ImageNet dataset, the AI ​​City competition aims to create high-quality video datasets, enabling academic and industrial research teams around the world to promote the latest technologies in various intelligent video analysis. Solve real-world problems.

The AI ​​City Challenge was initiated by Nvidia in collaboration with the State University of New York at Albany, Iowa State University, San Jose State University, and University of Washington. It attracted participation from many universities and institutions around the world, including the University of Washington, UIUC, Peking University, and IBM.

The competition started on January 20, 2018 and ended on April 5, with a total of three tasks (tracks). At first, more than 70 teams signed up, but due to various reasons such as difficult problems and tight time, 22 teams finally submitted their plans. At CVPR 2018, the organizer announced the results [2].

Among them, the University of Washington won the championship of task 1 and task 3, and the championship of task 2 traffic anomaly detection was won by the Shen Shengmei team of the Panasonic Research Institute in Singapore and the Institute of Automation of the Chinese Academy of Sciences. The runner-up of Task 2 is the team from Beijing University of Posts and Telecommunications.

AI City Competition: ImageNet for Intelligent Transportation Video Analysis

Last year, the first AI City Challenge focused on object detection, localization and classification of the video corpus of traffic cameras at fork roads. For the second AIC18 this year, the organizers decided to "out of the bounding box" and move closer to scenarios that are closer to real-world problems.

For example, the U.S. Road Traffic Administration urgently needs a system that can automatically analyze traffic video content, because it is unrealistic for humans to watch all these videos; in fact, the large amount of data collected by many cameras is wasted and not obtained. use efficiently.

After consulting with multiple transportation agencies and public safety agencies, the competition organizers focused on three tasks:

Estimate traffic flow characteristics, such as the precise location and speed of each visible vehicle at any time.

Use unsupervised methods to detect abnormal situations caused by collisions, parking, etc. This allows people to also participate in the decision-making process, paying attention to meaningful visual information in order to intervene in time to save lives.

Multi-camera tracking and object re-identification in urban environments. This is very useful for traffic analysis, as well as for the identification and prevention of crimes. It also enables users to react as quickly as possible to ongoing events.

The data for the competition was collected from data on selected intersections and highways from multiple cities and states in the United States (see the figure below). They set up a special team as a control group and used the data generated by these vehicles to generate Ground Truth.

AIC18 data set: data samples obtained from highways in Iowa and Silicon Valley

Traffic anomaly detection: problem definition is difficult, data is scarce, algorithm design is difficult

Anomaly detection of traffic accidents plays a very critical role in a safe city. Abnormal traffic conditions will greatly reduce the efficiency of traffic. Therefore, it needs to be detected and monitored. If an abnormality occurs, alarm and rescue will be carried out, and the incident will be discharged as soon as possible. The traffic is inconvenient and normal traffic is restored.

Traditional traffic anomaly detection methods mainly include electromagnetic induction loop coil type and wave type, which are based on the use of reflected wave frequency changes to detect vehicle information when vehicles pass by. However, the use of "magnetic" and "wave" detection methods can not provide comprehensive traffic information, so there are great limitations.

Video-based traffic anomaly detection is based on the use of visual sensors such as cameras and computer vision theory, relying on traffic flow information (such as optical flow) and tracking individual vehicles to detect anomalies, such as object tracking methods to identify abnormal behaviors [3].

However, the acquired vehicle trajectory usually contains a certain amount of noise due to occlusion problems. In order to overcome this problem, some scholars try to use the low-level features in the video to propose a probabilistic model-based anomaly detection framework for judgment [4] [5].

It is difficult to define abnormal traffic behaviors in actual scenes, and it is impossible to obtain abnormal prior information. At the same time, there are changes in camera perspective, high-density traffic, target occlusion, weather conditions (such as snowy weather), and changes in illumination (such as daytime and Dark night), affected by factors such as the low resolution of the collected data and the lack of real scene data, designing a robust anomaly detection system is facing huge challenges.

Champion technology sharing: the first dual-mode dynamic and static combined detection method

The video of Task 2 is a real traffic video recorded in various perspectives, under different weather and road conditions. These problems make it very difficult to design anomaly detection methods. Therefore, most successful methods are based on traffic motion flow analysis (for example, using optical flow) rather than trying to detect and track individual vehicles.

The scheme proposed by the champion team is called "Dual-Mode Vehicle Motion Pattern Learning" [1]. The person in charge of the competition and the first author of the paper Xu Yan introduced to Xin Zhiyuan that in the task of traffic anomaly detection, they are the first to propose a dual-modal dynamic and static joint detection method to solve this problem.

"We have fully explored the characteristics of traffic anomaly detection, that is, the speed of abnormal vehicles has obvious changes and the vehicles will be forced to stop. Based on these two characteristics, we have proposed two dynamic and static modes for analysis."

The result of the dynamic scenario. The above figure shows the normal speed, and the figure below shows the abnormal situation. In the chart on the right side of the lower row, you can see that one car has a different speed from the others, and finally stops.

"In the dynamic scheme, we first detect the driving vehicles, and track each vehicle based on reliable optical flow information. At the same time, we designed a simple and effective noise removal method based on the neighbor rule to solve the problem of vehicles in the tracking process. Occlusion problem. After obtaining a relatively robust vehicle trajectory, by calculating the relative speed of different vehicles and their changing trends, we can accurately find abnormal vehicles.

"In the static solution, we first model the background of the video and extract the background image. At the same time, abnormal vehicles will also be kept in the background image. We trained a two-classifier that only contains background and vehicles to effectively remove the background. The false positive result of the picture during vehicle detection.

"Finally, we proposed a scheme based on vehicle re-identification, which integrates the two modalities, which greatly improves the accuracy of the entire inspection system. Compared with other schemes, we propose a dual-modal dynamic and static detection scheme. It has the advantages of high accuracy, strong robustness, and low sensitivity to data resolution."

Flow chart for detecting road abnormalities based on static vehicle analysis

Summary: data, data, data

Xu Yan said that they put a lot of energy into this competition. From the very beginning, they were familiar with the rules of the game, data download analysis, task problem definition, different algorithm design in the middle, rapid experimental verification, failed result analysis, and many more at the end. Model fusion, submit experimental results, "we do our best in every step."

Among them, the most energy-consuming step is algorithm design and rapid verification. The team tried to propose several different solutions at the time, and through rapid experimental verification methods, only two solutions (bimodal analysis) were selected to submit the results.

At that time, one plan was to try to use the supervised method to collect some data, label the data, and finally use it for training. However, the experimental results were unsatisfactory. The reason was that the amount of collected data was too small, and the difference with the test data was obvious. After rapid experimental verification, the team finally gave up this scheme.

Due to the difficulty in defining anomaly scenes and the lack of training data, anomaly detection in monitoring scenes is a very challenging task. Based on the analysis of the test data, the team of Panasonic Shen Shengmei explored its characteristics and designed effective rules specifically to achieve very good results on the test data set.

"Through the observation of the failure results, the model we designed will miss extremely small abnormal vehicles, or incorrectly detect the background as a vehicle," Xu Yan said: "The reason is that the vehicle detection model we used does not perform well on small vehicles. Good, and the data for training the detection model is inconsistent with the data distribution of the final test, so improving the performance of the detection model is a direction for future work."

"On the other hand, our model performs well under the two defined abnormal conditions (car accident and breakdown), but may not perform well for other abnormalities (such as illegal vehicle steering). If we can obtain real data similar to the abnormality By analyzing these data, we can master some laws and use these data to train the model. As a task of supervised learning, it will greatly improve the robustness of the system."