Many traffic accidents occur in parking lots. One of the serious safety risks is vehicle-pedestrian conflict. Moreover, with the increasing development of automatic driving and parking technology, parking safety has received significant attention from vehicle safety analysts. However, pedestrian protection in parking lots still faces many challenges. For example, the physical structure of a parking lot may be complex, and dead corners would occur when the vehicle density is high. These lead to pedestrians’ sudden appearance in the vehicle’s path from an unexpected position, resulting in collision accidents in the parking lot. We advocate that besides vehicular sensing data, high-precision digital map of the parking lot, pedestrians’ smart device’s sensing data, and attribute information of pedestrians can be used to detect the position of pedestrians in the parking lot. However, this subject has not been studied and explored in existing studies. To fill this void, this paper proposes a pedestrian tracking framework integrating multiple information sources to provide pedestrian position and status information for vehicles and protect pedestrians in parking spaces. We also evaluate the proposed method through real-world experiments. The experimental results show that the proposed framework has its advantage in pedestrian attribute information extraction and positioning accuracy. It can also be used for pedestrian tracking in parking spaces.
It is difficult to regulate the behavior of vehicles and pedestrians in parking lots because there is not enough traffic guidance information, such as signals and signs on the road. For example, a car suddenly coming out of a parking space; a car driving in front of a pedestrian and then reversing into a garage; or a pedestrian suddenly appearing in front of a car. There are potential hazards everywhere that threaten pedestrian safety. With the development of advanced driver assistance systems and automated parking systems, the issue of pedestrian safety in parking vehicles has also attracted the attention of academics and engineers. To detect hazards as soon as possible, it is necessary to pay attention to the people around and perform adequate safety confirmation. Especially for pedestrians, even if the vehicle moves slowly, there is a risk of vehicle-person collision injuries and sometimes even fatal accidents. As representatives of vulnerable road users, older people and children need special protection. Older people have a narrower vision and weaker hearing; therefore, they often do not notice oncoming cars. Additionally, if they pay attention to the vehicle in front of them, their attention to other cars becomes negligent because it is not easy to divide their attention as they age. A study published in the Journal of Safety Research [
As one of the most important functions of the autonomous driving system, pedestrian detection systems have become a hot topic of research and development in recent years. It is typically integrated into collision avoidance systems using radar cameras and sensors to detect pedestrians and slow down and brake in time to reduce accident injuries. Major car manufacturers have introduced advanced pedestrian detection systems to identify pedestrians on the road and perform dynamic analysis to predict whether they will suddenly break into the driving route. In addition to traditional automotive manufacturers, Internet companies are developing pedestrian detection systems to enable smart mobility. For example, Google’s latest pedestrian detection system relies on camera images to capture pedestrians’ movement and optimize the efficiency of pedestrian detection. Other computer programs, such as vehicle-assisted driving systems, intelligent video surveillance, robotic navigation, drone monitoring, and pedestrian tracking, are also beginning to be applied to pedestrian protection. However, most of these existing methods are based on detecting sensors, such as camera, LiDAR, and RADAR, which are powerless when dealing with complex conditions and observing dead zone in parking lots. Therefore, we advocate that in addition to information from sensors, such as LiDAR and cameras, multiple sources of information, such as high-precision maps, pedestrian device sensors, and pedestrian attribute information, can be used to detect pedestrian location in parking lots. However, this subject has not been studied and explored yet. To fill this gap, we propose a pedestrian tracking method that integrates multiple information sources to provide pedestrian location and status information for vehicles to protect pedestrians in parking spaces. The contributions of this paper are summarized as follows: A framework for persistent pedestrian tracking in parking space. We proposed in this paper a framework that can be used to estimate the pedestrian position under occlusion and pedestrian attribute information extraction. Implementation of the proposed pedestrian tracking framework. We develop a pedestrian tracking system for pedestrian tracking in parking space using C++. An evaluation of the proposed system. To evaluate the proposed framework and system, we carry out real-world experiments in the parking space of a municipal service center. The experimental results indicate that compared with sensor fusion based method, the proposed framework has its advantage in pedestrian attribute information extraction and positioning accuracy.
The remainder of this paper is organized as follows. Section II introduces the work related to pedestrian detection and tracking. Section III introduces the structure, target, and function of the proposed framework for pedestrian detection and tracking. Section IV describes the implementation of the proposed method. In Section V, we conduct an experiment to evaluate the proposed method. Section VI discusses the effectiveness, novelty, and limitations of the proposed framework. Finally, Section VII presents the conclusion and future work.
There are many published pedestrian protection theories and methods. As a pioneering work, Gandhi and Trivedi conducted a systematic literature review of research on the enhancement of pedestrian safety to develop a better understanding of the nature, issues, approaches, and challenges surrounding the problem [
The most popular pedestrian detection and tracking solutions are object tracking methods based on passive sensors, such as surveillance cameras, LiDAR, and RADAR. Dai et al. proposed an improved labeled multi-Bernoulli (LMB) filter with inter-target occlusion handling ability for multi-extended target tracking using laser range finder [
In addition to these passive tracking methods, there are pedestrian tracking methods based on active devices, such as smartphones and inertial measurement unit (IMU) sensors carried by road users. Park et al. designed a stand-alone pedestrian tracking system for indoor corridor environments using smartphone sensors, such as a magnetometer and accelerometer [
Although several pedestrian tracking methods have been explored, the results of our literature survey suggest that there is no research on pedestrian tracking using multiple information sources (e.g., LiDAR and global positioning system (GPS) sensor data, IMU sensor data, and pedestrian attribute information) at the same time. On the other hand, there are studies on multiple data integration such as [
This section introduces the structure of the proposed pedestrian tracking system for parking spaces.
The proposed framework for persistent pedestrian tracking in parking spaces is based on an ongoing project of DM consortium called Dynamic Map (DM2.0) platform [
As shown in
The proposed framework is shown in
We use the pedestrian’s present position, walking direction, and attribute information (e.g., position of parking spaces and average walking speed) to predict the pedestrian’s approximate position at the next time step. If the pedestrian can be detected by LiDARs, the optimal matching position of pedestrians is regarded as the optimal estimation position. However, if the pedestrian is occluded, we use the extended Kalman filter to fuse the pedestrian device position (GPS&IMU fusion) and the predicted pedestrian position to obtain the optimal estimation of pedestrian position. When occlusion continues to occur, the optimal estimated value of the pedestrian position at the previous time step is taken as the real position of the pedestrian and iterated into the pedestrian position estimation at the next time step until the LiDAR and GPS matching successfully detects the pedestrian again. Meanwhile, when the occlusion is lifted, we calculate the deviation of the predicted pedestrian position and pedestrian device position data based on the actual pedestrian position to evaluate the reliability of these two information sources and decide the weight for data fusion when occlusion occurs again. Finally, the edge server publishes pedestrians’ position data and attribute information to DM2.0 PF and AVs for traffic data analysis and collision warning, respectively.
This section elaborates on the implementation of the proposed pedestrian tracking framework. We aim to provide a pedestrian tracking system in contract parking space to enhance parking safety. Then, we postulate that our target operating environment is as follows. All parking lot users are equipped with a smart device and have completed the registration of DM2.0 PF. For any parking lot user, a fixed parking space is guaranteed.
The common defect of pedestrian detection and tracking methods based on computer vision and LiDAR is that these methods are ineffective in solving the occlusion handling and pedestrian re-identification problems. Therefore, we advocate that more pedestrian information should be integrated to track pedestrians in an environment prone to produce observation blind spots. Among the easily available pedestrian information, user device data are one of the most meaningful information sources for pedestrian detection and tracking. Generally, smartphones are the most widely used user devices. Although the positioning accuracy of smartphone built-in sensors, such as GPS sensors, are unsatisfactory, it has the advantage of continuously sending pedestrian location information without shielding interference. Additionally, while sending the pedestrian location, the smartphone can send the device ID information to identify the pedestrian. Therefore, we first use the GPS and IMU information of user smartphones to obtain the user ID and imprecise location. For the raw GPS data, we first use the coordinate conversion formula published by the Geospatial Information Authority of Japan to convert the GPS data into the Japanese plane rectangular coordinate system. For the raw IMU data, the rotation matrixes for rotating about the x-axis, y-axis, and z-axis are expressed as
Thus, when the rotation order is x, y, z, the rotation matrix can be expressed as
The component of gravitational acceleration in the Cartesian coordinate system [x, y, z] is given as follows:
In this way, we can obtain the residual component of the pedestrian’s acceleration after removing gravity acceleration. We use the Kalman filter to fuse GPS and IMU data to obtain relatively accurate user device position. The data fusion process consists of two steps: prediction and correction steps.
where
After converting GPS position and LiDAR position data into DM2.0 PF coordinates, we can match the converted GPS data with LiDAR position data to obtain the position of a specific pedestrian through the pedestrian device ID contained in the GPS message. As shown in
where x and y are the x- and y-coordinates of the object detected by LiDAR, respectively; Ex and Ey denote the expected value of the x- and y-coordinates of the GPS source, respectively;
As the actual position coordinate variables of the GPS source follow the normal distribution of the inherent coordinates and variance values of the GPS sensor, we chose multivariate normal distribution for GPS and LiDAR matching. Furthermore, since the vertical sensing accuracy of the smartphone GPS sensor is very low (almost unreliable), we use the PDF of the 2D normal distribution instead of that of the 3D normal distribution. Additionally, our pedestrian tracking system is based on the 2D plane space of the x-y coordinate system. Therefore, we abandon the z-coordinate (height) and use only the x-coordinate and y coordinate positions of the horizontal plane.
As presented in
where
When the pedestrian is occluded, although the exact positions and tracks of the pedestrians are unavailable, we know the exit position of the parking lot (digital map information), user’s parking position (user attribute information managed by DM2.0 PF), and can obtain the pedestrian’s walking direction (toward the exit of the parking lot or his parking space), and instantaneous speed through history average speed data and real-time GPS and IMU data. We use the constant velocity model to estimate the position of pedestrians because the walking speed of pedestrians is relatively slow (usually, 1.1–1.5 m/s), and the acceleration is basically negligible. Furthermore, in addition to pedestrians, there are obstacles, such as parked vehicles in the parking lot (see
The direction of movement of each pedestrian (exit direction or parking lot direction) is first obtained from the GPS data, and the movement speed is calculated from a weighted average of the movement speed of the GPS data and the walking speed of the pedestrian attribute information. Then, we use LiDAR point cloud data to obtain the location of obstacles in the parking space and determine if there is an obstacle on the pedestrian route. If there is no obstacle, the system predicts that the pedestrian will move at a constant speed toward the destination. Meanwhile, if there is an obstacle, the system predicts that the pedestrian will move at a constant speed toward the target point of the adjacent route that is closest to the pedestrian’s original route. Originally, the prediction of the pedestrian’s path should be like a maze search to determine whether there is a path or not. However, according to the actual situation of a parking lot, there is no dead-end, and pedestrians would not go to the exit like a maze search. Therefore, in this section, we do not use recursive or backtracking methods to improve the real-time performance of the proposed system.
Using the proposed methods in Subsections 4.1, 4.2, and 4.3, we can obtain user device, LiDAR and smartphone matching, and predicted pedestrian positions, respectively. However, these position information sources have their advantages and shortcomings. Although smartphone position and predicted pedestrian position data could continuously provide pedestrian position, there may be a certain amount of error between these data and the actual pedestrian position. Meanwhile, LiDAR and smartphone matching results can accurately reflect pedestrians’ position, but it becomes unavailable when occlusion occurs. As these data are unreliable, we advocate fusing on these data to improve data reliability to provide accurate real-time pedestrian position estimation. Therefore, we propose a Kalman filter-based algorithm for continuous pedestrian tracking to carry out the multiple information sources fusion.
where
When LiDARs cannot detect the pedestrians (occlusion occurs), the proposed system estimates the pedestrians’ position by fusing the device position and predicted pedestrian position data. Furthermore, if occlusion occurs continuously, the estimation will continue iteratively. If LiDARs detect pedestrians again, the system outputs the matching results of LiDAR and GPS data with the maximum probability as the accurate position of the pedestrians and evaluate the reliability of the device position and predicted pedestrian position data to determine the Kalman gain for future estimation.
We implemented the proposed framework using C++ and conducted real-world experiments to evaluate our system in the parking space of a municipal service center in Kasugai city, Japan. As shown in
The output of our system includes a unique device ID used to identify the detected pedestrian, pedestrian device position, precise pedestrian position detected by the LiDARs, pedestrian identification probability, and the estimated pedestrian position when occlusion occurs.
Test route | Pedestrian | Tracking accuracy | |||||
---|---|---|---|---|---|---|---|
GPS+IMU-based method | The proposed method | Comparison | |||||
Accumulative error | Variance | Accumulative error | Variance | Accumulative error | Variance | ||
1 | Single | 60.813 | 7.021 | 6.361 | 0.141 | 10.460% | 2.003% |
2 | Single | 21.282 | 9.799 | 1.866 | 0.272 | 8.766% | 2.783% |
3 | Single | 49.810 | 8.047 | 2.586 | 0.166 | 5.191% | 2.067% |
4 | Single | 24.481 | 4.023 | 3.312 | 0.197 | 13.528% | 4.891% |
1 | Multiple | 113.295 | 18.554 | 14.935 | 0.855 | 13.183% | 4.611% |
2 | Multiple | 153.736 | 11.024 | 68.201 | 3.182 | 44.362% | 28.862% |
3 | Multiple | 193.379 | 10.113 | 25.908 | 0.939 | 13.398% | 9.283% |
4 | Multiple | 114.513 | 20.819 | 17.096 | 2.508 | 14.929% | 12.047% |
Avg. | 91.414 | 11.175 | 17.533 | 1.033 | 15.477% | 8.318% |
The GPS signal and IMU signal sent by the user equipment can be used to identify the pedestrians and will not be occluded by obstacles. However, its accuracy is too low to be trusted. LiDAR data have high accuracy and are close to the real pedestrian position. However, it cannot identify the pedestrians, and it cannot detect pedestrians when they are occluded. Our method integrates these two data sources and estimates pedestrians’ trajectory using their attribute information to improve the accuracy of the pedestrian position estimation. Although
Test route | No. of detected |
Tracking accuracy when returning from occlusion | ||
---|---|---|---|---|
GPS+IMU position data | Estimated pedestrian |
|||
1 | 2 | 3.031 | 0.502 | +83.447% |
2 | 6 | 2.003 | 1.360 | +32.090% |
3 | 9 | 4.237 | 1.575 | +62.834% |
4 | 7 | 3.799 | 3.812 | −0.340% |
1 | 3 | 3.354 | 2.490 | +25.763% |
2 | 1 | 2.015 | 1.408 | +30.152% |
3 | 7 | 4.536 | 3.362 | +25.878% |
4 | 6 | 5.726 | 4.023 | +29.734% |
Avg. | 3.588 | 2.317 | +36.195% |
This section discusses the effectiveness, novelty, and limitations of the proposed tracking framework for pedestrian protection in parking spaces.
Occlusion handling is an essential and difficult task in pedestrian detection and tracking. When there is no occlusion or partial occlusion, traditional methods based on sensor fusion and computer vision can well solve the problem of pedestrian detection and tracking. However, traditional methods are generally powerless when severe occlusion or complete occlusion occurs. Only based on road user device sensors can we obtain an inaccurate pedestrian position; however, it is inadequate for collision prevention and pedestrian protection. Therefore, to solve this problem, we proposed a solution for seamless pedestrian tracking. The proposed pedestrian tracking framework uses a Kalman filter to fuse the position information of pedestrian smart devices, LiDAR position information, and pedestrian attribute information. We also conducted a real-world experiment in parking spaces to evaluate the proposed framework. The experimental results show that the proposed method is more reliable than the GPS and IMU sensor fusion-based method. When detecting the pedestrian location, the proposed framework can also detect personal information of pedestrians, such as the elderly and children, to evoke the vigilance of drivers and automatic driving systems for decision-making support or providing voice guidance services. This could help the traffic safety of the parking space. Since the proposed framework tracks and manages real-time pedestrian position information, it is also a driving force for building a more comprehensive DM2.0 platform.
The results of the literature survey in Section 2 shows that there is no research on using pedestrian attribute information for pedestrian tracking.
Method | Target | Approach | Environment | Condition | Experiment |
---|---|---|---|---|---|
Dai et al. |
Vehicles, traffic flow | Laser range finder, |
Outdoor | Occlusion | Real-world, |
Chavez–Garcia et al. (2016) [ |
Pedestrians and vehicles | Sensor fusion |
Outdoor | N/A | Simulation |
Zhao et al. |
Pedestrians, vehicles, bicyclists | Sensor fusion |
Outdoor | Occlusion | Real-world |
Wang et al. |
Pedestrians | LiDAR and SVM | Outdoor | N/A | Real-world |
Zhao et al. |
Pedestrians and vehicles | Multi-LiDAR and BP-ANN | Intersection | N/A | Real-world |
Xu et al. |
Pedestrians | Sensor fusion |
Indoor | N/A | Real-world |
Wang et al. |
Trains, obstacles | Sensor fusion |
Railway track | N/A | Real-world |
Stadler et al. (2021) [ |
Pedestrians | Camera, |
Outdoor/ |
Occlusion | Simulation |
Park et al. |
Pedestrians | Sensor fusion |
Indoor corridor | N/A | Real-world |
Zhu et al |
Vehicles | Multi-data source fusion |
Outdoor | N/A | Real-world, simulation |
Ours | Pedestrians | Multi-data source fusion |
Parking space | Occlusion | Real-world |
Although the proposed pedestrian tracking framework can track pedestrians in parking spaces and deal with occlusion, the experimental results show that the positioning accuracy of multi-person tracking becomes worse than a single-person tracking. This is because the recognition of multiple pedestrians becomes incoherent when occlusion occurs, and the recognition accuracy of relocation is insufficient. The reason is that the system sometimes recognizes the detected pedestrians as pedestrians in occlusion. Additionally, our experiment is conducted in a parking lot with only one exit. It is more challenging to conduct the experiment in a complex environment. The experimental data show that when the continuous occlusion time is long (
This paper presents a parking lot pedestrian tracking framework using LiDAR, pedestrian smart device, and pedestrian attribute information data simultaneously. We also implemented the framework in C++ and conducted experiments in the real-world to verify its rationality and practicability. The experimental results show that the proposed method can reduce the error by 67.31% compared with the method using only GPS and IMU sensor fusion when tracking pedestrians in the parking space. This shows that our system can be applied to the parking lot environment and occlusion handling. The proposed system can be used to extract pedestrian attribute information. However, as stated in the previous section, there are limitations to our research. Solving the limitations is the future direction of this study. For instance, we plan to test the proposed system in a more complex environment with more pedestrians at the same time (e.g., the parking lot of a large shopping mall). Moreover, we aim to implement a pedestrian identification mechanism using machine learning to improve pedestrian re-identification accuracy for occlusion handling.