Radar based detection and tracking of a walking human
Share
Cite
Abstract
In this paper, a radar based approach for pedestrian detection and tracking is presented. The approach utilizes automotive radar for tracking the targets and a separate Doppler radar to provide extra information about the classes of the targets. The classification of objects as humans is based on analysis of the Doppler spectrum of targets as well as the speed of the targets gained through the tracking process. The presented method is able to make a difference between a non-human target and a human target if the human target is moving. At the moment the measurement platform needs to be stationary or move at very low speed.
Keywords
Human detection
tracking
automotive radar
Doppler radar
spectrum analysis
filtering
Radar based detection and tracking of a walking human
Juhana Ahtiainen*, Sami Terho*,
Sampsa Koponen**
*Department of Automation and Systems Technology, **Department of Radio Science and Engineering
Aalto University School of Science and Technology, Espoo, Finland, (e-mail: firstname.lastname@tkk.fi)
Abstract: In this paper, a radar based approach for pedestrian detection and tracking is presented. The
approach utilizes automotive radar for tracking the targets and a separate Doppler radar to provide extra
information about the classes of the targets. The classification of objects as humans is based on analysis
of the Doppler spectrum of targets as well as the speed of the targets gained through the tracking process.
The presented method is able to make a difference between a non-human target and a human target if the
human target is moving. At the moment the measurement platform needs to be stationary or move at very
low speed.
Keywords: Human detection, tracking, automotive radar, Doppler radar, spectrum analysis, filtering
1. INTRODUCTION
Human detection and tracking have been widely researched
topics in recent years. Radars are considered suitable sensors
for detecting and tracking objects. However, the classification
of the objects as humans with radar only, is a very
challenging problem. Therefore, most of the studies in the
field of human detection have been based on other sensors,
such as cameras and laser scanners.
Detecting humans with any kind of sensor system is a
challenging task due to the wide variety of positions and
appearances which humans can assume. A moving sensor
system and stationary people further complicate the task.
Also, the tracking of humans is especially hard since people’s
behaviors are often completely unpredictable. When dealing
with these problems outdoors, the adverse weather conditions
also present major limitations to the sensor system used.
Certain radars provide reliable measurements regardless of
the weather conditions, hence making radars a desirable
option when considering a human detection system.
In the following, a radar based measurement system for
detecting and tracking walking humans is presented. The
measurement system comprises of off-the-shelf automotive
radar and a continuous wave Doppler radar. The automotive
radar is used to track targets and measure their velocities.
Doppler radar is used to analyse whether or not there is a
human in the measurement sector. Information from both
radars is fused to achieve estimate whether a target is a
human or not. Two different test cases are analyzes with and
without the Doppler data revealing the advantages of using
the Doppler radar. The measurement system is able to track
and detect walking humans. Adverse weather conditions
deteriorate the classification of observed targets as humans
but do not affect the performance of the tracker. At the
moment, the sensor system needs to be stationary or move at
relatively slow speed. If the sensor system is moving, the
movements of the platform must be passed to the algorithm.
2. APPROACHES TO HUMAN DETECTION WITH
RADARS
Classifying targets as humans with radars is a challenging
task. There are, however, at least three different kind of
techniques applied to this problem. Yarovoy et al. (2008)
measured the reflectivity of a human in laboratory conditions
with ultra wide band (UWB) radar. They analyzed the
polarization of the reflected signal and discovered that there
are some frequencies where there is a maximum reflectivity
in one polarization and a minimum in the other. The
polarized signal, however, depends strongly on the shape,
posture and position of the person making it therefore very
unreliable method for human classification.
Nakane and Soshi (1998 & 2000) used dual band frequency
modulated continuous wave (FMCW) radar for
discriminating whether or not the detected object is an
animal, including a human. They compared the difference in
the power of the signal reflected form an object at different
frequencies (10 GHz and 60 GHz) and were able to establish
a threshold for the ratio of received intensity between two
frequencies, above which the detected objects can be
identified as animals. They received a patent for their method
in 2000.
Otero (2005) analyzed the Doppler spectrum of CW radar to
obtain a Doppler signature for a walking human. The
movement of a walking human’s legs, arms and torso cause
different components to the Doppler frequency. The motion
of the torso produces a steady Doppler shifted signal, which
is modulated by the motion of swinging arms and legs that
results in a Doppler signature that is very characteristic of
humans. Otero developed a simple, binary classifier that
classifies detection as either a person present or not present.
Thayparan et al. (2004) used wavelet transform to extract the
micro signatures created by human walking. Gürbüz et al.
(2007) evaluated different methods used to detect walking
humans from the Doppler data with the conclusion that
spectrogram based methods are problematic if one needs to
distinguish human from other periodically moving targets
due to high signal-to-noise ratio.
Most of the mentioned studies concentrated on identifying
the human and distinguishing it from other periodically
moving targets, such as walking animals. In the presence of
other moving targets in the radar’s field of view, this may be
impossible because many different spectral signatures are
superimposed together, and the information from one signal
is not enough to distinguish them from each other. However,
there are situations where it is enough to detect any walking
object, but it is not necessary to distinguish animals from
humans. In such cases it is enough to detect just the periodic
motion caused by walking.
3. SYSTEM DESCRIPTION
The measurement system consists of a off-the-shelf
automotive FMCW radar used in adaptive cruise control
systems (ACC) and of a CW Doppler radar used by the police
to catch speeding violators. The characteristics of these radars
are explained in the following chapters.
3.1 Automotive Radar
The automotive radar in the measurement system, a long
range radar 77 GHz generation 2 produced by Bosch, is
described in this chapter. Originally it is manufactured for
ACC systems to measure the distance and relative radial
velocity of vehicles ahead. The radar has four antennas with
superimposed beams giving the radar system an angular
coverage of about 16 degrees. The signal is transmitted
through each of 4 antennas simultaneously.
The radar transmits a continuous wave signal which
frequency is modulated as a function of time with a periodic
wave form. The periodic wave consists of three linear
frequency ramps with different ramp times and bandwidth.
Using three ramps enables the radar system to detect the
range and relative radial velocity of multiple targets at every
time step. Angular position is determined by applying
comparisons of amplitude and phase between adjacent
antenna beams (Troppman and Höger, 2008).
The received signals are processed in the radar
microcontroller and a complete target list is passed to the
CAN-bus of the radar. Maximum of 32 strongest reflections
are considered as targets. The following information is
presented in the target list from every target: Distance, lateral
distance from center line and its variance, relative radial
speed and relative radial acceleration. Also, time stamps of
each measurement is provided as well as some radar
diagnostics.
3.2 Doppler Radar
The Doppler radar used in the system is Kustom Signals HR12 (Kustom Signals 2009). It is speed control radar used by
police in some countries. The radar operates on K-band. The
exact beam width is not known, but it is very narrow, just a
few degrees. The intermediate Doppler frequency signal can
be acquired from the radar with a special connection. This
raw signal is sent to a computer for further signal processing.
4. ALGORITHM FOR DETECTING A WALKING
HUMAN FROM DOPPLER SPECTRUM
Detection of the periodic motion can be done by analyzing
the Fourier spectrogram of the Doppler frequency. Fig. 1
shows a typical spectrogram caused by a walking human. A
typical feature of this type Doppler signature is that the
bandwidth of the signal varies periodically. A straightforward
way to measure this bandwidth is to calculate the variance of
the signal. However, there may be other frequency
components caused by other moving objects, such as cars or
machines. Therefore, it is necessary to analyze small bands of
the frequency spectrum individually.
In this research, three slices of the spectrogram are analyzed.
The slices are marked to the spectrogram in Fig. 1. Six onedimensional signals are obtained by calculating the variance
and mean of the frequencies as a function of time. The
signals are shown in Fig. 2. The cadence frequency caused by
the periodic movement is clearly visible in these signals. The
frequency can be determined with spectral or cepstral
analysis, autocorrelation, or auto-difference function, that is,
a sum of squared differences of the signal with itself with
different offsets. In this research, the latter method was used,
because it was found to be the most reliable.
The auto-difference function Dn of the function Fn at time t
and shift s is calculated with
( ) () ( ) ∑ ( ) +
−=
= −− 2
2
2 ,
s w
sd w n n n dtFtFstD , (1)
where w is the width of the analyzation window. The
Euclidean sum of squares
() () ∑=
= 6
1
2 , , n
n stDstD (2)
is calculated to form one function D(t,s).
D(t,s) needs to be low to indicate strong correlation with the
shift s. To distinguish low values caused by strong correlation
from just low signal strength, the values of D need to be
normalized. A typical characteristics of a periodic signal is
that where the value D(t,s) is low, the value D(t,s/2)
representing the half shift is high. Therefore the
normalization can be done with
( ) ( )
( ) 2 ,
, , s tD
stD Norm stD =
. (3)
The cycle length of the periodic motion can now determined
by finding the local minima of the function DNorm(t,s) in s
direction. Because the pace of the walking is generally quite
slow, the function needs to be observed for at least two
seconds before a walking human can be detected reliably.
Fig. 3 shows the output of the DNorm(t,s) function.
Fig. 1. An excerpt of spectrogram containing measurement of
a walking human. The three numbered bands are analyzed
individually.
Fig. 2. Mean (blue curve) and standard deviation (red curve)
of frequencies as functions of time. The curves are
normalized to same scale.
Fig. 3. Function DNorm(t,s), with t as time and s as shift.
Darker values represent low values and of the function. The
yellow line indicates the local minima with low enough
values. This represents the detected periodicity in the signal.
5. TARGET TRACKING AND DATA FUSION
In this chapter a simple multitarget tracking method and data
fusion process used in the measurement system are presented.
Both, target tracking and data fusion, are research fields of
their own and are not addressed in this paper in more detail.
Interested reader should look up e.g. (Bar-Shalom and
Fortman 1998) and (Waltz and Llinas 1990).
5.1 Filtering the measurements of the automotive radar
Filtering out the clutter measurements is important in order to
reduce false detections. Human movements are modelled
with continuous Wiener process acceleration model (BarShalom et al. 2001). The process noise – changes in the
acceleration in this case – is modeled as continuous zeromean white noise. The measurement noise is also assumed to
be white. Since both dynamic model and measurement model
are linear, Kalman filtering (Kalman 1960) is an obvious
choice for the filtering problem.
Because there are possibly several targets in the field of view
at once, the data association problem needs to be solved. In
this case, a simple Nearest Neighbour (NN) method is used
(Blackman and Popoli 1999). This method associates the
nearest measurement within the validation region to the
target. If no target is associated to a measurement, a tentative
target is formed. If another measurement is associated to that
tentative target in the following time step, the target is
confirmed. Targets are deleted if there are no measurements
associated to them in several consecutive time steps. For the
validation region, the uncertainty ellipses of the
measurements and the targets are used. After the data
association is made, separate Kalman filters may be applied
to each target. The NN data association works fine since the
measurements are received with high enough frequency
(10 Hz) and they are not too close with one other. Therefore,
the usage of more complex data association algorithms is not
justified.
Each target is given an extra state – a human probability -
which is updated separately from the Kalman filter. The
human probability ranges from zero to one, one being that it
is absolutely certain that the target is human. If there is no
knowledge of the class of the target, a neutral value of 0.5 is
assigned to the human probability.
The velocity of each target is monitored and if it is close to
the average walking speed of a human, 1.35 m/s (Anderson
and Pandy 2001), the human probability is increased slightly.
If the velocity is not within the specified interval, the human
probability is decreased moderately. In this research, a
probability value 0.55 was used for targets moving with right
speed, and 0.45 for other targets. Probability value 0.9 was
used for Doppler radar detections. Targets with probabilities
over 0.9 were considered as humans. These values were
found experimentally.
These probability values are combined at every time step
with Bayes rule
))(1))(|(1()()|(
)()|( )|(
xpxypxpxyp
xpxyp yxp
−+ − = (4)
where y is the measurement and x is the state.
If a target travelling close to the average human walking
speed is tracked for a while, the human probability increases
every time step eventually enabling the classification of that
target as human.
5.2 Data fusion
Fusing data from both radars is a simple procedure, since the
data from the Doppler radar only provides additional
information about the class of the target. It is known that if
there is an observation with the Doppler radar, there is also a
target which is moving roughly at the average walking speed
of a human. Therefore, the target velocity is taken into
consideration at this point as well.
The Doppler measurements are modelled as Gaussian
distributions roughly covering the measurement sector of the
radar. An example of this kind of distribution can be seen in
the Fig. 4 marked with blue dashed line. Data association
between the Doppler measurement and a tracked target is
made if the distribution of the Doppler measurement, at
certain time step, intersects with the estimated position
distribution of that target. If the velocity estimate of such
target lies on the specified human walking speed interval, the
human probability of the Doppler measurement is combined
to the human probability of the target with Bayes rule. This
way, there can be stationary targets in the area of that
relatively large distribution of the Doppler measurement
which human probabilities will not get falsely updated.
The fusion process can be clarified with the help of Fig. 4.
The green ellipses in the figure represent the 2-sigma
uncertainty ellipses of the positions of three targets tracked at
that time. The red dots represent positions of targets
classified as humans and the small black circle inside the
biggest green ellipse represents a clutter measurement which
is rather uncertain. The Doppler measurement, illustrated
with blue dashed line, is associated to all of these three
targets. However, only the red targets are moving at
appropriate velocity and therefore only their human
probabilities are updated.
Fig. 4. Visualization of the fused data. Red dots represent
targets classified as humans.
A photograph of the situation illustrated in the Fig 4 can be
seen in the Fig 5. The left hand red dot in Fig 4 represents the
human up front in Fig 5 and the red dot on the right hand side
represents the human at the back. As can be seen in the Fig 5,
there is no target between these humans, hence making the
third measurement a false detection.
Fig. 5. A photograph of the data set gathered from harbour.
Two human targets are walking in the field of view of the
sensor system.
6. EXPERIMENTS
This chapter explains where and how the data sets used to
analyze the method presented in this paper are gathered.
Also, the performance of the method is presented here in a
way that the advantages of the use of the Doppler radar can
be seen.
Four data sets were collected, one inside a mine, one in a
harbor, one with simulated rain, and one where the
measurement platform was moving. The latter two caused
problems to the Doppler detection. In case of rain, the
numerous moving rain drops caused strong signal that
masked the reflection from a human. In case of moving radar,
the similar effect was caused by the environment that was
moving relative to the radar. Strong radar targets again
masked the signal of the human, and the detection could not
be performed. Therefore, only static cases without rain were
considered.
6.1 Measurements
The two data sets analysed here are collected from a mine
and from a harbor with same measurement platform. Radars
were attached to an ATV enabling easy mobility of the
sensors. In addition, a stereo camera was attached to the
measurement platform to produce image information from
the tests. The radar measurements were validated based on
the stereo camera data. The movements of the measurement
platform were recorded as well. Fig 6 shows a still picture
from the data set recorded in the mine. Similar image from
the other data set is presented in Fig 5.
Fig. 6. A photograph from the test sequence in the mine.
There is only one human target in the field of view. The walls
of the tunnel produce plenty of reflection back to the
automotive radar.
Both radars were attached to the same laptop computer,
automotive radar via USB-CAN adapter and the Doppler
radar via USB port attachable A/D converter. Since both
sensors were connected to the same computer, data was
automatically synchronized. The gathered data sets were
approximately 30 seconds in length.
In the data set from the mine, there was only one human
target walking in the field of view. The trajectory of that
human can be seen in Fig 7. The human started walking away
from the radar on the left hand side of the tunnel making a
turn back just after the 15 meter marker returning on the right
hand side of the tunnel.
The data set recorded at the harbor features two human
targets in the field of view at once. Trajectories of these
humans are illustrated in Fig 8. The other human walked
across the measurement sector once starting from the left in
the vicinity of the 7 meter marker ending up to the right on
the five meter marker. The other human started from the right
hand side on the 5 meter marker and walked away from field
of view on the left hand side just after the 20 meter marker.
The same human returned to the FOV on the left hand side
approximately on the 11 meter marker and walked out of the
measurement sector on the right around the 3 meter marker.
6.2 Results
Results of the measurement are summarized in Table 1. The
first row contains the combined number of true positive and
false negative observations. The second row contains the total
number of true positive observations. The third row contains
the number of observations made due to the clutter. The
fourth and fifth rows contain the total number observations
classified as humans with only the automotive radar and with
the combination of the automotive radar and the Doppler
radar respectively. The percentages are calculated with
respect to the combined value of true positive and false
negative observations. All the information in the Table 1 is
also included in the Fig. 7 and Fig. 8.
Fig. 7. The trajectory of the human target can be seen on the
figure. Black dots represent positions where the human was
observed but was not classified as human. The red dots are
observations which are classified as humans. The gray dots
represent positions where the human target was not observed
with the radar measurement system. Blue dots are radar
measurements originated from clutter or from some other
target.
Fig. 8. Trajectories of two humans in the harbour. The colour
codes follow the explanations in the Fig 7.
In the mine, there were a lot of reflections from the walls of
the tunnel. These measurements are not considered as clutter
since they are caused by real objects. Therefore these
measurements are excluded from the third line of the table.
However, the exclusion is somewhat subjective since there is
no exact method to tell which measurement originated from
the walls and which did not. In the harbor there was one
systematic false target which caused all the clutter
measurement.
The Doppler radar only contributes to the classification of the
targets. Hence, the advantages of the use of the Doppler radar
can be seen when comparing the fourth and the fifth line of
the table 1. In the mine, the effect of Doppler radar is that 20
more observations are classified as humans. At the harbor, 75
targets are classified as humans due to the Doppler radar. No
false human classifications were made in these data sets.
Table 1. Results
Mine Harbor
Targets present 89 95
Targets detected 80
(89.89 %)
79
(83.16 %)
Clutter targets 15
(16.85 %)
30
(31.58 %)
Targets classified
Bosch
29
(32.58 %)
0
(0 %)
Targets classified
Bosch + Doppler
49
(55.06 %)
75
(78.95 %)
7. CONCLUSIONS AND FUTURE WORK
The combination of automotive and Doppler radars is
promising. The inability of an automotive radar to distinguish
humans from other targets was compensated with Doppler
radar based detection. Since the beam width of the Doppler
radar was successfully approximated with Gaussian
distribution, the Kalman filter based multitarget tracking
method was suitable for the sensor fusion with minor
modifications.
Some aspects of the Doppler radar limit the applications of
the proposed system. Doppler radar cannot detect humans
who are stationary or walking across the radar’s field of view.
The radar can only detect the motion components that are
directed towards to or away from the radar. In addition,
Doppler radar is sensitive to rain and movement of the radar
itself. However, if the environment does not contain any
strong radar reflectors, the system can also be used when
moving. The relative speeds of walking humans change, but
because the proposed algorithm only observes the periodicity
of the motion, the walking can still be detected.
In presence of other moving targets, they can be falsely
classified as humans. This can be a result of walking-like
motion. In the test system, the measurement sectors were
very narrow for human detection application. A rotating
measurement system or multiple systems could be a solution.
Fundamentally FMCW and Doppler radars use the same
phenomenon, measuring the frequency shift between
transmitted and received signal. The automotive radar used
measures also the Doppler frequency. Therefore, if the raw
sensor data of the radar could be accessed, it might be
possible to implement the algorithms presented in this paper
using only the data from the automotive radar. The
combination of FMCW range and Doppler measurements
would also enable the system to distinguish Doppler
signatures caused by different people. The Doppler signatures
would be centered on the frequency respective to the distance
measured with FMCW principle.
ACKNOWLEDGEMENTS
This research has been carried out under funding from Tekes
(Finnish Funding Agency for Technology and Innovation),
and FIMA (Forum for Intelligent Machines ry).
REFERENCES
Anderson, F.C. and Pandy, M.G. (2001). Dynamic
optimization of human walking. Journal of
Biomechanical Engineering, 123:381-390.
Bar-Shalom, Y. and Fortman, T.E. (1988), Tracking and
Data Association, New York: Academic Press.
Bar-Shalom, Y., Li, X.R. and Kirubarajan, T. (2001).
Estimation with applications to tracking and navigation,
New York: John Wiley & Sons Inc.
Blackman S. and Popoli R. (1999). Design and Analysis of
Modern Tracking Systems, Norwood, MA: Artech
House, 1230 pp. ISBN- 978-1580530064.
Gürbüz, S.Z., Melvin, W.L. and Williams, D.B. (2007).
Detection and identification of human targets in radar
data. SPIE Defense and Security Symposium, Orlando,
FL, USA, April 2007
Kalman, R.E. (1960). A New approach to linear filtering and
prediction problem. Transaction of ASME, Journal of
Basic Engineering, 82. pp. 35–45.
Kustom Signals (2009).
http://www.kustomsignals.com/product_body2.asp?prod
uct_id=34&cat_id=7, Accessed 30.12.2009
Nakane, T. and Soshi, K. (1998). A study on recognition of
pedestrians using dual band radar. ITS World
Conference, Seoul, South Korea, October 1998.
Nakane, T. and Soshi, K. (2000). Animal body detection
system utilizing electromagnetic waves. US Patent
6072173, Available:
http://www.patentstorm.us/patents/6072173/description.
html
Otero, M. (2005). Application of a continuous wave radar for
human gait recognition. Technical paper, The MITRE
Corporation. Available:
http://www.mitrecorporation.com/work/tech_papers/tech
_papers_05/05_0330/05_0330.pdf
Thayparan, T., Abrol, S., Riseborough, E. (2004), MicroDoppler radar signatures for intelligent target
recognition. Technical paper, Defence R&D Canada.
Available:
http://www.ottawa.drdc-rddc.gc.ca/docs/e/TM-2004-
170-eng.pdf
Troppmann, R. and Höger, A. (2008). An introduction to
adabtive cruise control (ACC). Driver assistance
systems, Embedded design India. 4 pp.
Waltz, E. and Llinas, J. (1990). Multisensor data fusion,
Boston, MA: Artech house.
Yarovoy, A.G., Ligthart, L.P., Matuzas, J. and Levitas, B.
(2008). UWB radar for human being detection. IEEE
A&E Systems Magazine, volume (28), pp. 36-40.
ReferencesAnderson and Pandy, 2001
F.C. Anderson, M.G. PandyDynamic optimization of human walking
Journal of Biomechanical Engineering, 123 (2001), pp. 381-390
Y. Bar-Shalom, T.E. Fortman
Tracking and Data Association, Academic Press, New York (1988)
Y. Bar-Shalom, X.R. Li, T. Kirubarajan
Estimation with applications to tracking and navigation, John Wiley & Sons Inc, New York (2001)
S. Blackman, R. Popoli
Design and Analysis of Modern Tracking Systems, 978-1580530064, Artech House, Norwood, MA (1999), p. 1230
Google ScholarGürbüz et al., 2007Gürbüz, S.Z., Melvin, W.L. and Williams, D.B. (2007). Detection and identification of human targets in radar data. SPIE Defense and Security Symposium, Orlando, FL, USA, April 2007.
R.E. KalmanA New approach to linear filtering and prediction problem
Transaction of ASME, Journal of Basic Engineering, 82 (1960), pp. 35-45
Kustom Signals
Accessed 30.12.2009.
Nakane and Soshi, 1998Nakane, T. and Soshi, K. (1998). A study on recognition of pedestrians using dual band radar. ITS World Conference, Seoul, South Korea, October 1998.
T. Nakane, K. SoshiAnimal body detection system utilizing electromagnetic waves. US Patent 6072173
M. Otero
Application of a continuous wave radar for human gait recognition. Technical paper, The MITRE Corporation (2005)
T. Thayparan, S. Abrol, E. RiseboroughMicro-Doppler radar signatures for intelligent target recognition. Technical paper, Defence R&D Canada
Google ScholarTroppmann and Höger, 2008Troppmann, R. and Höger, A. (2008). An introduction to adabtive cruise control (ACC). Driver assistance systems, Embedded design India. 4 pp.
E. Waltz, J. Llinas
Multisensor data fusion, Artech house, Boston, MA (1990)
A.G. Yarovoy, L.P. Ligthart, J. Matuzas, B. LevitasUWB radar for human being detection
IEEE A&E Systems Magazine (28) (2008), pp. 36-40
