DENEB SYSTEMSOpenCV tracker comparison: which object tracker is best?
Which tracker should follow a drone, a car or a bird? We ran OpenCV's built-in trackers (CSRT, KCF, MOSSE and MIL), OpenCV's NanoTrack neural tracker and the three Sky Tracker profiles on the same 242 labelled clips, each started from the same box, and timed every one on the same CPU core.
Six trackers, one bird
Every tracker starts from the same hand-labelled box at 0:11.6 of one of our field clips: a bird far off against moving clouds, filmed with a hand-held camera.
KCF gives up within a second and MOSSE within three. When the camera swings at 0:15, CSRT loses the bird and MIL jumps to empty sky while still reporting a box. Sky Tracker default and correlation stay on it for all ten seconds.
The view follows the target and the clock shows time in the clip. A crossed-out box is more than 40 px from the target. Times per frame are from the table below.
Results
| Tracker | Type | Public dataset 224 clips | Field footage 18 clips | ms / frame 1080p | ms / frame 640×512 |
|---|---|---|---|---|---|
| Sky Tracker correlation + ML | Correlation + NanoTrack | 90% | 65% | 12 | 10 |
| NanoTrack | Neural (Siamese), OpenCV | 90% | 66% | 9.9 | 8.7 |
| CSRT | Correlation filter, OpenCV | 87% | 48% | 20 | 18 |
| Sky Tracker correlation | Correlation filter | 85% | 60% | 2.8 | 1.6 |
| MIL | Online learning, OpenCV | 84% | 54% | 27 | 23 |
| Sky Tracker default | Template tracker | 77% | 69% | 1.4 | 0.51 |
| KCF | Correlation filter, OpenCV | 46% | 18% | 2.1 | 1.3 |
| MOSSE | Correlation filter, OpenCV | 21% | 30% | 0.20 | 0.07 |
Accuracy is the share of labelled frames where the tracker's centre is within 20 px of the label; a frame the tracker reports as lost counts as a miss. Time is the tracker's own update call per frame on one core of an Apple M3 Pro, without video decoding. Lower is faster.
What the numbers say
Most accurate on the public dataset
Sky Tracker correlation with ML (90%) and OpenCV NanoTrack (90%) are level, ahead of CSRT (87%). Both run the same 1.8 MB NanoTrack network on the CPU; Sky Tracker adds its correlation tracker and lets each check the other.
Best on fast field footage
On our own 1080p and 720p clips, where targets move faster and change size, Sky Tracker default holds the target best (69%). CSRT drops from 87% to 48% and MIL from 84% to 54%.
Knows when it has lost the target
NanoTrack and MIL always report a box, even when it is on the background: on field footage 29% of NanoTrack’s frames are more than 40 px off. Sky Tracker default reports the target lost instead, and is that far off on 17% of the frames it claims.
Fast on one CPU core
Sky Tracker default updates in 1.4 ms per 1080p frame and correlation in 2.8 ms, against 20 ms for CSRT and 27 ms for MIL. Correlation with ML takes 12 ms when NanoTrack checks every frame, close to NanoTrack alone (9.9 ms); checking every third frame (1.1’s --ml-every 3) halves that, at the cost of a less steady box on fast footage.
Tightest box: CSRT
CSRT fits the box size most closely (mean overlap 0.63 on the public set, against 0.46 for correlation with ML). Sky Tracker is tuned to keep the centre on the target, which is what a gimbal or a pan-tilt head steers by.
KCF and MOSSE drop small targets
With OpenCV’s default settings, KCF and MOSSE give up on most small, low-contrast targets: they claim to be tracking on only 47% and 24% of the public frames.
Which tracker should you use?
- Any target, best accuracy: Sky Tracker
correlationwith ML (how it works). - Small, fast targets in the sky: Sky Tracker
default, the most accurate on fast field footage and among the fastest here (drone tracking). - Every millisecond counts: Sky Tracker
correlation, the same tracker without the neural network. - A tight box matters more than speed: CSRT.
- Free and open source: NanoTrack, if OpenCV's dnn module fits your build and you can handle a tracker that never reports the target lost.
How we measured
- Public dataset: 224 clips sampled with fixed seeds from the Svanström drone-detection dataset: visible 640×512 and infrared 320×256 video of drones, aircraft, helicopters and birds, 56 clips of each, about 70,000 frames labelled every frame.
- Field footage: 18 of our own 1080p and 720p clips of drones, RC planes, light aircraft, an airliner, birds and a rally car, 238 labelled frames.
- Every tracker starts from the first labelled box and runs to the end of the clip without a restart.
- Accuracy is the share of labelled frames where the reported centre is within 20 px of the label, averaged over clips. A frame the tracker reports as lost counts as a miss.
- OpenCV 4.14 trackers with their default parameters. NanoTrack uses the same v2 models that the correlation profile’s ML option ships with.
- Time is the tracker’s own update call on one thread (no video decoding), on an Apple M3 Pro, over the first 300 frames of four 1080p clips and eight 640×512 clips, one process at a time.
Dataset: F. Svanström, F. Alonso-Fernandez and C. Englund,Drone-detection-dataset. More Sky Tracker numbers, including frame rates per profile, are in thepublished benchmarks.
Questions
What is the best OpenCV tracker?
On these 242 clips NanoTrack is the most accurate of OpenCV’s own trackers (90% on the public set, 66% on field footage), followed by CSRT on the public set. NanoTrack needs OpenCV 4.7 or later with the dnn module and two small ONNX models; CSRT, KCF and MOSSE need the opencv-contrib tracking module.
CSRT or KCF?
CSRT is far more accurate (87% against 46% on the public set) and fits the box best, but it is about ten times slower (20 ms against 2.1 ms per 1080p frame). With default settings KCF loses most small targets.
Why does MOSSE score so low?
OpenCV’s MOSSE reports the target lost whenever its peak-to-sidelobe ratio falls below 8. That happens on most of the small, low-contrast targets in these clips, so it stops tracking early.
Do any of these trackers need a GPU?
No. Every number on this page comes from one CPU core. NanoTrack and Sky Tracker correlation with ML run a small neural network, also on the CPU.
Will the numbers hold on a Raspberry Pi or a Jetson?
Accuracy does not depend on the hardware; speed does. ARM boards are slower than a desktop core and we have not timed them for 1.0.0, so measure on your device. The free Sky Tracker demo has a Linux ARM64 build.
Can I check this on my own video?
Yes. The public dataset is free to download, and the free Sky Tracker demo runs all three profiles on ten seconds of any video you open.
Try it on your own footage
The free desktop demo tracks ten seconds of any video with all three profiles, on macOS, Windows and Linux.