Omron HVC-P2 Roadtest - Test Results Part 1

As I started to think about what my test matrix would look like, I quickly realized that I had far too many variables and that I needed to try to eliminate some of them.

The first parameter that I wanted to measure was detection acquisition time and to see how it varied based on function (body, face, etc), detected image size and stabilization.

I decided to try body detection first.  I labeled the images 1-5 with 1 being the largest.  Then I did 100 sequential detections for each image size with stabilization turned off and then on.

Here is the data from my first run:

HB1  Largest Size  100 Sequential Detections Stabilization Off

Run    Time     X      Y    Size  Confidence
  1    1077    864    600    768    849
  2    1128    864    600    768    838
  3    1067    864    600    768    825
  4    1065    864    600    768    897
  5    1078    864    600    768    730
  6    1110    864    600    768    701
  7    1049    864    633    768    717
  8    1080    864    600    768    776
  9    1124    864    600    768    829
 10    1083    864    600    768    744
 11    1108    864    600    768    759
 12    1111    864    568    768    312
 13    1108    864    600    768    815
 14    1077    864    600    768    743
 15    1116    879    632    864    847
 16    1123    873    645    832    846
 17    1031    864    600    768    845
 18    1198    864    600    768    739
 19    1062    864    600    768    632
 20    1124    864    600    768    794
 21    1046    864    600    768    815
 22    1096    864    600    768    865
 23    1107    864    600    768    686
 24    1062    860    619    714    639
 25    1093    864    600    768    719
 26    1065    864    625    768    877
 27    1128    864    600    768    758
 28    1054    864    600    768    758
 29    1120    864    600    768    872
 30    1140    864    600    768    835
 31    1109    864    600    768    735
 32    1083    864    600    768    756
 33    1061    864    568    768    384
 34    1127    864    637    768    412
 35    1102    864    600    768    773
 36    1031    864    600    768    779
 37    1112    864    568    768    324
 38    1066    864    632    768    781
 39    1121    872    617    768    787
 40    1035    864    568    768    346
 41    1111    864    600    768    801
 42    1109    864    600    768    826
 43    1049    864    600    768    774
 44    1093    864    632    768    793
 45    1085    864    568    768    321
 46    1085    864    629    768    319
 47    1077    864    600    768    874
 48    1188    864    600    768    837
 49    1111    864    624    768    854
 50    1046    864    629    768    754
 51    1094    864    600    768    787
 52    1093    872    643    832    871
 53    1068    864    600    768    758
 54    1180    864    630    768    709
 55    1187    864    568    768    314
 56    1077    864    600    768    610
 57    1114    864    600    768    688
 58    1062    864    600    768    719
 59    1125    864    568    768    307
 60    1059    864    624    768    878
 61    1092    864    600    768    792
 62    1062    864    626    768    873
 63    1063    864    628    768    748
 64    1127    879    634    864    812
 65    1046    864    600    768    774
 66    1093    896    657    864    446
 67    1148    880    634    864    793
 68    1109    864    600    768    839
 69    1088    864    568    768    353
 70    1102    864    600    768    570
 71    1046    864    630    768    743
 72    1140    864    600    768    921
 73    1093    864    600    768    835
 74    1062    864    600    768    794
 75    1062    864    600    768    890
 76    1140    864    600    768    784
 77    1109    875    627    768    718
 78    1140    864    626    768    814
 79    1030    864    600    768    765
 80    1077    864    568    768    323
 81    1093    864    600    768    627
 82    1109    864    631    768    853
 83    1047    864    600    768    868
 84    1096    864    568    768    319
 85    1109    864    600    768    773
 86    1050    864    633    768    783
 87    1109    864    632    768    678
 88    1046    875    647    832    829
 89    1078    864    631    768    763
 90    1202    864    632    768    817
 91    1062    864    600    768    671
 92    1140    880    635    864    765
 93    1031    907    641    864    326
 94    1187    864    600    768    786
 95    1079    864    600    768    809
 96    1097    864    629    768    882
 97    1066    864    600    768    873
 98    1102    864    600    768    807
 99    1062    864    629    768    749
100    1078    864    568    768    436

Data summary of Human Body Detection:

Time units are ms

Confidence scale is 0-1000

The test results were somewhat surprising.

1)  Acquisition time average does not vary significantly with the size of the body detected.

2)  Acquisition time average does not vary significantly with stabilization enabled or not

3)  Confidence average also does not vary significantly with either size or stabilization

4)  The average human body detection rate is only 1 per second (need to test with multiple bodies detected in image)

5)  I did not expect such a large variation (min-max) in the data using a static image

6)  The effect of stabilization appears to take the jitter out of the detection window but not the algorithm result

I decided to generate histograms for the Detection time and Confidence data to look at the characteristics of the variations.

Here are Histograms of the largest image size (HB1) with stabilization off and on.

Stabilization off

Stabilization on

I'm not quite sure what to make of the lack of repeatabilty.  It will be interesting to see if all the detection functions behave the same way.  The variation that I was attributing to detecting "live" images may just be inherent in their detection algorithms.

Based on what I see, I am going to reduce my data collection.  I will only run with Stabilization On and I will reduce the number of sequential detections from 100 to 25.  And to test the acquisition times for the different functions, I will only use the largest and smallest image sizes.