As I started to think about what my test matrix would look like, I quickly realized that I had far too many variables and that I needed to try to eliminate some of them.
The first parameter that I wanted to measure was detection acquisition time and to see how it varied based on function (body, face, etc), detected image size and stabilization.
I decided to try body detection first. I labeled the images 1-5 with 1 being the largest. Then I did 100 sequential detections for each image size with stabilization turned off and then on.
Here is the data from my first run:
HB1 Largest Size 100 Sequential Detections Stabilization Off
Run Time X Y Size Confidence 1 1077 864 600 768 849 2 1128 864 600 768 838 3 1067 864 600 768 825 4 1065 864 600 768 897 5 1078 864 600 768 730 6 1110 864 600 768 701 7 1049 864 633 768 717 8 1080 864 600 768 776 9 1124 864 600 768 829 10 1083 864 600 768 744 11 1108 864 600 768 759 12 1111 864 568 768 312 13 1108 864 600 768 815 14 1077 864 600 768 743 15 1116 879 632 864 847 16 1123 873 645 832 846 17 1031 864 600 768 845 18 1198 864 600 768 739 19 1062 864 600 768 632 20 1124 864 600 768 794 21 1046 864 600 768 815 22 1096 864 600 768 865 23 1107 864 600 768 686 24 1062 860 619 714 639 25 1093 864 600 768 719 26 1065 864 625 768 877 27 1128 864 600 768 758 28 1054 864 600 768 758 29 1120 864 600 768 872 30 1140 864 600 768 835 31 1109 864 600 768 735 32 1083 864 600 768 756 33 1061 864 568 768 384 34 1127 864 637 768 412 35 1102 864 600 768 773 36 1031 864 600 768 779 37 1112 864 568 768 324 38 1066 864 632 768 781 39 1121 872 617 768 787 40 1035 864 568 768 346 41 1111 864 600 768 801 42 1109 864 600 768 826 43 1049 864 600 768 774 44 1093 864 632 768 793 45 1085 864 568 768 321 46 1085 864 629 768 319 47 1077 864 600 768 874 48 1188 864 600 768 837 49 1111 864 624 768 854 50 1046 864 629 768 754 51 1094 864 600 768 787 52 1093 872 643 832 871 53 1068 864 600 768 758 54 1180 864 630 768 709 55 1187 864 568 768 314 56 1077 864 600 768 610 57 1114 864 600 768 688 58 1062 864 600 768 719 59 1125 864 568 768 307 60 1059 864 624 768 878 61 1092 864 600 768 792 62 1062 864 626 768 873 63 1063 864 628 768 748 64 1127 879 634 864 812 65 1046 864 600 768 774 66 1093 896 657 864 446 67 1148 880 634 864 793 68 1109 864 600 768 839 69 1088 864 568 768 353 70 1102 864 600 768 570 71 1046 864 630 768 743 72 1140 864 600 768 921 73 1093 864 600 768 835 74 1062 864 600 768 794 75 1062 864 600 768 890 76 1140 864 600 768 784 77 1109 875 627 768 718 78 1140 864 626 768 814 79 1030 864 600 768 765 80 1077 864 568 768 323 81 1093 864 600 768 627 82 1109 864 631 768 853 83 1047 864 600 768 868 84 1096 864 568 768 319 85 1109 864 600 768 773 86 1050 864 633 768 783 87 1109 864 632 768 678 88 1046 875 647 832 829 89 1078 864 631 768 763 90 1202 864 632 768 817 91 1062 864 600 768 671 92 1140 880 635 864 765 93 1031 907 641 864 326 94 1187 864 600 768 786 95 1079 864 600 768 809 96 1097 864 629 768 882 97 1066 864 600 768 873 98 1102 864 600 768 807 99 1062 864 629 768 749 100 1078 864 568 768 436
Data summary of Human Body Detection:
Time units are ms
Confidence scale is 0-1000
Stability Off | Stability On | |||||||||||
HB1 | Time | X | Y | Size | Confidence | Time | X | Y | Size | Confidence | ||
Average | 1094 | 866 | 607 | 775 | 719 | 1092 | 864 | 600 | 768 | 751 | ||
Min | 1030 | 860 | 568 | 714 | 307 | 1032 | 864 | 600 | 768 | 320 | ||
Max | 1202 | 907 | 657 | 864 | 921 | 1202 | 864 | 600 | 768 | 923 | ||
Stability Off | Stability On | |||||||||||
HB2 | Time | X | Y | Size | Confidence | Time | X | Y | Size | Confidence | ||
Average | 1098 | 872 | 582 | 439 | 849 | 1096 | 872 | 581 | 441 | 830 | ||
Min | 996 | 868 | 580 | 432 | 692 | 1004 | 872 | 581 | 441 | 731 | ||
Max | 1234 | 877 | 584 | 448 | 942 | 1274 | 872 | 581 | 441 | 948 | ||
Stability Off | Stability On | |||||||||||
HB3 | Time | X | Y | Size | Confidence | Time | X | Y | Size | Confidence | ||
Average | 1083 | 872 | 606 | 384 | 898 | 1067 | 871 | 608 | 384 | 873 | ||
Min | 985 | 871 | 592 | 384 | 798 | 980 | 871 | 608 | 384 | 700 | ||
Max | 1240 | 880 | 609 | 384 | 971 | 1180 | 871 | 608 | 384 | 947 | ||
Stability Off | Stability On | |||||||||||
HB4 | Time | X | Y | Size | Confidence | Time | X | Y | Size | Confidence | ||
Average | 1067 | 879 | 586 | 208 | 706 | 1079 | 877 | 585 | 208 | 702 | ||
Min | 983 | 875 | 583 | 204 | 457 | 997 | 877 | 585 | 208 | 455 | ||
Max | 1187 | 882 | 590 | 216 | 982 | 1187 | 877 | 585 | 208 | 987 | ||
Stability Off | Stability On | |||||||||||
HB5 | Time | X | Y | Size | Confidence | Time | X | Y | Size | Confidence | ||
Average | 1037 | 878 | 595 | 112 | 680 | 1045 | 877 | 593 | 112 | 699 | ||
Min | 971 | 875 | 588 | 96 | 457 | 975 | 877 | 593 | 112 | 451 | ||
Max | 1327 | 881 | 597 | 114 | 916 | 1140 | 877 | 593 | 112 | 937 |
The test results were somewhat surprising.
1) Acquisition time average does not vary significantly with the size of the body detected.
2) Acquisition time average does not vary significantly with stabilization enabled or not
3) Confidence average also does not vary significantly with either size or stabilization
4) The average human body detection rate is only 1 per second (need to test with multiple bodies detected in image)
5) I did not expect such a large variation (min-max) in the data using a static image
6) The effect of stabilization appears to take the jitter out of the detection window but not the algorithm result
I decided to generate histograms for the Detection time and Confidence data to look at the characteristics of the variations.
Here are Histograms of the largest image size (HB1) with stabilization off and on.
Stabilization off
Stabilization on
I'm not quite sure what to make of the lack of repeatabilty. It will be interesting to see if all the detection functions behave the same way. The variation that I was attributing to detecting "live" images may just be inherent in their detection algorithms.
Based on what I see, I am going to reduce my data collection. I will only run with Stabilization On and I will reduce the number of sequential detections from 100 to 25. And to test the acquisition times for the different functions, I will only use the largest and smallest image sizes.
Top Comments