Last week our team of judges had several meeting, in which we discussed the results of the evaluation and tried to put fair scores to the criteria. These criteria had only loosely been defined in the challenge, and our own challenge was now to ensure that we did not introduce new criteria which the challenge participants had not been aware before. It was, however, difficult to find a common baseline according to which we would evaluate.
We had collected the measurements, and the data were now available. But the question was then how to score a device which did measure more parameters than others. A device which would monitor fewer parameters would naturally also have fewer measurements, and therefore might get a lower score if the number of parameters would be one criterion. On the other hand, if we would calculate the average quality of the measurements that one device would produce, then a device with many parameters would be at a disadvantage, in case some of the measurements were not as good as others, and therefore the developer of such a complex apparatus would be put at a disadvantage. These were the kinds of discussions we had, and we ultimately came up with scoring which we believe is fair from various aspects.
Overall we did settle on these criteria with the following weighting:
- Ambition and Sophistication 20%
- Initial setup, installation, starting and operation. 15%
- Usability 25%
- Validation of Measurements 25%
- Cost 15%
This allowed a wide range of score capturing, and therefore each device did have a good chance of becoming the winner.
We are preparing a report in which we will in detail describe our thoughts and the results of the evaluation. And it is noteworthy that our judges team, while discussing and voicing several opinions, in the end came to full agreement with each of the scores and sub-scores, as we have included all the possible aspects which each judge did voice.
We do not yet announce a winner here, but I can state that we have determined a winner, based on the evaluation. I will forward our report to the Element 14 / Farnell Challenge organisers, who will then decide how to announce the winner.
Cheers,
Reinhold
Top Comments