Roadtest Scoring

While I agree that it may not be universally understood, I do think to an extent ranking systems and numbers are a sublanguage that are widely understood. I watched an interesting video and read the accompanying article discussing how the keypad on your phone came to be. A lot of us when asked now how numbers should be arranged on a phone would all (read most) tell you how it is now. Bell Labs did a study asking the general public how they thought these keys should be laid out (before anyone had seen a keypad or really owned a calculator or computer) and I believe something like 55% chose the way the phone is laid out now. While yes this isn't all or most, it is a majority, and shows that people share similar ideas and thoughts when it comes to numbers. But to your point, I do think that sometimes context is needed, but once a broad context is laid out, it then goes back to the majority of people thinking the same. If you see a questionnaire with a scale of 1-10 (1=bad, 10=great), I think most people would see 5 as neutral, not 7 or 8, so in a sense it is and is not a sublanguage understood by all.

Lets start with a proposal.  Is 1 the highest ranking and 10 the lowest?  Or is 10 the highest and 1 the lowest?  Most questionnaires that have a rating system like this say explicitly which is which.  That is because people come to the question from different backgrounds, cultures, education systems, etc.  To one person, 1 should be the highest, but to another 10 is surely the highest rating.  So, to say that a rating system which provides no advance explanation is understood universally is not true. 

This could be as simple as adding the guide to the drop-down boxes (at least for the 10, 5 and 1 scores). But what I would really like is a "does not apply" setting - some of the rating don't apply at all. E.g. when there is no demo material needed I never know what I should score this...

I'd like to know why you think that.

When we see a rating for anything, we first judge the person who applied the rating (which is very easy here because they're accompanied by a review) and then we get a feel of what that rating might mean.

The fact that the numbers get translated into something meaningful is very language like. If they'd stay just numbers in our minds, then yeah, it wouldn't be.

In contrast, the artificial scale is little more than a category system, it's just that the categories have a shortcut to name them by (the number).

Also, I agree with what dougw said and I think most people rate products like that, but you can never be sure without having an idea about the reviewer.

In my opinion it's easier to figure out what the reviewer meant when he applied a rating and subsequently, how much that rating is worth for you, than to figure out how wrong he is about certain assessments that lead him to place the product in a particular category.

Razvan

Thank you for all of the interesting and helpful comments.

I do think we need to add a guide to scoring (e.g.: 10 = Completely Agree, 8 = Mostly Agree, 1 = Completely Disagree, etc.) that will work for *all* questions.  We have been in discussions regarding that issue, the criteria themselves, and clarifying verbiage. 

Thanks again, we really do love the feedback.

Best,

Summer

Mark,

Good discussion.

I had been working recently  on the wording of the RoadTest Review form with one of our developers so the topic is on our minds.

IMO Nothing is perfect in life; entropy tends to rule the day. But there is pretty darn good and for me that's a 10.

But clearly, I'm hearing that perhaps I need to offer some guidance on the definition of a 10 or 5, etc.

I place most emphasis on what is said and the data to support conclusions. Rating systems aren't always the best indicators of a product; but we all take them into account whether it is buying a dev tool or choosing a restaurant.

I have no problem with low scores, if indeed they are the roadtesters honest opinion. But if a RoadTester were to give a 1 or 2, I would want to know why.

I also think the price-to-performance question is too subjective for my taste, but others feel it is an important question to give a supplier a feedback on their pricing. So, it can go both ways.

I'll probably reflect on this discussion a bit and maybe address it in my next roadtest conversation.

Randall

Due to the subjective nature it is probably more statistically relevant to use a 1 to 5 scoring system (star rating style). Scoring should also be qualified by the roadtester by explaining their expectation if it is not clear by what's being evaluated. Thus if product meets expectation they give a score of 3. If it is above or well above expectation then 4 or 5. Then there will be no average scores etc. and no concerns about whether 8 really means a 9 or a 7 etc. Also you can then use a 0 as a special case for a complete fail. However, I too agree with other comments about "what does exceeding expectation" really mean in terms of ranking. In these cases I would suggest providing commentary in the write up that states that a 5 means "product achieves spec", when this is relevant.

Rating systems are almost like a sub-language that everyone understands.

I don't think that's true at all.

I am of the same opinion as Douglas on this. If it behaves as expected and documented then it scores a 10.

I too cannot seem to quantify performance beyond expectation.

I guess that if you had thoroughly researched the product and it claimed it could for example send a radio signal a maximum of200 mtrs and you found in your application it actually did 1Km then that would be grounds to give it a real thumbs up IF you had implemented it the way it was intended to be used.

You MAY have for example added an antenna with additional gain that would explain the difference. This should have been noted when you do you road test in the subjective area of the test being something along the lines of - Hey guess what happens when you put a high gain antenna on brand xyz radio module - you can get 1Km range.

I think from memory I scored a 9 from 10 on one of the aspects of one of my road tests. I cannot recall why now. But I do remember it was because I thought the product SHOULD have performed better and did not. It may have been battery life for example.

Cheers

Warren.

Interesting topic. The comments show there are definitely many opinions on how to score products, and the results may be misleading if the reader has a different opinion from the reviewer.

Full disclosure - I'm one of those who would likely score products with all 10s.

Before I ever apply for a road test, I do a lot of reading/investigation/studying of the product literature and I most likely wouldn't apply if I didn't already like what I find out.

When scoring I try to score against product claims, not some hypothetical perfect product that is not defined and the manufacturer was not claiming to achieve.

If the product does what it claims to do or what I expect it to do, I score it a 10.

I don't see how we can expect a product to do more than what it claims. If it did, it would imply the documentation was inaccurate.

I think it is the only objective way to score a product - there is no objective scale that can be defined for how much a product exceeds expectations. (flabbergasted!, truly gobsmacked! - I like colourful language, even in a review, but these terms aren't well defined)

If I am surprised with some performance aspect, positive or negative, it often means I didn't do enough homework to understand the product.

I have purchased products in the past that have not lived up to their claims, but the products I've road tested have all been great.

Beyond objective scoring there is an opinion side to a review which is more subjective. I use the review body to provide some insight into my opinions.

When I do a road test, I always show real applications and I try to show how well the product works and how easy it is to use as well as its feature set.

I'm not sure this clearly explains my thinking, but it is my 2 cents anyway ...