RoadTest Reviews: A Discussion About The Rating System

The point system proposed surely is a great improvement, as it establishes a clearly defined rating criteria, making it a reference for all.

The problem I see with it is that most people are accustomed either to "academic-based" scoring systems, like the ones where 10 is excellence, 5 or 6 is sufficient, and anything less is unsatisfactory, or to  "expectation-based" rating systems, where 10 is given when the product/service meets the expectation and anything less than 10 denotes some negative, and if you define a different rating system, unfortunately people tend to "fall-back" to the system they are most used to, creating some ambiguity in the valutation.

For a 5 star rating system, despite the way the star criteria are set, the most common interpretation is as a "meeting expectation" feedback system, where 5 stars = expectation fully met (NB: not necessarily meaning "excellence"), and anything less than 5 star usually has some negative element in it. Expectation are normally built by gathering information beforehand (i.e. reading the marketing and technical documentation in our case).

Personally, I too would go for a 5 star rating system, something like:

5 - Expectation fully met

4 - Most important expectation met

3 - Basic expectation met

2 - Basic expectation NOT met

1 - No expectation met

0 - Not rated (impossible to assess)

Besides being visual and very simple, this kind of rating is also very common, therefore most people are already accustomed to it.

Fabio.

I have to disagree with you here Fabio, one of the concepts of modern quality assurance is the idea of "Surprise and Delight" - the feeling the customer has when the product does something good that they did not expect. The scoring system should allow for this possibility.

For example, in the 70s when I was involved with a HiFi company it was not usual to provide connectors with an amplifier. We packed  power leads for the switched mains outlets, speaker connectors and some phono plugs, the idea being that it didn't cost much but really scored  a hit with every customer who would otherwise have needed a ttrip to the shops before they could try the new toy.

I think this should have earned an extra star !

(Recently the "Surprise and Delight" concept has got  a bit distorted and morphed somewhat into promotional activity - but the original concept is still valid.)

MK

I agree, "Exceeds expectations" should be an possibility.

But I would want the middle option to be a "it's ok", "can be used but does not fill me with joy"

Hi Michael,

Perhaps I didn't explain myself well. Although I can see where you are coming from, and in principle I agree with you, my point was about what is in place now and how it is used. Unfortunately when you look around, just about all the star rating systems use the top rating for meeting expectation, and any less of a 5 rating is interpreted as somehow negative. Element14 could do differently, and use the "exceed" expectation for the 5 star, but by doing so it would generate ambiguity in the valutation, as a product meeting all the expectation would have to score only a 4 star.

I suppose what I'm trying to say is when choosing a rating system, I think we should take into account how people are using other feedback systems out there, so that the system is more familiar and the valutation is less susceptible of interpretation.

Fabio.

I agree that being different from other systems could be  a problem - perhaps we shouldn't use stars but some other symbol ..........

It is very tricky - in the end one has to read the text to get a real understanding of what the reviewer intends.

MK

Perhaps

ΩΩΩ

or

Perhaps

ΩΩΩ

or