RoadTest Reviews: A Discussion About The Rating System

I agree, "Exceeds expectations" should be an possibility.

But I would want the middle option to be a "it's ok", "can be used but does not fill me with joy"

Hi Michael,

Perhaps I didn't explain myself well. Although I can see where you are coming from, and in principle I agree with you, my point was about what is in place now and how it is used. Unfortunately when you look around, just about all the star rating systems use the top rating for meeting expectation, and any less of a 5 rating is interpreted as somehow negative. Element14 could do differently, and use the "exceed" expectation for the 5 star, but by doing so it would generate ambiguity in the valutation, as a product meeting all the expectation would have to score only a 4 star.

I suppose what I'm trying to say is when choosing a rating system, I think we should take into account how people are using other feedback systems out there, so that the system is more familiar and the valutation is less susceptible of interpretation.

Fabio.

I agree that being different from other systems could be  a problem - perhaps we shouldn't use stars but some other symbol ..........

It is very tricky - in the end one has to read the text to get a real understanding of what the reviewer intends.

MK

Perhaps

ΩΩΩ

or

I too consider the top score as more of 'delights the customer' rather than 'no room for improvement', because technology always evolves.

There are cultural differences too, for example I sometimes see the 5-point scoring results for presentations, and invariably the score is higher presented in the US, whereas it is always slightly lower in Europe. Or there is the one mean person who scores 1/5 for everything because they attended the wrong presentation for them, and the others scored 5/5 : ) So the 5 points or stars means something different depending on the audience. With the product reviews some are scored extremely low but then it becomes apparent from the text that the user didn't apply the product to the same use-cases as expected, or had some difficulties which are unique to him/her and would not generally apply to that product, e.g. faulty item.

I tend to agree with the comments that having an "exceed" or "surprise and delight" option is a good idea as that gives balance (i.e. upside and just downside).

However, I would suggest maybe rephrasing some of the questions to make it clearer or have an intro explaining how scoring should be carried out.

So for example "Product performed to expectations" should be rephrased to something like "Degree to which product exceeded or failed to perform to expectations". The "Degree to which..." is now the key focus in the scoring and this part is now common to all questions and could be used as header, or something similar.

Hi Randall,

When we evaluated proposals we used a simple color code.

Red = Did not meet requirements.

Yellow = Met part of the requirements.

Green = Met requirements.

Blue = Exceeded requirements.

This system made it easy to summarize everyone's review and come up with a consensus rating for each item.

Only after this process was the cost evaluated as only the green proposals were considered.

DAB

Sometimes documentation is provided but at the quality they might as well have left it out as the feeling you get from such bad documentation is worse than no documentation at all..

Kas

using the system I propose:

lacking communication is not counted while bad communication has a negative influence on the total score.