Actions
Forum Thread Details
  • Replies 19 replies
  • Subscribers 2757 subscribers
  • Views 2929 views
  • Users 0 members are here
Related

RoadTest Reviews: A Discussion About The Rating System

rscasny

For any RoadTest Review, the reviewer not only provides his or her personal take on the product being reviewed but also a numbered ranking (10-point scale) for 6 different questions.

 

Here is a random example:

 

image

My gut feeling is that  I haven't clarified well enough what the 10-point scale means. As a result, there may be a lack of certainty when giving a grade. In addition, a lot of RoadTesters grade these questions in similar ways. So, I'd like to revisit our grading system. To start here's my idea:

 

  • 10 points: Outstanding
  • 9 points: Very Good Satisfaction
  • 8 points: Good Satisfaction
  • 7 points: Adequate but had to work through some programs
  • 6 points: Needs Work
  • 5 points  Barely Satisfactory
  • 4 points: Below Average
  • 3 points: Unsatisfactory
  • 2 Points:Totally Unsatisfactory
  • 1 point: Time to Rethink

 

I'd like to get the RoadTester Group's opinions on the rating system. Do you think it needs more clarification? Keep it the way it is?

 

Go ahead and suggest your own. Perhaps the problem is that we need to ask different questions?

 

Sincerely,

 

Randall Scasny

RoadTest Program Manager

Parents

  • The point system proposed surely is a great improvement, as it establishes a clearly defined rating criteria, making it a reference for all.

    The problem I see with it is that most people are accustomed either to "academic-based" scoring systems, like the ones where 10 is excellence, 5 or 6 is sufficient, and anything less is unsatisfactory, or to  "expectation-based" rating systems, where 10 is given when the product/service meets the expectation and anything less than 10 denotes some negative, and if you define a different rating system, unfortunately people tend to "fall-back" to the system they are most used to, creating some ambiguity in the valutation.

    For a 5 star rating system, despite the way the star criteria are set, the most common interpretation is as a "meeting expectation" feedback system, where 5 stars = expectation fully met (NB: not necessarily meaning "excellence"), and anything less than 5 star usually has some negative element in it. Expectation are normally built by gathering information beforehand (i.e. reading the marketing and technical documentation in our case).

     

    Personally, I too would go for a 5 star rating system, something like:

     

    5 - Expectation fully met

    4 - Most important expectation met

    3 - Basic expectation met

    2 - Basic expectation NOT met

    1 - No expectation met

    0 - Not rated (impossible to assess)

     

    Besides being visual and very simple, this kind of rating is also very common, therefore most people are already accustomed to it.

    Fabio.

Reply

  • The point system proposed surely is a great improvement, as it establishes a clearly defined rating criteria, making it a reference for all.

    The problem I see with it is that most people are accustomed either to "academic-based" scoring systems, like the ones where 10 is excellence, 5 or 6 is sufficient, and anything less is unsatisfactory, or to  "expectation-based" rating systems, where 10 is given when the product/service meets the expectation and anything less than 10 denotes some negative, and if you define a different rating system, unfortunately people tend to "fall-back" to the system they are most used to, creating some ambiguity in the valutation.

    For a 5 star rating system, despite the way the star criteria are set, the most common interpretation is as a "meeting expectation" feedback system, where 5 stars = expectation fully met (NB: not necessarily meaning "excellence"), and anything less than 5 star usually has some negative element in it. Expectation are normally built by gathering information beforehand (i.e. reading the marketing and technical documentation in our case).

     

    Personally, I too would go for a 5 star rating system, something like:

     

    5 - Expectation fully met

    4 - Most important expectation met

    3 - Basic expectation met

    2 - Basic expectation NOT met

    1 - No expectation met

    0 - Not rated (impossible to assess)

     

    Besides being visual and very simple, this kind of rating is also very common, therefore most people are already accustomed to it.

    Fabio.

Children
  • in reply to gecoz

    I have to disagree with you here Fabio, one of the concepts of modern quality assurance is the idea of "Surprise and Delight" - the feeling the customer has when the product does something good that they did not expect. The scoring system should allow for this possibility.

     

    For example, in the 70s when I was involved with a HiFi company it was not usual to provide connectors with an amplifier. We packed  power leads for the switched mains outlets, speaker connectors and some phono plugs, the idea being that it didn't cost much but really scored  a hit with every customer who would otherwise have needed a ttrip to the shops before they could try the new toy.

     

    I think this should have earned an extra star !

     

    (Recently the "Surprise and Delight" concept has got  a bit distorted and morphed somewhat into promotional activity - but the original concept is still valid.)

     

    MK