Roadtest Scoring

I think the scale could use some definition.  That way everyone is on the same page about what the score really means. 

Hi Mark,

I do agree with you that the scoring scheme for the 1-10 ratings should be made clear and this would give more meaningful road test results. I've seen far too many ranked 60/60 for my linking!

With regards to DAB's comment on your low score, I think he was trying to elicit some more information out of you as you gave a below average score (which is absolutely fine) but there is nothing in your write up which says what the issues you had with the quality of the software were, so how is the manufacturer meant to rectify these issues? How are others going to know if they are going to be affected by the same issues if they want to use the platform? The point of the road test is that you evaluate the product and give an honest and fair review with a reasonable amount of detail such that others can find benefit in the review.

Best Regards,

Rachael

Great idea.

A definition (showing 5 as average) would be really useful.

Unfortunately this scoring is a one size fits all, which may not be the best method.

We do tend to see mature products (not always) so I'd expect the Demo Software and Demo easy to use to be up there in the score.

Price to performance is a very subjective one.

Some means of comparing it against other similar products might be more meaningful.

Mark

Hi Mark,

Interesting observation. I guess it's down to personal expectations. For engineering products I would expect them to function to their written spec, so on that alone, the minimum they should get is a 5/10 if you consider 5 to mean an average performing product as you say.

If I saw a 4 (I don't know which review you're referring to) then that would ring alarm bells for me, because to me it would suggest that the product doesn't work or meet the spec.

Many products have nice features beyond what would be expected, so that pushes the score beyond 5 for me.

Anyway, everyone is different, there are all sorts of studies on how people rate and rank things.

If you've made an effort to justify your product conclusions then I think people will read through that too.

So what would be more concerning is if the score and the justification are inconsistent or the justification is absent.  From reading that, I can calibrate myself briefly to the scoring scale.

If I'm looking to buy a product I will definitely look beyond the score. It is nice to have a number, but the actual explanation/justification is more important to me. But that's just me, for others maybe the score is more important, e.g. if they are trying to rapidly decide out of dozens of products which ones they should focus their attention on. I do that with IMDB for instance, i.e. generally ignoring any movie that scores less than 7. But for engineering products it is different.

Also, I think a 10 is acceptable, it isn't (to me) suggesting the product has no way to ever improve, it is suggesting that the product delights the customer in that area, e.g. documentation, demo software, etc.

Everyone is different, and it would be quite restrictive if we were forced to never score a product with a 10 because everything can be improved and optimized over time. I think better to use personal judgement in this area. Some things deserve a 10, some things deserve 9, but it is a personal scale. The text justification is the supporting material for whatever score you give.

I agree to your scoring considerations as well as the detsils of shabaz

As born and living in europe the scoring you mention with 5 average for me it is normal and I always need t9 make a lot of calculations when I try to understand the US school ranking method that I am no used to

Enrico

Hi Mark,

I meant no rebuke.  I was just trying to get you to expand on your score.

The value of a road test is to provide the community and the vendor with useful feedback.

Numbers are just that.  What we want is the data and analysis that led to the number.

Hence my comments.  I do not dispute your assessment.  I just want to make sure that I understand the context in which you made that assessment.

I agree that a better definition for the scoring is needed.  1 to 10 can mean anything.  So more elaboration is required.

Thank you for taking the time to bring up this important issue.

DAB

None taken, just got me thinking. Lot of 5 star road tests.

Hi Mark,

I think it's important to keep in mind that not everybody reading these reviews will be familiar with the custom scoring scheme. That means that rating products based on your personal feelings is actually the preferred method, because your personal scale is likely closer to my personal scale than an artificial/forced scale would be.

Rating systems are almost like a sub-language that everyone understands. The reason we understand them so well is because we know ourselves and we know other people, so we have an intuitive understanding of what a particular score means, especially if it's surrounded by a lengthy review.

You can obviously take that away and replace it with an artificial scale, but the scale will have to be displayed at all times when a score is displayed, otherwise it will confuse the reader and even if he knew it was an artificial scale, it wouldn't make sense unless he compared the meaning of the current score with the meanings of the other possible scores, so he could understand where the product stands.

I guess it can be pulled off, but would it really be an improvement?

Razvan

Interesting topic. The comments show there are definitely many opinions on how to score products, and the results may be misleading if the reader has a different opinion from the reviewer.

Full disclosure - I'm one of those who would likely score products with all 10s.

Before I ever apply for a road test, I do a lot of reading/investigation/studying of the product literature and I most likely wouldn't apply if I didn't already like what I find out.

When scoring I try to score against product claims, not some hypothetical perfect product that is not defined and the manufacturer was not claiming to achieve.

If the product does what it claims to do or what I expect it to do, I score it a 10.

I don't see how we can expect a product to do more than what it claims. If it did, it would imply the documentation was inaccurate.

I think it is the only objective way to score a product - there is no objective scale that can be defined for how much a product exceeds expectations. (flabbergasted!, truly gobsmacked! - I like colourful language, even in a review, but these terms aren't well defined)

If I am surprised with some performance aspect, positive or negative, it often means I didn't do enough homework to understand the product.

I have purchased products in the past that have not lived up to their claims, but the products I've road tested have all been great.

Beyond objective scoring there is an opinion side to a review which is more subjective. I use the review body to provide some insight into my opinions.

When I do a road test, I always show real applications and I try to show how well the product works and how easy it is to use as well as its feature set.

I'm not sure this clearly explains my thinking, but it is my 2 cents anyway ...