Actions
Forum Thread Details
  • Replies 22 replies
  • Subscribers 2753 subscribers
  • Views 4162 views
  • Users 0 members are here
Related

When Is A RoadTest Review Too Much or Too Little?

rscasny

I have been spending more time reading (including proofreading) roadtest reviews. Some of our suppliers are reading your reviews and providing me feedback on them as well. One of the main reasons why a sponsor participates in the RoadTest program is to get feedback about the product, documentation, design tools, and so on. So, your reviews could be read by a variety of stakeholders within the sponsor's organization (engineers, business managers, marketing, etc.). Your reviews are highly visible. (So, make sure you do a spell check before you publish them, please!) While I have always said you don't be a superior writer to be a roadtester, roadtesters should try to spot basic errors and correct them. If you need help, please contact me.

One thing I have noticed recently as I read reviews is that some reviews cover a lot of things. Some have indexes to separate blogs that function as tutorials. I have also noticed that some are fairly short and don't "tell the story behind the review" visually to make a reader think "Wow" this is cool, incredible, etc."

So, when is a review too much or too little?

I do not desire to establish stringent max-word-count guidelines. I never will. After I choose you as an official roadtester, I need you to "take ownership" of the roadtest. I fully understand that people will write reviews differently. In fact, that is something I ponder when I go through the roadtest applications. I want to obtain different kinds of reviews, which in my mind represents different kinds of engineering minds and, yes, different kinds of customers.

But back to my core question: When is a review too much or too little?

There are some experienced roadtest reviews who can juggle gads and gobs of information and, like a conductor at a symphony, produce a multi-faceted, "perfect" review. These are great productions.

But there are also some reviewers who can easily perform the testing but when one reads the review it doesn't flow so smoothly and makes it a bit harder to read.

I personally think: don't forget the basics.

A review should (a) explain what you are going to do and why, (b) provide a background on the product because not all readers will be familiar with it, (c) conduct and describe the results of your tests (this can include an unboxing), (d) go through how you experienced the product and (e) draw some conclusions.

For some roadtesters, too much is when there are side information that bogs down the basics, so they never get to the basics. But for other roadtesters, they can add the side information to explain things that perhaps are not explicitly stated in the sponsor's documentation; these are skilled roadtesters. 

When is a roadtest review too little? The comments are a guide. If our members ask the roadtester a lot of questions, then maybe your review lacks something.

People learn a lot visually. Take a cartesian grid. One can learn more from a graph of a sensor temperature vs Time than just a table of temp data. When a review has no visuals that support what is being stated in the text of your narrative, it is probably too little. Finally, a just-the-right-length review is persuasive. Read the comments of a bunch of reviews. When you see comments like "nice review," you know you have hit a homerun.

Thank you for your time.

Randall Scasny

RoadTest Program Manager

  • in reply to Fred27

      Like  , I also agree on the "potential purchaser" target audience.  SOMETHING sparked your interest enough to apply for the RoadTest.  It had to be more than wanting new gear -  otherwise you probably would not have been selected.  Does the RoadTest item deliver on that spark of interest?  If so, share the experience.  If not, why not?

    I also agree that the cost of the item carries some expectation - within reason.  While insulated screwdirviers and some of the MCU reviewed run about the same price, I would have a hard time coming up with something to say about the screwdrivers.  "Yup.  They're insulated like they said."  I accept the shorter reviews.  BUT... the MCU... tell me why I want that one.  Why should I learn another toolchain?  What makes it worth that effort?  Again, there was something that stood out.  Let's hear about it.

  • Please forgive me as I am playing devil's advocate here but If you were to measure the time and effort that goes into producing some of the better RoadTests against what it would cost to produce the equivalent material internally you would probably find that the manufacturer is getting thousands of dollars worth of 'copy' and exposure in exchange for a free sample of their product.

    Sometimes though, due to a variety of factors, a RoadTest doesn't live up to the applicant's original intentions.  I don't think this is any great sin.  Furthermore I think the manufacturers have to be willing to accept this gamble.  It's a reasonable risk for the level of reward they sometimes receive.

    As far as length goes my impression is that RoadTests have gotten longer over time.  Don't get me wrong I will not fault anyone for writing a long RoadTest.  I suspect, however,  that a subconscious bar has developed where authors feel they have to make a lengthy review.  If this bar keeps moving towards unattainable levels fewer people will have the time and energy to tackle a RoadTest and of course fewer people will read through the reviews as well.  So while there is some concerns about RoadTests not being long enough I think we have to worry about the other side of the coin as well and make sure they aren't too long.

  • in reply to Fred27

    Agree. I was thinking of writing something similar but thankfully the community got to read your good concise answer instead of my wordy waffle.

    I would add that for microcontroller dev boards a key question I try and address in the review, as it's on my mind when buying something new, is whether you can get up and running in a reasonable time period (based on experience with other boards) without getting bogged down in the learning process. Often with new boards there is very little to work with other than your own experience and the manufacturers documentation. So I try and highlight the pitfalls and the gaps you had to fill along the road test journey.

    Sometimes a short review can be seen as a good thing, especially if everything works very well.  I leave content filling to news reporters and thus I don't expect engineers to follow that wordy approach, just for the sake of it. 

  • It's always a tricky one. I try to target my road tests towards what I think a potential purchaser might want to know.

    For a microcontroller / FPGA boards that's roughly "does this do what I need and is it easy to get started coding with it". Here I'll tend to focus on the toolchain, documentation and usability as to be honest most devices are more than capable for most projects.

    For bigger stuff like a scope, I've tried to target a couple of groups - a serious puchasers with a detailed review and most E14 members who are thinking "that's cool, I wish I could afford one" with a video summary.

    The content length tends to depend on what I want to say, but obviously is a little influenced by the "size" (i.e. cost mostly). For bigger ticket items it's hard to say that a review could be too long. I've definitely seen the odd one in the past that have been a bit short.

  • in reply to Jan Cumps
    Jan Cumps said:
    It takes a brave manufacturer to do that, and to embrace it

    The phrase:

    "Well, if they don't like the review, maybe they should improve and make a better product?" has certainly passed element14 Community staff's lips before when it's been unanimously bad Smiley

  • in reply to cstanton
    cstanton said:
    A lot of their approach is still 'old school' in this regard, rather than what I consider transparency and accountability.

    That is true. And there is a risk when sending out a product, without control on what the feedback will be.

    It takes a brave manufacturer to do that, and to embrace it.

  • in reply to feiticeir0

    yes, and maybe help the reader to understand if it would be a fit in their design / lab. If we can show how well (or not so well) the product performs its advertised functions, it will help possible customers.

  • in reply to Jan Cumps
    Jan Cumps said:
    I would prefer that all communication is public, in the community.

    There are certainly companies that do not want that to be the case, for the sake of their own public relations and reputation. A lot of their approach is still 'old school' in this regard, rather than what I consider transparency and accountability.

    Jan Cumps said:
    I try to write for users of the device, not for the sponsors. I do try to review all things e14 or sponsor suggest before the test starts, but the audience is the potential user

    This is definitely the way to write that has the most value.Companies already have their own opinion of their hardware :D 

  • I totally agree with you and against myself I maybe talking.  Sometimes I take the long road to explain something simple. I just want to make sure the reader understands it well, but sometimes I complicate a bit to much.

    I've seen Reviews that are extensive tutorials on the use of the hardware, and I think it makes it to extensive or even boring - to much to read. Some people might like it, others don't.

    In my opinion, a Road Test should explain a bit how it works and some steps taken (a tutorial) to reach the destination we set out to reach, so that the reader can reproduce it without any problem. Nothing too fancy.

    At least, it should lead the reader to further discovery and investigation if he or she caught an interest on the hardware being reviewed.

    That's it.

  • in reply to wolfgangfriedrich

    I would prefer that all communication is public, in the community.

    I try to write for users of the device, not for the sponsors. I do try to review all things e14 or sponsor suggest before the test starts, but the audience is the potential user.