Actions
Forum Thread Details
  • Replies 54 replies
  • Subscribers 2569 subscribers
  • Views 5673 views
  • Users 0 members are here
Related

2 Roadtests, 4 Reviewers, 0 Reviews: When A RoadTester Does Not Complete a Review

rscasny

Recently, I selected roadtest applicants who were both new roadtesters  and have not participated much on the community, primarily because I have had some other members tell me that I am using experienced roadtesters too much. (This is never my intent; I select the best applicants. When I roadtest writes only a 1-sentence application, I don't have much to go on and these folks usually don't get selected.)

 

I had to report roadtest results to 2 sponsors. Well, the results for these 2 roadtests were not great: 4 roadtesters haven't done their reviews, they haven't logged in for 3 to 4 months, and they have not responded to all our follow ups that we do for every roadtest. Before I selected these 4, I contacted them and they agreed to commit to delivering the review in 60 days.

 

Now, 2 roadtests is a fraction of all roadtests. For many roadtests we get 100% compliance. I'd say overall we are getting a compliance rate in the upper80s/lower90s, which is good and a far cry from where we were 2 years ago when we were at the sub-20% level.

 

Speaking for the sponsor and myself, we understand that in some situations people will be unable to complete the review. After all, people do get ill and emergencies do arise in everyone's lives, so I never expect to have 100% compliance for every roadtest. I recall someone who had a tree fall on his house in an ice storm. That's a major tragedy in my book. Of course I would not expect a review. But these instances are atypical and very few compared to all roadtests and roadtesters I deal with.

 

While we give the official roadtesters the kits and the shipping for free, someone does pay for these things. Typically, the sponsor pays for the kits and element14 pays for the shipping costs to the roadtesters. But we are more than happy to do this because we value our roadtesters' opinions and we are happy to absorb the costs when the reviews are published.

 

But the question comes to mind: what happens when a roadtester doesn't complete the review:

 

1. The sponsor does not receive all the feedback he is seeking when he enrolled in the roadtest program as a sponsor.

2. Someone who does not complete the review prevents someone who could have done the review from doing it.

3. It prevents our members from learning about the product.

4. It damages the credibility of the program.

5. The delinquent roadtesters prevent themselves for being considered for a high-priced product. If I can't reliably use someone for a $35 US roadtest, how can I expect them to do the review for a $3500 US roadtest!

6. I am placed in the embarrassing situation to explain why 4 people I selected did not do their reviews.

 

There are other reasons but 6 reasons is enough to make my point.

 

It is my responsibility to get the required results for both our sponsors. My bosses also have expectations and expect that I will solve problems. I want people to use the RoadTest program to have fun with technology,  and get some products or equipment that they may not have the budget to get on their own. But I also need mature, responsible members who are engaged in our community, demonstrate an appropriate level of professionalism, and are responsive to our follow up requests. I like to think most roadtesters are professional. I surely do not think these 4 delinquent roadtesters represent the RoadTest group.

 

But going forward, I will have to modify my approach in selection somewhat. And I probably will not hand out all the kits if I don't feel I have enough roadtesters who are knowledgeable about the product, and are mature, professional and reliable. If you were in doubt that I am very serious about people completing their reviews, I hope this discussion makes my position very clear. I apologize to all my loyal and responsbile roadtesters who do complete their reviews. But this is a community and I have to communicate my concerns as well as be open to new ideas and feedback: it's a two-way street.

 

If you can't finish the review, contact me and tell me or my colleague why. If you need a bit more time, that's fine if you need to do more testing. (Needing more time to finish your Differential Equations homework or finishing your digital electronics lab is not a reason for needing more time.) Communicate and keep us in the loop; you will find in your career that developing communication skills is as important as honing those engineering skills. Even if you received a non-working product, you are still required to post a review. Put it out to the members: "I took these readings and I believe it's faulty. Help me troubleshoot it." There are plenty of eager members who are more than happy to help.

 

I want everyone to benefit from a Roadtest. I think completed roadtests are a great place to learn. I think they are a fantastic vehicle for the new engineer who is trying to shine in their first job interview. Read our e-book on IoT careers to learn more shining in a job interview: https://www.element14.com/community/community/publications/ebooks?ICID=hp-iotcareerebook-rotational#?ICID=ebooks-downloa…

 

But you can't reap the benefits of the roadtest program if you don't do your reviews.

 

Sincerely,

 

Randall Scasny

RoadTest Program Manager

  • in reply to gpolder

    That's not a bad idea. One thing I like about the new application template is that you can get your proposal across without having to write loads. It saves everyone time - especially Randall reading all of them. However, I always do some research. I'll often have an IDE installed before the items turn up, etc. I'd that's part of my road test I'll often start writing or making notes. Formalising this might be helpful and if the road tester had done done of the work already they might be keener to continue.

  • in reply to colporteur

      wrote:

     

    CI#4 I'm going to assume no RoadTest without product distribution is better than no RoadTest and product gone with no return. If you send kits to marginal participants, then you are putting yourself at risk. How you can use the product for a future test, is up to you. I rather liked the grab bag RoadTest, I got selected for.

     

    Although, as you pointed out yourself, the "grab bag" RoadTest had a very poor rate of completed reviews!

  • Randall,

     

    I’m sorry for jumping in late on this discussion, but I’ve been very busy lately and of course all the E14 network issues. 

     

    First, I would like to thank you for all your great work.  I don’t envy your job.  I have worked with high school students for a while and what they try to pass off for homework is sad sometimes.  What is more amazing is how mad they get when you don’t accept their work. 

     

    Second, I may have missed it, but you didn’t mention what sorts of roadtests were not being completed on time.  There can be a lot of unexpected work involved in any roadtest. I have noticed some reviews appear relatively late.  As some have already mentioned, on high ticket items it would seem fair to add to the rules for roadtests that until you successfully completed a number of smaller roadtests you would not be legible for larger items.  Do you ever have to contact a reviewer to tell them their review was not satisfactory?

     

    Third, I know you mentioned that you had tried to contact those you who have not completed reviews, with some not replying.  I have also noticed a few reviewers that don’t enter their reviews using the form on the product page, but through the roadtest blog page.  Their reviews never seem to show up as completed. 

     

    Thanks again for doing a great job.

     

    Regards, Gordon

  • It is sad to hear that after the great efforts from the element14 staff things like this still happen to a great program.

     

    I mainly came to say that I agree with and other members that advertising free products as a way to attract people to the community -which understandably is an effective marketing formula, is probably not a good approach IMO. Where I come from there is a saying "the first impression is the one that counts"; if a person joins the community (or participates for the first time) because of the "you can get this for free" ad it will inevitably attract freeloaders adding more risk of failure. This not only impacts the RT program, this also impacts the programs like Design Challenge where you can get free stuff in advance and then vanish if that was your intention. This is evident in some profiles where you can see the members just created the account to apply for the free stuff and stay silent without any contribution to the community.

     

    I do agree that strategies that keep the Road Test program open to newcomers is something we all benefit from -I myself was a new Road Tester once, new comers, marketing, readers and e14 will benefit from this as well, I consider useful and perhaps regular contributions to the community should be a must before starting in the program and I also agree with others that in order to test the waters members should start with something cheap and/or easy as a way to prove that a quality review can be delivered on time before handing more expensive/complex/specialized equipment.

     

    Luis

  • in reply to rscasny

    I think there may be other factors that might be useful in choosing applicants.

    • one has to do with matching road tests to people who can handle them. Specifically predicting how hard a particular road test will be to complete, given a particular skill set or sometimes any skill set. I often try to predict what the response rate will be for each road test and find the predictions are significantly better than just random chance, so predictions might be useful in deciding how to stack the deck.
    • part of the application could be a requirement to estimate how many hours will be spent on the road test and blog. (separate numbers) I am sure it would be useful to   if an applicant spent some effort to explain how much time they are willing and able to commit to a road test.

    I think Randall is doing a superb job screening, filtering and encouraging applicants, but there are some road tests where it will always be problematic to get a high completion rate.

    • Part of this is due to applicants not honoring their commitments but this is not the only reason
    • When I look at a new road test, I sometimes think things like: "wow - there aren't too many members who will be able to handle this road test" or "yikes - this road test will require a scary amount of work" or "this is new and different with rudimentary support materials so it will take a lot of learning and perseverance". When I get those types of reactions, it almost always means there will be few completed road tests, partly because other experienced road testers will shy away from applying due to similar reactions. The road test may scare off knowledgeable road testers leaving only naïve applicants that rapidly discover they can't handle it.
    • Part is due to the nature of the specific equipment being road tested. The manufacturer may not have made it easy enough to do a road test.
    • Sometimes the equipment is so esoteric, the average member doesn't have a use for it and is not very motivated once they realize this.
    • Sometimes the road test is just very hard and even with the best intentions, success is not likely (for some, blogging about a failure is super hard)

    The bottom line is when there is a poor response rate, don't automatically assume it was due to poor applicant selection criteria or poor screening, there are a lot of factors that will occasionally conspire to yield a poor result. Sometimes it is even predictable before the candidates are selected.

    It is worth examining each anomaly, but some of it is just statistical reality.

  • in reply to dougw

    Hello Doug,

     

    I agree with your points about matching a RT to an applicant with the right skill set required and also agree that Randall and his staff do a remarkable job with the RT program but, when a reviewer doesn't proactively seek for help -in case of issues-, hasn't produced any kind of content within the required 60-day period or even worst, doesn't respond to follow up requests I believe in the majority of the cases these all point to the same direction.

     

    Luis

  • I know I haven't really posted much here on this community in the past but I read this post and just wanted to provide some insight as to what I have seen in the past for companies that I have done beta testing for in the past and I feel that these methods may work here if you are looking for a method to thin out those that aren't serious about providing acurate and or honest reviews. Please see the poorly formatted list of methods if seen before:

     

    1) Company XYZ provides the ability to test a product with high value that must be returned at the end of the test for "modifications" to a new user they are told that they must provide a deposit that will be returned after the success of the beta... experienced and proven testers don't have to meet this requirement... (as a new tester I wasnt happy about the deposit method but for items that really peaked my interest I dealt with it and put the deposit on my credit card or to my PayPal...and did the test and as soon as the company received the confirmation from FedEx that the package was received the funde were promptly refunded. This I'm sure kept the dishonest from applying)

     

    2) (my favorite method) The testing company requires everyone to complete a review/test on a website that has known issues... I'm not sure how many there are but they look for people to find and report on that and then if they have done so to a good standard they are selected for future tasks... if not some feed back it provided and they must wait a month before reapplying. This helps weed out the people that tend to not provide quality results.

  • in reply to Fred27

    I could not agree with you more on the Word FREE enticing reviewers who arw only there because the item is FREE. Unfortunately, it doesn't leave a lot of reassurance and trust for those of us who are new to this website and would be grateful for the chance to test out a product. Maybe you could have the new users (speaking as one myself) purchase the product then, offer a refund once their review has been completed.  I would be very willing to do that since chances are it's a product I probably wanted to buy anyway.

     

    Well, I'm just glad I stumbled across this site I'm search of information for an IT project I'm going to be undertaking soon.  You have NOT heard that last of me!

    Thanks all for a really great, informational site. I'm looking forward to learning lots from the questions I will surely have, lol.

  • in reply to mom2673

    Hi MaryAnn.

     

    Welcome to element14.

     

    We have tossed around your idea about buying the product. I'm not ready to do that, and I'm not sure how easy it would be to do in your particular system.

     

    But please apply to a roadtest that you are interested in. Write a good application. You may find yourself getting a product to review.

     

    Randall Scasny

    RoadTest Program Manager

  • Hi Randall,

     

    I will put forth an experience from the opposite viewpoint.

     

    My own experience is that I was not chosen as a RoadTester on early attempts although I feel they were good submissions and topics I had familiarity with.  While not selected on my first attempt I went ahead and did my project anyway with a board I bought.  In other words I really was serious about doing it. Of the 3 persons actually selected (all of them level 1 as was I), none actually wrote up a RoadTest.  This was before you were managing the program but it actually dissuaded me from making entries for a while....

     

    I was not selected until after I had become active and in retrospect this seems reasonable. In general I believe extra weight should be given to those who are active on e14 and a statement to that effect could be included in the instructions. If the company making the product has specific requirements for testers (like current job or educational /work background) that should be stated also.  This might help RoadTesters with their expectations of being selected and give incentive to become more active.