Actions
Forum Thread Details
  • Replies 54 replies
  • Subscribers 2760 subscribers
  • Views 7123 views
  • Users 0 members are here
Related

2 Roadtests, 4 Reviewers, 0 Reviews: When A RoadTester Does Not Complete a Review

rscasny

Recently, I selected roadtest applicants who were both new roadtesters  and have not participated much on the community, primarily because I have had some other members tell me that I am using experienced roadtesters too much. (This is never my intent; I select the best applicants. When I roadtest writes only a 1-sentence application, I don't have much to go on and these folks usually don't get selected.)

 

I had to report roadtest results to 2 sponsors. Well, the results for these 2 roadtests were not great: 4 roadtesters haven't done their reviews, they haven't logged in for 3 to 4 months, and they have not responded to all our follow ups that we do for every roadtest. Before I selected these 4, I contacted them and they agreed to commit to delivering the review in 60 days.

 

Now, 2 roadtests is a fraction of all roadtests. For many roadtests we get 100% compliance. I'd say overall we are getting a compliance rate in the upper80s/lower90s, which is good and a far cry from where we were 2 years ago when we were at the sub-20% level.

 

Speaking for the sponsor and myself, we understand that in some situations people will be unable to complete the review. After all, people do get ill and emergencies do arise in everyone's lives, so I never expect to have 100% compliance for every roadtest. I recall someone who had a tree fall on his house in an ice storm. That's a major tragedy in my book. Of course I would not expect a review. But these instances are atypical and very few compared to all roadtests and roadtesters I deal with.

 

While we give the official roadtesters the kits and the shipping for free, someone does pay for these things. Typically, the sponsor pays for the kits and element14 pays for the shipping costs to the roadtesters. But we are more than happy to do this because we value our roadtesters' opinions and we are happy to absorb the costs when the reviews are published.

 

But the question comes to mind: what happens when a roadtester doesn't complete the review:

 

1. The sponsor does not receive all the feedback he is seeking when he enrolled in the roadtest program as a sponsor.

2. Someone who does not complete the review prevents someone who could have done the review from doing it.

3. It prevents our members from learning about the product.

4. It damages the credibility of the program.

5. The delinquent roadtesters prevent themselves for being considered for a high-priced product. If I can't reliably use someone for a $35 US roadtest, how can I expect them to do the review for a $3500 US roadtest!

6. I am placed in the embarrassing situation to explain why 4 people I selected did not do their reviews.

 

There are other reasons but 6 reasons is enough to make my point.

 

It is my responsibility to get the required results for both our sponsors. My bosses also have expectations and expect that I will solve problems. I want people to use the RoadTest program to have fun with technology,  and get some products or equipment that they may not have the budget to get on their own. But I also need mature, responsible members who are engaged in our community, demonstrate an appropriate level of professionalism, and are responsive to our follow up requests. I like to think most roadtesters are professional. I surely do not think these 4 delinquent roadtesters represent the RoadTest group.

 

But going forward, I will have to modify my approach in selection somewhat. And I probably will not hand out all the kits if I don't feel I have enough roadtesters who are knowledgeable about the product, and are mature, professional and reliable. If you were in doubt that I am very serious about people completing their reviews, I hope this discussion makes my position very clear. I apologize to all my loyal and responsbile roadtesters who do complete their reviews. But this is a community and I have to communicate my concerns as well as be open to new ideas and feedback: it's a two-way street.

 

If you can't finish the review, contact me and tell me or my colleague why. If you need a bit more time, that's fine if you need to do more testing. (Needing more time to finish your Differential Equations homework or finishing your digital electronics lab is not a reason for needing more time.) Communicate and keep us in the loop; you will find in your career that developing communication skills is as important as honing those engineering skills. Even if you received a non-working product, you are still required to post a review. Put it out to the members: "I took these readings and I believe it's faulty. Help me troubleshoot it." There are plenty of eager members who are more than happy to help.

 

I want everyone to benefit from a Roadtest. I think completed roadtests are a great place to learn. I think they are a fantastic vehicle for the new engineer who is trying to shine in their first job interview. Read our e-book on IoT careers to learn more shining in a job interview: https://www.element14.com/community/community/publications/ebooks?ICID=hp-iotcareerebook-rotational#?ICID=ebooks-downloa…

 

But you can't reap the benefits of the roadtest program if you don't do your reviews.

 

Sincerely,

 

Randall Scasny

RoadTest Program Manager

  • in reply to shabaz

    Though I have participated in two competitions and have been selected for one more competition Project14 but have taken part in the element14 community from January 2019 which is very less in comparison to senior members and the experience you all senior members have by being in such a wonderful community for many years . No one can become an overnight star and it is a gradual process so I have to work harder and gain a lot more knowledge but I really admire a lot about this amazing community it has provided with a lot of exposure to me and learn a lot of new things and the community members are also very helpful and are ready to help even when I ask them personally. Well, I am still a student and have to manage my studies and exams as well but I think this experience and knowledge I am getting will definitely help me in future so I try to invest my time here from my present time and try to participate in Element 14 Community. This time will try out my hands on the RoadTests & Reviews community and try to submit applications.

    Thanks

    Aabhas Senapati

  • in reply to aabhas

    Hi Aabhas,

     

    You've made effort with Project14, so you're not a new member any more : ) Your thing finder project was interesting : ) And you made an effort to understand how to use the tools, like the code syntax and insert image features in the online editor.. so many people don't. Even really skilled engineers can have difficulty communicating information (partially stereotype, but partially true) , so Project14, the RoadTests, and other activity writing could be great ways to get an edge when job-hunting.

  • I agree to that you should have data of any payment method so that if the roadtester is not able to complete the review without any genuine reason then he has to pay for the hardware , this would eliminate participation of all the members who want to take part just for hardware and not fulfill responsibility of writing the review .

    But you should not remove participation of the new members rather see their participation in the community and get to know about them and you can even take help from   and sort out that who are taking part in project 14 , or any other contests and their credibility to be given the hardware or not .

    I am saying this because I am myself a new member to the community and have participated in Project14 but have not at all participated in RoadTests & Reviews  but will also take part here and I think this is also a great exposure and platform for roadtesting writing the official review of a commercial product and I think that you definitely is a risk giving new members a chance then you should have  less ratio of new members bieng declared roadtesters  but you should not remove most participation of new members as many members like me may be seeking to try their hands on this part of the community along with dedication.

    Though I am no one to tell you as I don't have the responsibilities what you have and you might have some target that you have given to the manufacturer/sponsor about the reviews bieng received but this is my personal  opinion.

    Thanks

    Aabhas Senapati

  • in reply to Fred27

    No problems - happy to be mentioned , especially when it's positive in nature image. Thanks for your comments - it's great to be acknowledged for my efforts. I am truly humbled, but I'd have to say that on the whole, practically all of the Top Members have a track record of delivering quality RoadTests, as do a few new (but active) members. Some RoadTests are more difficult than others and life can get in the way (it certainly did for me in my triple-concurrent RoadTest recently) but I don't take my responsibilities lightly. There are, of course, the jealous types and I don't doubt that the statistics may look lopsided - but RoadTests are not a game of "chance", so I don't take those comments to heart. Participate or not, this is ultimately your own choice - win or not, this is ultimately the choice of element14 and the sponsor.

     

    Needless to say, I do try to make sure I deliver what I promise, or if not, there's a good reason for it and perhaps some bonus content to go with it to make up for it. My reviews can be a bit dry and systematic - but a good quality on-time RoadTest ensures that everyone is happy. This means that element14 forum members and other visitors get some good content to look at, it means the sponsor is happy and gets some exposure/engagement/conversation about a given product (positive or negative) or some feedback to help fix issues, it means the element14 community raises its profile and perhaps means more RoadTest opportunities for all in the future. In my opinion - fail to deliver, and it could cost more than just *your* ability to get future RoadTests regardless of the value of the item.

     

    We've mentioned the whole issue about "free" in the past and whether it was attracting the right types of applicants. At one stage, RoadTest "by invitation" was an alternative where members would be selected for a given product instead, which would have been completely opposite. I suppose, the issue boils down to fairness (perceived or actual) and barriers to entry - so maybe we just have to accept that in order to give new members a try, this is the price to pay. I wouldn't be against having tiered eligibility depending on items - but whether this is realistic from an administration point of view and perhaps block potentially good RoadTesters from going for an item is something to think about. We all have our own areas of expertise - the lower cost items are almost always EVMs/dev boards, so perhaps some people may not be able to deliver a good RoadTest for those and end up unfairly excluded for something else more expensive that is perhaps in their area of expertise. I'd love to see us discover more like-minded highly-technical engineers, hobbyists, etc and would love to see reviews from them.

     

    I know has a tough job balancing it all, so I'm glad that we do get some insight into the challenges that are being faced. I'm not always easy to deal with either - I often come up with queries and special requests to contact manufacturers during RoadTests, and I've left some honest but not-as-favourable reviews in the past, so I'm glad to still be welcome around here. Above all, I wouldn't sleep comfortably if I wasn't being honest with myself, the community and the world.

     

    - Gough

  • in reply to genebren

    I kind of like the idea that a new member could or should publish a road test/blog about something they already have to establish some credibility. This avenue is already available of course, but it isn't mandatory. If it were required for some road tests that a previous road test had to be published, it wouldn't be that big of a barrier for new members, and it could be a nice boost to site content. There is usually plenty of time between a road test announcement and the application deadline to publish a road test. If this criterion was trialed on half the road tests, it would be easy to see if it worked better than the current requirements in terms of number of applicants (deterrent factor) and successful completion of road tests.

    How many currently successful road testers would have been keen enough to publish an un-sponsored road test prior to applying for a sponsored road test?

    It is a tricky question - obviously for the few members who already publish un-sponsored road tests it is a no-brainer, and with what I know now, I wouldn't hesitate to publish an un-sponsored road test, but back when I first started I'm pretty sure I would be wondering if all this work would actually pay off. I guess it is already the case that people wonder if the effort of writing an application is going to pay off, and this is even more effort. When you first start publishing it is not clear what all the benefits might be, so the real road test equipment may be the only motive for a new applicant. If it is considered to be a lottery with low probability of "winning" it won't appeal to everyone.

  • That is a shame that you took a shot to appease someone but then get burned with no RoadTest reviews.  I'm curious, was the person that had complained about only experienced RoadTesters getting selected one of the 4 that you took a chance with? 

     

    I'm sure there are good folks that are either new to the site or had seen the RoadTest posted on something like LinkedIn or another site who are more than willing to produce a good review, yet they may not get a shot now due to the actions of the 4 you mention.  Outside of doing it like the hotels do by requiring a RoadTester to supply a credit card number before receiving the product, there must be another means to screen a potential RoadTester.   Perhaps requiring them to have participated in something like a Project14 contest or have posted a certain amount of content on the site, or a reference to postings elsewhere, before being eligible? 

  • Randall,

     

    I think the program is working quite well the way you have shaped it so far, and this is definitely credit to you It is frustrating that, despite all the efforts and the good will placed to try keeping the program as open to newcomers as possible, you get this kind of response. But, as pointed out in other discussions on the topic, I think those cases of "missed reviews" do not detract from the success of the program, as it is unrealistic to expect a 100% return for all the roadtests.

     

    For securing a sustained success for the roadtest program, it is important it is perceived as an open program, hence I believe it is vital that new blood is continuously brought on board. As I already had chance to explain, I fear that making the application process too hard or convoluted, or introducing a sort of "rite of passage" is a risk: lets not forget that many potential roadtesters hold a full time job, and need to be attracted to the program somehow. If they feel they need to invest too much time, they might just not bother trying at all.

     

    As I said, I feel the program is working quite well as it is, and your statistics seem to confirm that, so, ultimately, I think you should resist any urge to change the application process, especially if such change could involuntarily end up creating a preferential route for the experienced roadtesters. And if the only good applications you receive are coming from the experienced roadtesters, perhaps I would still try to contact a few newcomers, and ask if they can expand on their application. If they show interest and commitment, then you could decide if any of them is worth the risk.

     

    Here obviously I'm not talking about the people that are out to game the system: no matter how easy or hard you make the application process, there will always be people that will play it to their advantage, so trying to shape the application process to weed-out such people ends up being  counterproductive for all the rest of genuine applicants. All you can do is make the suppliers aware of the risk (they probably already know that anyway).

     

    Thank you for all the hardwork you put to make the roadtest program such a great one!

     

    Fabio

  • I think maybe roadtesters could be divided into tiers, lets say 4 levels:

     

    tier 0: Is the level of any new member.

    tier 1: Is reached after writing a good blog post (ideally a review), this would show that the member can actually write something for the community.

    tier 2: is reached for roadtesting an item that costs <100 USD

    tier 3: is reached for roadtesting an item that costs <1000 USD

     

    Something like this would probably increase the number of written reviews, not to 100% but probably closer.

  • Randall,

     

    I understand the desire to have new blood involved in Roadtests, but maybe it is should be necessary for Roadtesters to have participated in the community first, by either competing in other contests or blogging prior to being approved to participating in a Roadtest.  But not matter where you set these thresholds for participation, some people will game the system and renege on their promise to complete the necessary review.  It could be the allure of something for nothing, or some other personality flaw, that would allow a person to do this.  I just don't know, but you do have difficult choices to make and from where I stand, I believe that you have done the best job possible of fairly assigning these opportunities to wide section of this community.

     

    Thanks for all that you do!

     

    Gene

  • I hope that the two roadtests were for relatively inexpensive products.

     

    Sadly, you are bound to get people who see their opportunity to get something for nothing but, as you have said before, if you don't take some sort of risk, you'll never get new roadtesters. This time last year, you took a chance on me as I had not come across the element14 community before and I am glad you did. I delivered that first roadtest and have done several others since then. The benefits for me have been way more than the "free" products - I have learned so much, not just from the stuff I have tested, but from reading the reviews that others have written, and people's blogs and design challenges. It is a real community, albeit virtual.

     

    I'm guessing that you already tend to go for "tried and tested" roadtesters for the expensive gear, which is what other people are suggesting. I see it as an apprenticeship in a way - build up your skills, experience and reputation before taking on the bigger challenges.