Actions
Forum Thread Details
  • Replies 54 replies
  • Subscribers 1998 subscribers
  • Views 903 views
  • Users 0 members are here
Related

2 Roadtests, 4 Reviewers, 0 Reviews: When A RoadTester Does Not Complete a Review

rscasny

Recently, I selected roadtest applicants who were both new roadtesters  and have not participated much on the community, primarily because I have had some other members tell me that I am using experienced roadtesters too much. (This is never my intent; I select the best applicants. When I roadtest writes only a 1-sentence application, I don't have much to go on and these folks usually don't get selected.)

 

I had to report roadtest results to 2 sponsors. Well, the results for these 2 roadtests were not great: 4 roadtesters haven't done their reviews, they haven't logged in for 3 to 4 months, and they have not responded to all our follow ups that we do for every roadtest. Before I selected these 4, I contacted them and they agreed to commit to delivering the review in 60 days.

 

Now, 2 roadtests is a fraction of all roadtests. For many roadtests we get 100% compliance. I'd say overall we are getting a compliance rate in the upper80s/lower90s, which is good and a far cry from where we were 2 years ago when we were at the sub-20% level.

 

Speaking for the sponsor and myself, we understand that in some situations people will be unable to complete the review. After all, people do get ill and emergencies do arise in everyone's lives, so I never expect to have 100% compliance for every roadtest. I recall someone who had a tree fall on his house in an ice storm. That's a major tragedy in my book. Of course I would not expect a review. But these instances are atypical and very few compared to all roadtests and roadtesters I deal with.

 

While we give the official roadtesters the kits and the shipping for free, someone does pay for these things. Typically, the sponsor pays for the kits and element14 pays for the shipping costs to the roadtesters. But we are more than happy to do this because we value our roadtesters' opinions and we are happy to absorb the costs when the reviews are published.

 

But the question comes to mind: what happens when a roadtester doesn't complete the review:

 

1. The sponsor does not receive all the feedback he is seeking when he enrolled in the roadtest program as a sponsor.

2. Someone who does not complete the review prevents someone who could have done the review from doing it.

3. It prevents our members from learning about the product.

4. It damages the credibility of the program.

5. The delinquent roadtesters prevent themselves for being considered for a high-priced product. If I can't reliably use someone for a $35 US roadtest, how can I expect them to do the review for a $3500 US roadtest!

6. I am placed in the embarrassing situation to explain why 4 people I selected did not do their reviews.

 

There are other reasons but 6 reasons is enough to make my point.

 

It is my responsibility to get the required results for both our sponsors. My bosses also have expectations and expect that I will solve problems. I want people to use the RoadTest program to have fun with technology,  and get some products or equipment that they may not have the budget to get on their own. But I also need mature, responsible members who are engaged in our community, demonstrate an appropriate level of professionalism, and are responsive to our follow up requests. I like to think most roadtesters are professional. I surely do not think these 4 delinquent roadtesters represent the RoadTest group.

 

But going forward, I will have to modify my approach in selection somewhat. And I probably will not hand out all the kits if I don't feel I have enough roadtesters who are knowledgeable about the product, and are mature, professional and reliable. If you were in doubt that I am very serious about people completing their reviews, I hope this discussion makes my position very clear. I apologize to all my loyal and responsbile roadtesters who do complete their reviews. But this is a community and I have to communicate my concerns as well as be open to new ideas and feedback: it's a two-way street.

 

If you can't finish the review, contact me and tell me or my colleague why. If you need a bit more time, that's fine if you need to do more testing. (Needing more time to finish your Differential Equations homework or finishing your digital electronics lab is not a reason for needing more time.) Communicate and keep us in the loop; you will find in your career that developing communication skills is as important as honing those engineering skills. Even if you received a non-working product, you are still required to post a review. Put it out to the members: "I took these readings and I believe it's faulty. Help me troubleshoot it." There are plenty of eager members who are more than happy to help.

 

I want everyone to benefit from a Roadtest. I think completed roadtests are a great place to learn. I think they are a fantastic vehicle for the new engineer who is trying to shine in their first job interview. Read our e-book on IoT careers to learn more shining in a job interview: https://www.element14.com/community/community/publications/ebooks?ICID=hp-iotcareerebook-rotational#?ICID=ebooks-downloa…

 

But you can't reap the benefits of the roadtest program if you don't do your reviews.

 

Sincerely,

 

Randall Scasny

RoadTest Program Manager

Parents

  • Hi All.

     

    I'd like to thank everyone who has participated in this discussion. It's help me clarify the problem and think about solutions.

     

    Change Idea #1: I used to send a standard availability email when I notify the roadtest finalists wether they are new roadtesters or experience. I will start sending out a separate email for new roadtesters that spells out the responsibilities of a roadtester and the consequences of not doing a review.

     

    Change Idea #2-- I am thinking that I should have an onboarding webinar for new roadtesters. It may be a way to build relationships with new people and they can hear from me and what I expect.

     

    Change Idea #3--I have had some instances when a member was not available because he was doing another roadtest. I am considering setting up later due dates for these people instead of choosing someone else who wrote a below average application.

     

    Change Idea #4--If I got 5 kits to hand out and I honestly feel I don't have 5 reliable roadtesters, I may have to select less than 5 and reserve the other kits for a project.

Reply

  • Hi All.

     

    I'd like to thank everyone who has participated in this discussion. It's help me clarify the problem and think about solutions.

     

    Change Idea #1: I used to send a standard availability email when I notify the roadtest finalists wether they are new roadtesters or experience. I will start sending out a separate email for new roadtesters that spells out the responsibilities of a roadtester and the consequences of not doing a review.

     

    Change Idea #2-- I am thinking that I should have an onboarding webinar for new roadtesters. It may be a way to build relationships with new people and they can hear from me and what I expect.

     

    Change Idea #3--I have had some instances when a member was not available because he was doing another roadtest. I am considering setting up later due dates for these people instead of choosing someone else who wrote a below average application.

     

    Change Idea #4--If I got 5 kits to hand out and I honestly feel I don't have 5 reliable roadtesters, I may have to select less than 5 and reserve the other kits for a project.

Children
  • in reply to rscasny

    They all sound like good ideas. I don't think there would be any harm suggesting some criteria that might help get you selected though. "Preference given to members who have watched and commented on the onboarding webinar", "Preference given to members who have already written a blog about a development board". Something like that. They can all be at your discretion, but a hint at some expected prior effort.

     

    I've also been wondering if there's any way experienced road testers could mentor those who want to be selected. However, if the problem is a lack of effort rather than a lack of skills on the part of the missing road tests then I don't suppose there's much that can be done.

     

     

    I notice that the Azure Sphere dates have been pushed back a bit. I hope that doesn't mean that of the 359 applicants there weren't 50 good ones.

  • in reply to Fred27

    Thanks.

     

    No, we are trying to see how we can get all the applicants involved in the Azure Sphere board.

  • in reply to rscasny

    All the change ideas are good IMHO. But

     

    Change Idea #2-- I am thinking that I should have an onboarding webinar for new roadtesters. It may be a way to build relationships with new people and they can hear from me and what I expect.

     

    This is the best I think. And if you want to involve some of the already known road testers / top members etc. maybe this can be a good moment for create a community relationship. Anyway this sound very good.

     

    Enrico

  • in reply to rscasny

    CI#1: This has merit. The new parties know the rules going in. The more they know, the easier it is for you to take action. What consequences can you enforce?

     

    CI#2: This idea has some merit. I am assuming the completion of the webinar for new applicants can be tracked. I suggest once they are accepted, you require them to complete the process a week before shipping the goods. Failure to comply, you can withdraw their names.

     

    CI#3 is valid. It is not unreasonable if a person is committed to a RoadTest, the scheduling for another RoadTest can be delayed. You can make that decision depending on the commitment to your vendors.

     

    CI#4 I'm going to assume no RoadTest without product distribution is better than no RoadTest and product gone with no return. If you send kits to marginal participants, then you are putting yourself at risk. How you can use the product for a future test, is up to you. I rather liked the grab bag RoadTest, I got selected for.

     

    How long do you have to be a member to be eligible? I think a cooling down period may help screening, especially if advertisement outside this site is attracting undesirables. Active members for <30 Days need not apply.

     

    Is there someway to create a minor league, that players start out in and work their way to the big leagues. Or maybe a tiered RoadTest participation roster. The vendors can help select the tier they want their product tested at. Their is a junior, minors and professional levels. Maybe you have to work the AAA "triple A" club (i.e. webinar, why me plea and other obstacles) before getting to the juniors. From there you work you way up to the pro's.

     

    I'm thinking the mission badges must be able to help you there. You have people training in the minors for a shot.

     

    Just spit balling ideas while I am waiting for the Canadian world cup of soccer womans game to start. Go Canada!

     

    Sean

  • in reply to rscasny

    Dear ,

     

    Some interesting ideas for solutions:

    Change Idea #1: A good idea, low implementation cost, but perhaps won't do much as initially, everyone will be "willing". Even with the standard e-mail, your new RoadTesters would have at least replied to accept the item ... before going silent later. I suspect it may not have much of an effect in the end.

     

    Change Idea #2-- Perhaps a more difficult idea to implement, as many of us RoadTesters are scattered all over the world, thus time zones can be an issue along with internet connectivity. It is, unfortunately, why I've ended up not attending many of the sponsor's webinars even though I would have liked to. I say that from the perspective of someone that's on rather unreliable quota limited LTE (~20GB/month) who often has to curtail video entertainment to make it through the month. Holding a video conference can be a little difficult for some people.

     

    Change Idea #3--This might be a good idea, but I'm concerned that the sponsor may not always be willing to have "late" content appear as the marketing value might not be there. Furthermore, some of the RoadTesters who are also reviewing the product may misunderstand the special deadlines awarded and incorrectly criticise an established RoadTester. There is also the loss of resources in the sense that when all RoadTests for a given product run in parallel, we can consult other RoadTesters for suggestions to overcome problems. This may not work as well in the case of disjoint start-end dates. Another potential problem is that later RoadTesters may have their impartiality compromised if they view or are influenced by already published reviews.

     

    Change Idea #4--I think you've probably been doing this to some degree already. I don't see this as a bad thing - there's no point awarding to applicants who don't meet a minimum standard.

     

    I'm not sure there is any "magic bullet" solution to the issues, but these are just my thoughts.

     

    - Gough

  • in reply to rscasny

    As part of the onboarding webinar and/or email for new testers, it might be helpful to show/link examples of satisfactorily completed road tests to illustrate what you're looking for in a completed RoadTest review.

  • in reply to rscasny

    I think ideas #1 and #2 are good: any guidance and information about the program, how to apply, what is expected and how to prepare and deliver a RoadTest review will be of invaluable help for all the members, not just the new ones.

     

    Ideas #3 I don't think it is a good one. I can see where you are coming from with this, and what you are trying to address here, but I think the RoadTest program should have few and very clear rules, and one of them should be the start date and the end date of the RoadTest itself. Starting moving dates to accomodate members schedules can cause confusion and lead to odd situations.

     

    Idea #4: to some extent I believe this is already happening, so I don't see it as a bad thing in itself . The problem is that you might end up not taking any risk at all on new members this way, and that could be a problem. As already pointed out by many fellow members and yourself, writing a good review, unless you are particularly gifted person, is a skill that requires lots of "trial and error" attempts, and lots of feedback. Writing a not -so-good application doesn't tell much on the real potential of a candidate, just like writing a very good one is not a guarantee of success. Even with interviews, sometimes you cannot find out what to expect from a candidate, so ultimately the only way is to take a risk, trust your instinct and choose some candidates regardless. Not choosing, to play safe, is a bad bet.

     

    So, in short, I'm in favour to any change that introduces clarity in the process, helps members understand what is expected from them and encourage participation. The other changes proposed, I'm not sure what they aim at. If it is to reduce the risk of "no-show" review, I'm afraid they won't make much difference. If it is to try getting good quality reviews, then I think  perhaps, for clarity and transparency sake, could be more useful to resurrect something like the RoadTest Plus program, where you can invite experienced members to do a review. For some (or all) RoadTests, you could run a RoadTest Plus (project based rather than just test based) where all the unassigned kits from the regular RoadTests are made available.

     

    Personally, as I said, I think the program is working quite well as it is now. I don't know what bargaining power you have with the suppliers, but I suspect it could be possible to make sure that they agree with assigning, for example, 75% of the kits for regular members, 25% expressly reserved for new members. That way, you can manage all the stakeholders expectations while making the program rules as clear and transparent as possible and chances are given to new members.

     

    One last change I would love to see: some feedback from you or the supplier about the published review. I understand it would generate more work for you and the supplier, and it could be a bit of double-edged sword, but for all the members that genuinely are trying to do a good job of their review, this feedback would be invaluable. Preferably, this should be delivered as private conversation, to avoid any problem. I know, for one, I would love to receive such feedback, to understand where I need to work more, in order to improve.

     

    That's my 2p.

     

    Fabio

  • in reply to rscasny

    Hi ,

     

    I like all the change ideas and would like to propose an additional one:

     

    Selected roadtesters should write an introduction and testplan that goes further than the application, prior to sending out the equipment.

  • in reply to gpolder

    That's not a bad idea. One thing I like about the new application template is that you can get your proposal across without having to write loads. It saves everyone time - especially Randall reading all of them. However, I always do some research. I'll often have an IDE installed before the items turn up, etc. I'd that's part of my road test I'll often start writing or making notes. Formalising this might be helpful and if the road tester had done done of the work already they might be keener to continue.

  • in reply to colporteur

      wrote:

     

    CI#4 I'm going to assume no RoadTest without product distribution is better than no RoadTest and product gone with no return. If you send kits to marginal participants, then you are putting yourself at risk. How you can use the product for a future test, is up to you. I rather liked the grab bag RoadTest, I got selected for.

     

    Although, as you pointed out yourself, the "grab bag" RoadTest had a very poor rate of completed reviews!