RoadTest Review Success Rate

Hi Sean,

Thanks for writing.

I am always looking at the rate of completed reviews. Like in any job, I have to make reports to my boss and the sponsor. If someone doesn't complete their review without a valid reason (illness, family crisis, or other relevant reason they told me about, etc.), I have no other choice but to veer away from using that individual.  It would not be fair to other members who applied if I were to use someone who does not do their reviews. We follow up multiple times with roadtesters whose reviews are late. Sometimes we give them extensions of there are extenuating circumstances.

For every roadtest, I actively look for new roadtesters. I sometimes spend hours, days looking at applications. I sometimes will contact a new roadtester and ask follow up questions to them. I usually read the applications 3 times. I often select new members/new roadtesters, but sometimes they don't reply to my notification email which is used verify they are indeed available to do the review and their mailing address is correct. Shipping globally can be expensive, so we need the address verification to ensure we don't send to an incorrect address.

Yes, I use Top members but not because they are Top members alone. Top members complete their reviews; they have a passion for technology; they are knowledgeable and all of them are up for a challenge and enrich our community. Before selecting anyone, I look to see if people have completed all their reviews. If they haven't and they have not notified me about their obstacle, I try not to select them because I don't want to overload them; most of our roadtesters are employed in some form.

Why aren't some members applying for roadtests? Well, some people are unavailable or don't have the need for the product or they have roadtested previously a similar product and want to give someone a chance to get a product. This is fine. All of us have busy lives and they have other priorities.

I have gotten some great new roadtesters over the course of 2022. I really appreciate all element14 members' participation in the roadtest program. It's a great learning experience.  I will say that when I took over the roadtest program in 2016, the review completion rate was about 15%. Over the years, I put policies in place and evolved the application process. The last time I audited the roadtests, I think overall it was in the upper 80s. A good number of roadtests have 100% completion. 

I know you have participated in the program. And I appreciate your participation,

Randall Scasny

RoadTest Program Manager

This topic is of course of crucial interest to rscasny and he periodically shares a few statistics. He has done a great job of structuring the program to provide wide access to the program while ensuring the maximum completion rate. Some risks need to be taken on unproven members and they don't always pan out, but the alternative of only using the same testers all the time is far worse.

If you are applying, it is very useful to include some demonstrated proof that you would complete the road test. I don't think it is top member specific, but naturally a demonstrated track record of completing road tests, or at least blog posts, is considered in choosing some road testers.