Actions
Forum Thread Details
  • Replies 54 replies
  • Subscribers 2761 subscribers
  • Views 7087 views
  • Users 0 members are here
Related

2 Roadtests, 4 Reviewers, 0 Reviews: When A RoadTester Does Not Complete a Review

rscasny

Recently, I selected roadtest applicants who were both new roadtesters  and have not participated much on the community, primarily because I have had some other members tell me that I am using experienced roadtesters too much. (This is never my intent; I select the best applicants. When I roadtest writes only a 1-sentence application, I don't have much to go on and these folks usually don't get selected.)

 

I had to report roadtest results to 2 sponsors. Well, the results for these 2 roadtests were not great: 4 roadtesters haven't done their reviews, they haven't logged in for 3 to 4 months, and they have not responded to all our follow ups that we do for every roadtest. Before I selected these 4, I contacted them and they agreed to commit to delivering the review in 60 days.

 

Now, 2 roadtests is a fraction of all roadtests. For many roadtests we get 100% compliance. I'd say overall we are getting a compliance rate in the upper80s/lower90s, which is good and a far cry from where we were 2 years ago when we were at the sub-20% level.

 

Speaking for the sponsor and myself, we understand that in some situations people will be unable to complete the review. After all, people do get ill and emergencies do arise in everyone's lives, so I never expect to have 100% compliance for every roadtest. I recall someone who had a tree fall on his house in an ice storm. That's a major tragedy in my book. Of course I would not expect a review. But these instances are atypical and very few compared to all roadtests and roadtesters I deal with.

 

While we give the official roadtesters the kits and the shipping for free, someone does pay for these things. Typically, the sponsor pays for the kits and element14 pays for the shipping costs to the roadtesters. But we are more than happy to do this because we value our roadtesters' opinions and we are happy to absorb the costs when the reviews are published.

 

But the question comes to mind: what happens when a roadtester doesn't complete the review:

 

1. The sponsor does not receive all the feedback he is seeking when he enrolled in the roadtest program as a sponsor.

2. Someone who does not complete the review prevents someone who could have done the review from doing it.

3. It prevents our members from learning about the product.

4. It damages the credibility of the program.

5. The delinquent roadtesters prevent themselves for being considered for a high-priced product. If I can't reliably use someone for a $35 US roadtest, how can I expect them to do the review for a $3500 US roadtest!

6. I am placed in the embarrassing situation to explain why 4 people I selected did not do their reviews.

 

There are other reasons but 6 reasons is enough to make my point.

 

It is my responsibility to get the required results for both our sponsors. My bosses also have expectations and expect that I will solve problems. I want people to use the RoadTest program to have fun with technology,  and get some products or equipment that they may not have the budget to get on their own. But I also need mature, responsible members who are engaged in our community, demonstrate an appropriate level of professionalism, and are responsive to our follow up requests. I like to think most roadtesters are professional. I surely do not think these 4 delinquent roadtesters represent the RoadTest group.

 

But going forward, I will have to modify my approach in selection somewhat. And I probably will not hand out all the kits if I don't feel I have enough roadtesters who are knowledgeable about the product, and are mature, professional and reliable. If you were in doubt that I am very serious about people completing their reviews, I hope this discussion makes my position very clear. I apologize to all my loyal and responsbile roadtesters who do complete their reviews. But this is a community and I have to communicate my concerns as well as be open to new ideas and feedback: it's a two-way street.

 

If you can't finish the review, contact me and tell me or my colleague why. If you need a bit more time, that's fine if you need to do more testing. (Needing more time to finish your Differential Equations homework or finishing your digital electronics lab is not a reason for needing more time.) Communicate and keep us in the loop; you will find in your career that developing communication skills is as important as honing those engineering skills. Even if you received a non-working product, you are still required to post a review. Put it out to the members: "I took these readings and I believe it's faulty. Help me troubleshoot it." There are plenty of eager members who are more than happy to help.

 

I want everyone to benefit from a Roadtest. I think completed roadtests are a great place to learn. I think they are a fantastic vehicle for the new engineer who is trying to shine in their first job interview. Read our e-book on IoT careers to learn more shining in a job interview: https://www.element14.com/community/community/publications/ebooks?ICID=hp-iotcareerebook-rotational#?ICID=ebooks-downloa…

 

But you can't reap the benefits of the roadtest program if you don't do your reviews.

 

Sincerely,

 

Randall Scasny

RoadTest Program Manager

  • in reply to dougw

    I wasn't being serious ... but hey, the mis-application of statistics in everyday life seems to be a pet peeve of statisticians so I thought I'd throw it out there. In all, we do have several things going on - things can often break down when we examine very small numbers (e.g. a RoadTest with just 1 or 2 kits on offer), so applying aggregate figures would be unwise. Likewise, where there are non-equal chances of selection, that often will throw things out the window, not to mention that the pool of applicants will depend on how "interesting" a device is and the sponsor's requirements.

     

    In the end, statistics is still why some people can still rock-up to a casino and play roulette, putting everything on red with a ~47% chance, play once, win (and never come back) even though statistically, the odds were pointing towards a loss image.

     

    - Gough

  • in reply to Gough Lui

    I wasn't trying to be dangerous image It is just a simple hypothetical example illustrating that if you don't have 100% completion rates, sooner or later a lot of non-completions can happen at the same time.

    We have been speculating extensively about why these results occurred, without much real root cause information, and we theorize what can be done to eliminate these types of results. It is all interesting and possibly useful but it isn't definitive. I don't see the harm in also speculating that statistics might provide some insight and a different perspective of what happened. If it looks useful, then collecting some real data might provide some interesting statistical analysis.

  • in reply to dougw

    Some very dangerous assumptions ... you can't take an aggregate rate (i.e. 80% completions on average) and then apply it to individual RoadTesters as it's not a game of equal likelihood, especially when those who fail to deliver are disqualified and those who have delivered in the past may have greater motivation to maintain their record. If all events are independent with no biases (i.e. assuming the mix of candidates stays constant and Randall picks randomly without any "tiering") then perhaps your calculation could hold, in a rough back of the envelope kind of way. Looking at our own track records, we can see that we're holding up the high-end of the bar. Somehow I think that having zero returns on a RoadTest is highly improbable, although perhaps for kits with just one or two candidates, this may be more likely. I have some faith that some people would deliver a report just to be "the first" to do so ... they often get more views!

     

    But more likely than not, external influences may well be the dominant part of the equation:

    • New members without much experience vastly underestimate the effort, equipment needed to deliver on their proposed test plans (perhaps due to a lack of research). They may run into an insurmountable hurdle and perhaps get discouraged - maybe even during the write-up. Maybe they see someone else has already done everything they said they would, and did it even better, thus they don't deliver.
    • The product itself isn't interesting enough on its own to warrant further time investment upon meeting one of these hurdles. Or perhaps they decided to register anyway to have a go, being too optimistic and ignoring the consequences and then get cold feet or lose confidence.
    • They never receive the product, or it gets damaged in transit and they can't be bothered to chase up.
    • The product goes out much later than initially assumed and then other commitments get in the way - this has happened to me a few times but I've worked hard to make sure I get around to it. Other people may not be as lucky.
    • The product goes out at a time when people may be busy with other things (e.g. holiday periods).
    • The person chosen may not have the same sense of responsibility or may have other complicating issues (i.e. health of themselves, close family members, etc).

     

    I think that perhaps focusing on completion rate is missing the other part of the equation - review quality. It was mentioned prior that feedback from the supplier could be solicited - but I think it may even be useful just to get feedback from element14 themselves just to know what they think about our performance. Delivering no review and delivering a substandard (or blatantly incorrect) review are perhaps not that far apart on the value scale.

     

    - Gough

  • Suppose Randall has increased the probability of completed road tests to 80%. (Well done Randall) image

    This still means 1 out of 5 will not complete their road test. image

    So if there are 4 road testers, what is the probability that none will complete their road test?

    This is simple probability theory, but my probability theory is super rusty, however by my calculation this means

    1/5 x 1/5 x 1/5 x 1/5 = 1/625.

    So even if you have an 80% success rate, every 600 road tests or so you could have a road test with no completions, and it could happen at any time - the 6th road test or the 600th road test.

    This is just an example and of course many other factors can affect the rate, but the statistics alone say sometimes the whole road test will flop .... It is unfortunate but it doesn't automatically mean the wheels fell off or something went terribly wrong.

  • Hi Randall,

     

    I will put forth an experience from the opposite viewpoint.

     

    My own experience is that I was not chosen as a RoadTester on early attempts although I feel they were good submissions and topics I had familiarity with.  While not selected on my first attempt I went ahead and did my project anyway with a board I bought.  In other words I really was serious about doing it. Of the 3 persons actually selected (all of them level 1 as was I), none actually wrote up a RoadTest.  This was before you were managing the program but it actually dissuaded me from making entries for a while....

     

    I was not selected until after I had become active and in retrospect this seems reasonable. In general I believe extra weight should be given to those who are active on e14 and a statement to that effect could be included in the instructions. If the company making the product has specific requirements for testers (like current job or educational /work background) that should be stated also.  This might help RoadTesters with their expectations of being selected and give incentive to become more active.

  • in reply to mom2673

    Hi MaryAnn.

     

    Welcome to element14.

     

    We have tossed around your idea about buying the product. I'm not ready to do that, and I'm not sure how easy it would be to do in your particular system.

     

    But please apply to a roadtest that you are interested in. Write a good application. You may find yourself getting a product to review.

     

    Randall Scasny

    RoadTest Program Manager

  • in reply to Fred27

    I could not agree with you more on the Word FREE enticing reviewers who arw only there because the item is FREE. Unfortunately, it doesn't leave a lot of reassurance and trust for those of us who are new to this website and would be grateful for the chance to test out a product. Maybe you could have the new users (speaking as one myself) purchase the product then, offer a refund once their review has been completed.  I would be very willing to do that since chances are it's a product I probably wanted to buy anyway.

     

    Well, I'm just glad I stumbled across this site I'm search of information for an IT project I'm going to be undertaking soon.  You have NOT heard that last of me!

    Thanks all for a really great, informational site. I'm looking forward to learning lots from the questions I will surely have, lol.

  • I know I haven't really posted much here on this community in the past but I read this post and just wanted to provide some insight as to what I have seen in the past for companies that I have done beta testing for in the past and I feel that these methods may work here if you are looking for a method to thin out those that aren't serious about providing acurate and or honest reviews. Please see the poorly formatted list of methods if seen before:

     

    1) Company XYZ provides the ability to test a product with high value that must be returned at the end of the test for "modifications" to a new user they are told that they must provide a deposit that will be returned after the success of the beta... experienced and proven testers don't have to meet this requirement... (as a new tester I wasnt happy about the deposit method but for items that really peaked my interest I dealt with it and put the deposit on my credit card or to my PayPal...and did the test and as soon as the company received the confirmation from FedEx that the package was received the funde were promptly refunded. This I'm sure kept the dishonest from applying)

     

    2) (my favorite method) The testing company requires everyone to complete a review/test on a website that has known issues... I'm not sure how many there are but they look for people to find and report on that and then if they have done so to a good standard they are selected for future tasks... if not some feed back it provided and they must wait a month before reapplying. This helps weed out the people that tend to not provide quality results.

  • in reply to dougw

    Hello Doug,

     

    I agree with your points about matching a RT to an applicant with the right skill set required and also agree that Randall and his staff do a remarkable job with the RT program but, when a reviewer doesn't proactively seek for help -in case of issues-, hasn't produced any kind of content within the required 60-day period or even worst, doesn't respond to follow up requests I believe in the majority of the cases these all point to the same direction.

     

    Luis

  • in reply to rscasny

    I think there may be other factors that might be useful in choosing applicants.

    • one has to do with matching road tests to people who can handle them. Specifically predicting how hard a particular road test will be to complete, given a particular skill set or sometimes any skill set. I often try to predict what the response rate will be for each road test and find the predictions are significantly better than just random chance, so predictions might be useful in deciding how to stack the deck.
    • part of the application could be a requirement to estimate how many hours will be spent on the road test and blog. (separate numbers) I am sure it would be useful to   if an applicant spent some effort to explain how much time they are willing and able to commit to a road test.

    I think Randall is doing a superb job screening, filtering and encouraging applicants, but there are some road tests where it will always be problematic to get a high completion rate.

    • Part of this is due to applicants not honoring their commitments but this is not the only reason
    • When I look at a new road test, I sometimes think things like: "wow - there aren't too many members who will be able to handle this road test" or "yikes - this road test will require a scary amount of work" or "this is new and different with rudimentary support materials so it will take a lot of learning and perseverance". When I get those types of reactions, it almost always means there will be few completed road tests, partly because other experienced road testers will shy away from applying due to similar reactions. The road test may scare off knowledgeable road testers leaving only naïve applicants that rapidly discover they can't handle it.
    • Part is due to the nature of the specific equipment being road tested. The manufacturer may not have made it easy enough to do a road test.
    • Sometimes the equipment is so esoteric, the average member doesn't have a use for it and is not very motivated once they realize this.
    • Sometimes the road test is just very hard and even with the best intentions, success is not likely (for some, blogging about a failure is super hard)

    The bottom line is when there is a poor response rate, don't automatically assume it was due to poor applicant selection criteria or poor screening, there are a lot of factors that will occasionally conspire to yield a poor result. Sometimes it is even predictable before the candidates are selected.

    It is worth examining each anomaly, but some of it is just statistical reality.