2 Roadtests, 4 Reviewers, 0 Reviews: When A RoadTester Does Not Complete a Review

Unfortunately you are going to get some people like that. I don't envy your job having to pick road testers. It must be tricky trying to balance the requirements of the suppliers (who are the ones that count, really) but also trying to keep the community on side. I can tell you how I'd approach it, but obviously I'm not aware of other constraints and restrictions you're under.

I'd let people prove themselves on a simpler, cheaper item before letting them lose on anything bigger. Let them know that too. If a shiny desirable piece of kit has "Must have completed 2 good roadtests" as criteria maybe that might motivate people to put some effort into the smaller ones. Whilst I can understand how some people might think some members get to do all the good stuff, it's the suppliers who are paying for this to get done. In my opinions we road testers get the better end of the deal. Gough Lui is a great example. [I hope you don't mind me mentioning you by name, Gough.] He may roadtest loads of cool things and be regularly picked, but you can't argue with the results. He comes through with a great road test every time. A few other familiar names fall into that category too. I'd certainly pick someone like that over an unproven new member for the bigger stuff.

I also feel that saying "free" too much is a bad thing. I see it a lot if the road tests are promoted on twitter for instance. I suspect that's a good way to get more applicants, but are they the ones you want? Are they the ones who will complete them. Sure, the small print says you need to do the roadtest, but the freeloader types are probably still hearing "free". (Extra bonus - less applications to trawl though. That Azure Sphere one will certainly take you a while!) As far as I'm concerned it's a trade - some hardware for some time and effort. Maybe it legally has to remain an informal deal, or it has to technically be free, but it's a agreement as far as I'm concerned. (I tend to get my road tests done quickly. I realise I feel I'm in debt until it's done. It's like a friend has lent you some cash when you forgot your wallet - and you want to make sure you pay them back.)

Anyway, the road test program is a great asset for E14 and improving all the time. We all appreciate the great job you're doing. I hope it continues and continues to improve.

It's astonishing yet tiresome dealing with unreliable+irresponsible/unprofessional people. Sorry you have to experience it : (

Going through the review process helps in learning stuff, building a portfolio of experience, and improves communication skills no end, as we've probably all observed in each other.

Sadly, that is the outcome I predicted several years ago.

If you want to satisfy the vendors, you have to use trusted road testers.

You can always allocate a couple of inexpensive gear tests to newbies so that they can prove they belong on the trusted list, but you cannot risk expensive gear to newbies.

DAB

No doubt there are some individuals who want a chance to get their hands on free shite free stuff. Professional members that frequent this site realize there is no free. You may note I am one of the first people to jump on messages that are looking for answers without providing sufficient details. The free goods whether they are product or answers are an exchange for an investment commitment. The old adage your word is your bond doesn't always apply. I appreciate things come up that may inhibit or delay completing a RoadTest. That is when you are "required", I used the word specifically, to send a note outlining your failings and how you will correct the situation. I know I don't like writing those letters!

I do hope the process can retain some mechanism for proving new entrants. New people are required to evolve and expand the content. I write for pleasure and have the bonus of getting paid for some of my scribbles. I have a writing style that has change little over the years. I have a formula that works but I am cognitive of the understanding some change is necessary to stay fresh and expand your audience. I try to incorporate change to improve the product, I find it hard but a must. New participants are needed have to evolution. Giving up is not an solution.

I'm sitting here thinking, maybe an essay on why you would be a good candidate might be a solution. Then I realize JHC<talking the judeo-christian gods name in vain if translated>, someone has to read the dribble and that someone is you.

I worked in a Junior hockey program that gave the university eligible players $1000 if they submit an essay on how hockey impacted their lives. One of twenty four players on the team could win the free money. Only half of the players were in university so that improved the odds. No one, I mean no one, submitted an essay before the deadline. I know if I was playing hockey back when christ was a cowboy, I could have used the $1K.

I ask you to find some mechanism to enable new participation but I accept the process has to be efficient in order to be effective.

Sean

I hope that the two roadtests were for relatively inexpensive products.

Sadly, you are bound to get people who see their opportunity to get something for nothing but, as you have said before, if you don't take some sort of risk, you'll never get new roadtesters. This time last year, you took a chance on me as I had not come across the element14 community before and I am glad you did. I delivered that first roadtest and have done several others since then. The benefits for me have been way more than the "free" products - I have learned so much, not just from the stuff I have tested, but from reading the reviews that others have written, and people's blogs and design challenges. It is a real community, albeit virtual.

I'm guessing that you already tend to go for "tried and tested" roadtesters for the expensive gear, which is what other people are suggesting. I see it as an apprenticeship in a way - build up your skills, experience and reputation before taking on the bigger challenges.

Randall,

I understand the desire to have new blood involved in Roadtests, but maybe it is should be necessary for Roadtesters to have participated in the community first, by either competing in other contests or blogging prior to being approved to participating in a Roadtest.  But not matter where you set these thresholds for participation, some people will game the system and renege on their promise to complete the necessary review.  It could be the allure of something for nothing, or some other personality flaw, that would allow a person to do this.  I just don't know, but you do have difficult choices to make and from where I stand, I believe that you have done the best job possible of fairly assigning these opportunities to wide section of this community.

Thanks for all that you do!

Gene

I think maybe roadtesters could be divided into tiers, lets say 4 levels:

tier 0: Is the level of any new member.

tier 1: Is reached after writing a good blog post (ideally a review), this would show that the member can actually write something for the community.

tier 2: is reached for roadtesting an item that costs <100 USD

tier 3: is reached for roadtesting an item that costs <1000 USD

Something like this would probably increase the number of written reviews, not to 100% but probably closer.

Randall,

I think the program is working quite well the way you have shaped it so far, and this is definitely credit to you It is frustrating that, despite all the efforts and the good will placed to try keeping the program as open to newcomers as possible, you get this kind of response. But, as pointed out in other discussions on the topic, I think those cases of "missed reviews" do not detract from the success of the program, as it is unrealistic to expect a 100% return for all the roadtests.

For securing a sustained success for the roadtest program, it is important it is perceived as an open program, hence I believe it is vital that new blood is continuously brought on board. As I already had chance to explain, I fear that making the application process too hard or convoluted, or introducing a sort of "rite of passage" is a risk: lets not forget that many potential roadtesters hold a full time job, and need to be attracted to the program somehow. If they feel they need to invest too much time, they might just not bother trying at all.

As I said, I feel the program is working quite well as it is, and your statistics seem to confirm that, so, ultimately, I think you should resist any urge to change the application process, especially if such change could involuntarily end up creating a preferential route for the experienced roadtesters. And if the only good applications you receive are coming from the experienced roadtesters, perhaps I would still try to contact a few newcomers, and ask if they can expand on their application. If they show interest and commitment, then you could decide if any of them is worth the risk.

Here obviously I'm not talking about the people that are out to game the system: no matter how easy or hard you make the application process, there will always be people that will play it to their advantage, so trying to shape the application process to weed-out such people ends up being  counterproductive for all the rest of genuine applicants. All you can do is make the suppliers aware of the risk (they probably already know that anyway).

Thank you for all the hardwork you put to make the roadtest program such a great one!

Fabio

That is a shame that you took a shot to appease someone but then get burned with no RoadTest reviews.  I'm curious, was the person that had complained about only experienced RoadTesters getting selected one of the 4 that you took a chance with? 

I'm sure there are good folks that are either new to the site or had seen the RoadTest posted on something like LinkedIn or another site who are more than willing to produce a good review, yet they may not get a shot now due to the actions of the 4 you mention.  Outside of doing it like the hotels do by requiring a RoadTester to supply a credit card number before receiving the product, there must be another means to screen a potential RoadTester.   Perhaps requiring them to have participated in something like a Project14 contest or have posted a certain amount of content on the site, or a reference to postings elsewhere, before being eligible? 

I kind of like the idea that a new member could or should publish a road test/blog about something they already have to establish some credibility. This avenue is already available of course, but it isn't mandatory. If it were required for some road tests that a previous road test had to be published, it wouldn't be that big of a barrier for new members, and it could be a nice boost to site content. There is usually plenty of time between a road test announcement and the application deadline to publish a road test. If this criterion was trialed on half the road tests, it would be easy to see if it worked better than the current requirements in terms of number of applicants (deterrent factor) and successful completion of road tests.

How many currently successful road testers would have been keen enough to publish an un-sponsored road test prior to applying for a sponsored road test?

It is a tricky question - obviously for the few members who already publish un-sponsored road tests it is a no-brainer, and with what I know now, I wouldn't hesitate to publish an un-sponsored road test, but back when I first started I'm pretty sure I would be wondering if all this work would actually pay off. I guess it is already the case that people wonder if the effort of writing an application is going to pay off, and this is even more effort. When you first start publishing it is not clear what all the benefits might be, so the real road test equipment may be the only motive for a new applicant. If it is considered to be a lottery with low probability of "winning" it won't appeal to everyone.