How are reviewers selected?

I was wondering the same thing just never got around to posting it.  I watched to video and it says the more you interact with the community the better the odds but it alsways seems like the winners are people that have very little interaction with the community.

So, I thought I'd weigh in on this topic because I was one of the folks selected to review the Arduino Uno and there appears to be alot of comment/critism since that particular Road Test was closed and folks selected. I'm not sure if that was the catalyst for Michael's post or not, but there seems to be a  lot more comments since that road test closed. On that basis, I thought I would provide my thoughts below.

Let me first start by saying that I was no where near the beginning in signing up. I say that because I believe some folks thought the first few applicants were selected. Now, I do not know why I was selected other than I put some careful thought into my application (not that others didn't), offered a perspective and method of review that I belieive will be beneficial to both the sponsors and community at large. I think my perspective and offer was perhaps somewhat different and maybe that criteria was the basis for myself being chosen. Not sure, but I will say that my plan is to offer a careful, thoughtful and detailed review. 

I am thankful for being chosen, have not had an opportunity to review before (although I was selected once - more on that below), and don't expect to in the future unless I have something I feel I can offer to a sponsor and the community (and be lucky to be selected again). I have however thought, geez, that would be cool - but honestly didn't think I would have the time or commitment to do a good job. I did not apply to those road tests. I don't apply to every one. I have applied to a few others however and not been selected. I assumed it was because the folks who were sponsoring the road test were selecting reviewers they thought could offer a certain perspective on their kit. I'm OK with that as they are providing the kits for free! If they select a 'random' reviewer or reviewers that offer a certain perspective, or whatever, the kits are free and I suppose it should be up to them or Element 14 to decide.

I feel it's a real positive just to have these folks/sponsors step up and offer free kits. Kudo's to Element 14 to bring this together. I also feel by not necessarily posting a bullet list of criteria you get a wider scope of applications and allow the sponsors to select folks that perhaps match their goals. I am personally OK with that since the kits are free. I can just imagine if you said, we are selecting reviewers based on 'a', 'b', 'c' and 'd' criteria that you'd get 200 applications saying how they would meet 'a', 'b', 'c' and 'd' with little differentiation. I have been disappointed in the past and thought all my applications had merit. I do however believe that if you are selected, in some way, by applying you agree that in exchange for a 'free' kit you have an obligation to write a review. Whether it is good and detailed I suppose is up to the readers of the review. I suppose that's what a comment and rating section are for.

Michael's 3rd and 4th points are good ones. I would offer that I hope it does come into consideration if potential reviewers have reviewed previously. I'm not sure about a 'black' list, but certainly a poorly rated review by the peers in this community should have an impact on the suitability for future selections. I would definitely agree with there. However .... 

I previously offered that I was selected but did not post a review for a prior road test (Freescales Tower). Like allot of things; what you might see doesn't reveal the entire story. I was 'selected' and continue to be listed as 'review pending', however, I did not actually receive the kit.  I attempted on several occasion to determine where or why but assumed the kit just never got shipped. This was around the time Alistar was leaving and perhaps this road test got caught in the middle. I'm not sure but you'll note in that road test others are listed as 'review pending' as well. From my perspective I would say they never received the kits either although I can't say for sure. In my mind it does suggest there are perhaps instances where 'road test reviewers' may not actually be in a position to review. Should that factor? Perhaps, but I think my point is that just because your listed, doesn't mean you actually received the kit and are in a position to write the review. I'm not trying to excuse people, just offering a perspective based on experience.  I'm sure there are people that apply to get kits just because they are free and never intend or bother to post a review, but I am also sure (because I was one) that sometimes people don't actually end up with the kits.

I don't intend that as a criticism of the Road Test crew at the moment, just a statement on a perspective based on experience. The folks running the road test list now seem to be very diligent at getting things right, or at least trying there best, and supporting the community. I think that's great and look forward to contributing and offering a balanced review of the road test(s) I have been selected for. Hopefully my review(s) will meet the standard of 'good' and 'detailed'. I'll look forward to comments on my review. Until then, I thought I would pass along my perspective as a selected road tester.

Hi Guys.

Thanks for posting your questions about RoadTest. My team manages the process, and I'm happy to explain how we select winners.

First, it is not 100% random. We look at a few factors to narrow the list down, including--

• The reason you enter when applying. This is a critical piece for us. We find that people who really put some serious thought in describing what they plan to build with the product tend to actually come back after getting it and post their review. This is really important for us, as the rest of our community here loves to see detailed reviews.
• Has the applicant recently won another RoadTest? If so, we will probably not select them right away as we want to give others a chance to participate.
• Related to the above point, did the applicant win in the past and not post a review? That's a big strike against being selected again. Sending top products to members of our community for free and not getting a review back hinders our ability to obtain great products in the future, since our manufacturer partners also want to get detailed feedback on their products.
• Did the applicant give us his/her complete shipping information? Sounds like an obvious one, but you would be surprised. :-)
• Does the applicant work in an area of engineering that makes him/her particularly well-suited to review the product in question? This is where having a complete user profile can help make the difference.
• Is the applicant active on element14? If so, it increases the odds that he or she will follow-up after receiving their product by posting a review. We select new members, but tend to favor people who ask questions, participate in discussions, etc.

Most of our criteria are based on increasing the chance of getting a detailed product review back from our winners. The list above is culled from our experience managing RoadTest, and is designed to increase the benefits to our community by selecting winners who will follow through.

I hope this sheds some light on the process; please let me know if you have any questions.

And thanks for being such an awesome community!

Sagar Jethani

Head of Content

element14

Thanks for the reply and incite into the selection process. I was recently selected for the Agilent U1272A HH DMM and U1177A Bluetooth Adapter. I wasn't as active in the community prior to being selected but have since become much more active. I've been looking through past Road Tests and reviews and noticed the low average of reviews to number of kits that where sent out. So I was curious to know how people where selected and what happened to those who never write a review. I also found it odd that I was chosen to review the Agillent DMM. As I mention in my review it was odd because the Road Test was looking for applicants with an Android device. I didn't see that till after I had submitted my application. Since I don't have an Android device I figured I was out of the running from the start.

I don't like to apply for everything cause I don't have the skills or the experience to be a good review candidate for every item. The IMO iSmart is a good example of an item I wouldn't even know where to start in reviewing. Now that I know what your looking for I'll be even more focused on how I'll review the item and putting together a more comprehensive application on items I am able to work with.

Thanks again.

In my life long experience I have found that the main reason people ask this type of question on things that are being given away is because of the simple fact that they were not selected. Therefore by some unknown reason they feel cheated and in some cases angry. ( I've seen grown men take candy from kids at parades.)

Personally, I don't care how they are selected - the product is free, it's free to join, it's free to sign up for a road test. It might be a bit different if I had to pay a fee to enroll or even to join the web sight, but it's FREE. element14 can choose them however they dame well please.

However, I do agree that anyone selected that does not submit at least a decent review should be overlooked on future road test. I also agree that corporations that are giving the item away should have the right to hand pick those they feel will give the best review and benefit most from the product. It wouldn't make much sense to have a total beginner win a road test for a Spectrum Analyser. In this last case, an Arduino is perfect for the total beginner and I feel that more experienced people here should not have applied.

Just to note, it seems someone brings this up after every roadtest.

I'll have to agree with you R_Phonenix and it's the only reason I applied for the Arduino, since I am a beginner.  I do like the response from Sagar and now I have a better understanding of what to do and expect. 

I have been reading reviews and impressed with some of the details people give and then read some, where all they talk about is the unboxing,

I'll continue reading, learning new products, and maybe some day I'll be smart enough to review that Flux Capacitor.

thanks for the reply Sagar.  that really clears things up for us.  i agree with other people that if your selected and dont send in a review you should not have the option to test again because the odds of you repeating are high.

i also didnt post my coment earlier because im mad i didnt get selected there just isnt anything posted anywhere that really explains how the selection proccess works.  i would almost have element14 post a quick doc somewhere on the roadtest site that explains this process so people cant get mad because they are informed.

Hi Mookiebud,

I think the answer for the many "Review Pending" statuses is that was an E14 CMS error. The only credible winner list is the one posted below the test.

It happened to me too: on the Freescale MC9S08LL I'm still with that "Review Pending" status but when I asked I got a reply that I was not on the winner list. It's very easy to get some errors on a huge and growing CMS with many functions added gradually.

R_Phoenix wrote:

 

In my life long experience I have found that the main reason people ask this type of question on things that are being given away is because of the simple fact that they were not selected. Therefore by some unknown reason they feel cheated and in some cases angry. ( I've seen grown men take candy from kids at parades.)

 

Personally, I don't care how they are selected - the product is free, it's free to join, it's free to sign up for a road test. It might be a bit different if I had to pay a fee to enroll or even to join the web sight, but it's FREE. element14 can choose them however they dame well please.

 

However, I do agree that anyone selected that does not submit at least a decent review should be overlooked on future road test. I also agree that corporations that are giving the item away should have the right to hand pick those they feel will give the best review and benefit most from the product. It wouldn't make much sense to have a total beginner win a road test for a Spectrum Analyser. In this last case, an Arduino is perfect for the total beginner and I feel that more experienced people here should not have applied.

 

Just to note, it seems someone brings this up after every roadtest.

Sometimes people ask this kind of a question just to get a better incite on how the process works.

How did the Arduino Uno become a focal point here? The fact that this was posted soon after the Uno Road Test closed out has no bearing on the questions asked here. No matter when this was posted it would have came after some Road Test had ended. Please don't try to make a person look bad for being inquisitive.

It may seem to you someone asks this question after every Road Test but I can assure you it doesn't come up that often. I did a good search through the Road Test Discussions to make sure this kind of topic hadn't come up in at least the past year. I would have much rather found the answer in an old discussion then start a new duplicate one.

It's really good you have asked this, because (in contrary to what R_Phoenix said) we got an anwser from the source, which helps us actually making *good* applications (thanks Sagar!). Of course, not everybody will read this, but people really interested in making the reviews will find this post - and will benefit from reading it.

So - thanks to you, and thanks to Sagar.