How are reviewers selected?

So, I thought I'd weigh in on this topic because I was one of the folks selected to review the Arduino Uno and there appears to be alot of comment/critism since that particular Road Test was closed and folks selected. I'm not sure if that was the catalyst for Michael's post or not, but there seems to be a  lot more comments since that road test closed. On that basis, I thought I would provide my thoughts below.

Let me first start by saying that I was no where near the beginning in signing up. I say that because I believe some folks thought the first few applicants were selected. Now, I do not know why I was selected other than I put some careful thought into my application (not that others didn't), offered a perspective and method of review that I belieive will be beneficial to both the sponsors and community at large. I think my perspective and offer was perhaps somewhat different and maybe that criteria was the basis for myself being chosen. Not sure, but I will say that my plan is to offer a careful, thoughtful and detailed review. 

I am thankful for being chosen, have not had an opportunity to review before (although I was selected once - more on that below), and don't expect to in the future unless I have something I feel I can offer to a sponsor and the community (and be lucky to be selected again). I have however thought, geez, that would be cool - but honestly didn't think I would have the time or commitment to do a good job. I did not apply to those road tests. I don't apply to every one. I have applied to a few others however and not been selected. I assumed it was because the folks who were sponsoring the road test were selecting reviewers they thought could offer a certain perspective on their kit. I'm OK with that as they are providing the kits for free! If they select a 'random' reviewer or reviewers that offer a certain perspective, or whatever, the kits are free and I suppose it should be up to them or Element 14 to decide.

I feel it's a real positive just to have these folks/sponsors step up and offer free kits. Kudo's to Element 14 to bring this together. I also feel by not necessarily posting a bullet list of criteria you get a wider scope of applications and allow the sponsors to select folks that perhaps match their goals. I am personally OK with that since the kits are free. I can just imagine if you said, we are selecting reviewers based on 'a', 'b', 'c' and 'd' criteria that you'd get 200 applications saying how they would meet 'a', 'b', 'c' and 'd' with little differentiation. I have been disappointed in the past and thought all my applications had merit. I do however believe that if you are selected, in some way, by applying you agree that in exchange for a 'free' kit you have an obligation to write a review. Whether it is good and detailed I suppose is up to the readers of the review. I suppose that's what a comment and rating section are for.

Michael's 3rd and 4th points are good ones. I would offer that I hope it does come into consideration if potential reviewers have reviewed previously. I'm not sure about a 'black' list, but certainly a poorly rated review by the peers in this community should have an impact on the suitability for future selections. I would definitely agree with there. However .... 

I previously offered that I was selected but did not post a review for a prior road test (Freescales Tower). Like allot of things; what you might see doesn't reveal the entire story. I was 'selected' and continue to be listed as 'review pending', however, I did not actually receive the kit.  I attempted on several occasion to determine where or why but assumed the kit just never got shipped. This was around the time Alistar was leaving and perhaps this road test got caught in the middle. I'm not sure but you'll note in that road test others are listed as 'review pending' as well. From my perspective I would say they never received the kits either although I can't say for sure. In my mind it does suggest there are perhaps instances where 'road test reviewers' may not actually be in a position to review. Should that factor? Perhaps, but I think my point is that just because your listed, doesn't mean you actually received the kit and are in a position to write the review. I'm not trying to excuse people, just offering a perspective based on experience.  I'm sure there are people that apply to get kits just because they are free and never intend or bother to post a review, but I am also sure (because I was one) that sometimes people don't actually end up with the kits.

I don't intend that as a criticism of the Road Test crew at the moment, just a statement on a perspective based on experience. The folks running the road test list now seem to be very diligent at getting things right, or at least trying there best, and supporting the community. I think that's great and look forward to contributing and offering a balanced review of the road test(s) I have been selected for. Hopefully my review(s) will meet the standard of 'good' and 'detailed'. I'll look forward to comments on my review. Until then, I thought I would pass along my perspective as a selected road tester.

So, I thought I'd weigh in on this topic because I was one of the folks selected to review the Arduino Uno and there appears to be alot of comment/critism since that particular Road Test was closed and folks selected. I'm not sure if that was the catalyst for Michael's post or not, but there seems to be a  lot more comments since that road test closed. On that basis, I thought I would provide my thoughts below.

Let me first start by saying that I was no where near the beginning in signing up. I say that because I believe some folks thought the first few applicants were selected. Now, I do not know why I was selected other than I put some careful thought into my application (not that others didn't), offered a perspective and method of review that I belieive will be beneficial to both the sponsors and community at large. I think my perspective and offer was perhaps somewhat different and maybe that criteria was the basis for myself being chosen. Not sure, but I will say that my plan is to offer a careful, thoughtful and detailed review. 

I am thankful for being chosen, have not had an opportunity to review before (although I was selected once - more on that below), and don't expect to in the future unless I have something I feel I can offer to a sponsor and the community (and be lucky to be selected again). I have however thought, geez, that would be cool - but honestly didn't think I would have the time or commitment to do a good job. I did not apply to those road tests. I don't apply to every one. I have applied to a few others however and not been selected. I assumed it was because the folks who were sponsoring the road test were selecting reviewers they thought could offer a certain perspective on their kit. I'm OK with that as they are providing the kits for free! If they select a 'random' reviewer or reviewers that offer a certain perspective, or whatever, the kits are free and I suppose it should be up to them or Element 14 to decide.

I feel it's a real positive just to have these folks/sponsors step up and offer free kits. Kudo's to Element 14 to bring this together. I also feel by not necessarily posting a bullet list of criteria you get a wider scope of applications and allow the sponsors to select folks that perhaps match their goals. I am personally OK with that since the kits are free. I can just imagine if you said, we are selecting reviewers based on 'a', 'b', 'c' and 'd' criteria that you'd get 200 applications saying how they would meet 'a', 'b', 'c' and 'd' with little differentiation. I have been disappointed in the past and thought all my applications had merit. I do however believe that if you are selected, in some way, by applying you agree that in exchange for a 'free' kit you have an obligation to write a review. Whether it is good and detailed I suppose is up to the readers of the review. I suppose that's what a comment and rating section are for.

Michael's 3rd and 4th points are good ones. I would offer that I hope it does come into consideration if potential reviewers have reviewed previously. I'm not sure about a 'black' list, but certainly a poorly rated review by the peers in this community should have an impact on the suitability for future selections. I would definitely agree with there. However .... 

I previously offered that I was selected but did not post a review for a prior road test (Freescales Tower). Like allot of things; what you might see doesn't reveal the entire story. I was 'selected' and continue to be listed as 'review pending', however, I did not actually receive the kit.  I attempted on several occasion to determine where or why but assumed the kit just never got shipped. This was around the time Alistar was leaving and perhaps this road test got caught in the middle. I'm not sure but you'll note in that road test others are listed as 'review pending' as well. From my perspective I would say they never received the kits either although I can't say for sure. In my mind it does suggest there are perhaps instances where 'road test reviewers' may not actually be in a position to review. Should that factor? Perhaps, but I think my point is that just because your listed, doesn't mean you actually received the kit and are in a position to write the review. I'm not trying to excuse people, just offering a perspective based on experience.  I'm sure there are people that apply to get kits just because they are free and never intend or bother to post a review, but I am also sure (because I was one) that sometimes people don't actually end up with the kits.

I don't intend that as a criticism of the Road Test crew at the moment, just a statement on a perspective based on experience. The folks running the road test list now seem to be very diligent at getting things right, or at least trying there best, and supporting the community. I think that's great and look forward to contributing and offering a balanced review of the road test(s) I have been selected for. Hopefully my review(s) will meet the standard of 'good' and 'detailed'. I'll look forward to comments on my review. Until then, I thought I would pass along my perspective as a selected road tester.

Hi Mookiebud,

I think the answer for the many "Review Pending" statuses is that was an E14 CMS error. The only credible winner list is the one posted below the test.

It happened to me too: on the Freescale MC9S08LL I'm still with that "Review Pending" status but when I asked I got a reply that I was not on the winner list. It's very easy to get some errors on a huge and growing CMS with many functions added gradually.