Roadtesting stats

It would be interesting to know the remarkable people in the category

12: 1

13: 1

14: 2

15: 0

16: 0

17: 1

18: 1

Sure, here they are:

1: 359

2: 68

3: 32

4: 11 ['dixon415', 'ralphjy', 'Fred27', 'waelect', 'ninjatrent', 'ciorga', 'clem57', 'greenben214', 'tayken', 'manlong', 'bhaveshj']

5: 9 ['gordonmx', 'packetgeek', 'jkutzsch', 'jancumps', 'WarrenW', 'matfur92', 'Workshopshed', 'violet', 'rwslaugh']

6: 10 ['Kilohercas', 'mcb1', 'michaelwylie', 'Nitin_Bhaskar', 'geralds', 'sakthi.1260', 'DAB', 'mconners', 'dwinhold', 'refaqtor']

7: 7 ['gam3t3ch', 'balearicdynamics', 'nermash', 'ljakes', 'Instructorman', 'ss_shrenik', 'gregoryfenton']

8: 2 ['s2000gt', 'jdlui']

9: 0

10: 2 ['MARK2011', 'fvan']

11: 3 ['gsgill112', 'lui_gough', 'peteroakes']

12: 1 ['dougw']

13: 1 ['kas.lewis']

14: 2 ['gpolder', 'ipv1']

15: 0

16: 0

17: 1 ['hlipka']

18: 1 ['shabaz']

Hey neuromodulator

Interested as to how you got these stats. I've just run a quick query on the database and can confirm your stats are mostly correct with a few noteworthy exceptions (like Shabaz has 19, not 18)

I'm under general instructions not to post stats to the site, but in this case with them already been posted I don't see any harm in posting. I thought I'd share how your counts compare with a database count. I've disregarded all rows with a count less than 6 for brevity. This count is absolute - but includes hidden areas. The presence of our own in the list proves that - that's one of our testers and those reviews are hidden from view.

From my personal perspective, I find it interesting to investigate the people who apply a lot but do not get to review often (i.e. app count much greater than review count), or the people who apply rarely but seem to get to review every time they apply (i.e. app count is similar or just higher than review count), but that's because of my perspective as a coder.

The way I got the stats was through automating the process of downloading the data that can be found under previously posted link. More specifically a json file can be downloaded with 50 entries, so after 20 requests I downloaded all published reviews.

Yes, Shabaz posted his latests review after I gathered the data. I'm counting posted roadtest reviews, any winner that did not write a review is not counted at all. One of the things that I wanted to count was the number of won roadtests as opposed to just the number of published reviews (the stats I'm showing). One my hypothesis is that there are a couple of factors besides the application itself that are being weighted for the selection of the roadtesters. I want to know what these factors are, and want to know how much they increase or decrease the chances of winning a roadtest. For instance I've read that rscanny recommends new members, but on the other side there is also a higher chance that they don't deliver a proper roadtest compared to roadtesters with proved trajectory, so maybe it still is better to have many reviews under the belt than none at all. Same when it comes to points, how important are they for the selection process? What about having experience previously reviewing similar items, does it increase chances of being selected again? I'm sure this also depends a lot on the companies, but there probably is also some selection criteria tendency.

Yeah measuring the effectivity of applications would be interesting too, I don't think applications are public, so that's not a stat I could compute. In neuroscience we do a lot of this kind of stuff, just with different kind of data. We have tons of data (most of the times, incomplete, corrupted, or of bad quality) and we have to figure out if there are any hidden patterns or relationships, and also figure out their possible underlying cause.

neuromodulator  wrote:

One my hypothesis is that there are a couple of factors besides the application itself that are being weighted for the selection of the roadtesters.

The selection process has been described before - Randall picks some (and has described his criteria) and then the final choice is down to the supplier. I have no doubt that Randall does exactly as he says, although of course nobody is free of the many cognitive biases that affect us all no matter how hard we try.

What might be more useful - especially for the E14 staff - is to determine which members who have yet to prove themselves will actually complete their road tests to the satisfaction of the suppliers. That's the tricky one.

Interesting data and testing!

I just want to say that I do look for new roadtesters for every one. It's a challenge because many times new roadtesters write single-sentence applications, which I just cannot use; not enough information to determine whether they can do the testing..

The application needs to be good enough so I can tell whether the person is qualified, confident and committed.

Randall Scasny

RoadTest Program Manager

I saw days ago a post of yours in which you needed more five people to an RT and you also explained about NOT writing a single line application, but explain how it's going to be done and what's going to be done (I'm sorry I don't remember where is it). Your discussion is relevant. Wouldn't it be useful to pop up that warn as soon as I click the enroll button?

I always thought one single line wouldn't win, but after read that I was sure.

I was going to talk about that in one of my future conversations with the RoadTester group, but your question made me think to do it now. Our developers improved the functionality so I can modify the roadtest application somewhat with a template. You will now see the following message once you click the enroll button and scroll to the comments section of the form:

This seems to be working; I have been getting better applications in the past week.

Perfect,

It wouldn't be better without that STOP sign. I hope applications improve and those who are really committed get right.

I can remove the stop sign.

Randall

No, on the contrary. It's good. No one will skip reading.

I liked

sure, I also am very curious to that.

For myself I also expect to end up high there, mainly due to 3GHz Spectrum Analyzer - R&S FPC1000 - Review

sure, I also am very curious to that.

For myself I also expect to end up high there, mainly due to 3GHz Spectrum Analyzer - R&S FPC1000 - Review