Should you have to pay to opt-in to a RoadTest ?

Hi  !

I love your sentiment and see where you are going and I must agree.

People who will pay for it - I'm not discussing the amount - will more likely do the road test than those who don't.  Still, most of the time, the products from the road test are more valuable than the price one would pay for it, making it desirable to still pay less for a "freebie" and still don't do the road test !

Just to put it out there, a alternate solution would be to allow to participate in the road tests just Element14's members that are above certain points - this way is a certain that those members are in for the real thing - since they are members for some time and really participate in the community - and not just for the free stuff. This would exclude some, but this way those who're excluded will participate more for the change to participate in the road test.

If you fell that this will not work, maybe just do it for some products - the more expensive ones .

Well, just my two cents.

I love the road test program on e14 as is. That said, if a deposit system is needed to ensure more participation, it wouldn't deter me from writing road test applications. Neither would a pay-to-play scheme. I would always submit a blog if I sign up for it, regardless of other "incentives". I find it hard to figure out how to ensure others, who don't operate this way, follow through - their lack of integrity would not seem to have a simple fix. When avarice trumps integrity, the system can get played from every angle. Obviously, if you implement a system that makes scamming impossible, it would also make it unpalatable for many diligent members. I can't think of a combination of incentives and deterrents that would change somebody's value system. Maybe if they had to have their mother vouch for them ... 

No... well, um, maybe.  Shining the light on the freeloader problem does give one pause.  Perhaps an escrow payment of an appropriate amount would work.  Do what you signed-up for and the money is returned.  Otherwise, you forfeit it.  Might work.

Hello!

As a new member I am still amazed by the generosity of e14 - valuable equipment is given for writing short articles. On the other side I understand that returning tested items after the test is costly and problematic - item is not in "new" condition, can be damaged, so reselling it by E14 is not so simple.

So - how to limit the chance of fraud in the process? I think that pay-in scheme (when pay-in amount would be proportional to item's price) would make harm to the program - how many good reviews are you getting now from - for example - students or people based in lower-income countries? Those would have more difficulty in participating, even if up-front charge would be eventually returned.

But - maybe some reputation system would work? E14 organizes many challenges, so maybe people wanting to do road-test should first prove themselves in competitions not requiring up-front investment from E14? So - maybe something like:

- complete some quizzes in area connected with tested item (to check if somebody is capable of testing given item - some tests are for highly specialized hardware - and has invested some time into the account)

- optionally: post something in Project14 (that will check writing skills also) - or (as some people that would be great review-writers would prefer not to create their own projects) participate in another road-test (but for low value item)

Maybe separating road-tests into different levels: cheap items for all, expensive for experienced/proven review-writers? And put many more road-tests for low value items (so the users would have a chance to prove themselves and participate in testing of more valuable items)?

As a side note - I am not participating in road-test program for now - I think that writing good review for items I have seen in ongoing road-tests is beyond my skill today, so please take my opinion with a grain of salt...

As a RoadTest veteran with 34 completed (and on-time) RoadTests since my first in 2013, I’ll offer my 2c, as the topic of people running away with equipment and/or producing sub-standard reviews seems to be a recurring one, albeit somewhat less often since rscasny  took the helm.

The RoadTest program is perhaps one of the biggest draws to element14 and I think the publicity does not shy away from the fact the program is “free” to participants. This draws in people of all levels and backgrounds, which is perhaps important given the wide variety of things that may be RoadTested. The choice of RoadTesters based on a comprehensive application process is intended to ensure the best applicants receive the opportunity and the strategy that @rscasny has of picking a mix of both known, experienced RoadTesters and new RoadTesters has been carefully refined over the years to ensure the pool of good RoadTesters can grow over time. To some extent, the Reliable RoadTester badge had been introduced as part of this as well and the insistence that RoadTesters self-identify if they have previously failed to deliver on-time and the clause that they can be excluded based on failure-to-deliver.

When I joined, the process was much more ad-hoc and applications varied wildly in quality and content. Delivery of reviews was very hit-and-miss, quality was much the same. Admittedly, some of my earlier reviews definitely don’t measure up to the quality of my latest pieces of work. I spent a lot of time writing a blog of my thoughts about the RoadTest program back in 2020 (of which I am extremely grateful for).

I think the first issue (which has previously been identified and discussed ad-nauseum) really boils down to attitude and value. Something given away for free is often valued less than one which is paid for. Advertising RoadTests as “free” may give the perception that it is without obligation, which is untrue. Likewise, it may create the perception that reviewers are merely volunteering their time to produce a review. I’ve always taken the approach of this being a professional engagement – goods in exchange for review in a fixed timeframe. Participants should really be put into the frame of this as a “serious job” delivering real-world insights rather than “volunteering their time”. Sometimes things do legitimately get in the way, other times things go awry, and I feel flexibility here is a good thing too, but something to be considered on a case-by-case basis.

If you would have asked me to pay money to review an item back in 2013 when I started, I would have said no. It’s not because I don’t want to pay, it’s that I would have never had the means to pay! As a student with no credit card … it’s just not a possibility. Similarly, the logistics of handling money back and forth, especially with currency conversions potentially happening (which means losses) is a drama and a half. Imagine getting charged a 3% fee + their conversion margin both ways! Yikes!

But if you ask me now … I can’t say that I’m entirely opposed to it provided the deposit is reasonable and we can avoid “losing” money in the process of the refund. I deliver my reviews on-time and to a high standard, and have nothing to lose in that instance, provided you’re willing to go the distance to do it. But as someone else has rightly pointed out – if it’s cheaper than buying the equipment, there’s nothing to stop them from running off with the unit anyway. I still think it is a legitimate concern that by doing so, you will be locking out people like the 2013-version of me … and making some difficult-to-recruit RoadTests even more difficult perhaps.

I think the release condition for the deposit is an issue as well. Some very poor reviews have been delivered just to fulfil the condition of delivering “a review”, but I would be disheartened to see a good piece of equipment go to a member that delivers a two-paragraph, all-text review with no care that demonstrates nothing more than what is on a datasheet. The arbiter of what is sufficient effort for two months may need to refer to the original proposal and agree it meets a certain amount of promised goals to be able to release the deposit. Perhaps proposals should be framed as key deliverables instead.

Because of the administrative overhead, perhaps it is good to restrict this to higher-value items. I wonder if this would be problematic from a taxation point of view as well. Perhaps those with greater than a given number of completed RoadTests could escape paying entirely – although this may be considered unfair by others, perhaps if they have delivered lower-value reviews a certain number of times, they can join the upper-bracket of “deposit-free” RoadTesters. I think it makes sense if deposits are taken, that they be taken on-shipment or selection, rather than on application, again for administrative reasons.

But in the end, I suppose this is just a difficult thing to enforce. Perhaps there is no 100% guarantee in life – even good people may fail from time to time as well. But I feel that it’s an interesting conversation cstanton as having money in the form of a deposit definitely does mean that the reviewer has something at-stake and an incentive to complete. But in return, I feel this may also impact the (perceived, or actual) impartiality of the reviews. If a user feels their money is at risk if they don’t deliver a satisfying (positive) review, then this is only going to jeopardise the process and the trust the readers have in the reviews that are posted.

Maybe for certain products, no deposit is necessary and instead, the licensing system can be exploited to achieve this function. I’ve had gear lent to me from T&M companies that come with time-locked licenses that will refuse to function once your lease period is over (or revert to basic functionality only). Perhaps for such capable equipment, they be shipped with time-locked licenses set to expire one month post-RoadTest. Providing they do complete a satisfactory review, the permanent license keys will be provided to those members, and if they don’t, then the gear is either rendered impaired/useless and then they have an incentive to return it to the manufacturer (or you) to get it re-licensed for another go. This won’t work for all equipment, but exchanging serial codes is probably easier than exchanging money for many members, and it doesn’t restrict functionality during the review period. Won’t stop people parting out the machine for spares … unfortunately …

But perhaps the trust-the-application model we have at the moment is still the simplest, from an administrative standpoint. If the losses are at an acceptable level, perhaps that’s just the “cost of doing business”. I can’t see any changes making things necessarily fairer for all, but then again, there were heated debates by disgruntled RoadTest applicants that were not selected as they felt they were being discriminated against. If we formalise the establishment of a “high tier” RoadTester, I fear we may be inviting this debate back into the fore and reigniting some very unpleasant discussions (as someone who was the target of a few members/former-members).

As far as I know, the companies may also have some say in this … perhaps different companies will accept different levels of risk in non-review. Or perhaps some companies will have the time to make an informed choice by looking at prior work and statistics in terms of members and their delivery track record …

- Gough

I liked your ideas, I agree with you.

I think it might put off a lot of people, especially students and new grads, but still, sometimes raising something like this can make people brainstorm : ) 

Maybe the issue can be more generalized, into all manner of things where committment should be shown. For instance, the promise that one is not a spammer, and the promise to be helpful, etc., are community activities or qualities which equally benefit from commitment.

There's some film where just after WW2 a banker goes back to his job, and he approves a loan to a customer who happens to be another WW2 veteran.

The head of the bank doesn't like it, and questions what was the customer's collateral.

The WW2 banker then says that the customer had no physical property or valuables, but that he approved the loan based on judgement, because he saw collateral in the guy himself, because during WW2 they had no property other than what was inside them, and that's what they learned to recognise. It was like a peer-to-peer approval.

Applying that to the current era where there's a ton of spam coming this way with AI, I wonder if forums could be opt-in based on such a scheme, for instance, any new member would need to be approved by someone else (perhaps the approval request goes to three randomly chosen known real users, and if any one of them approves it then the application is accepted. The collateral would be whatever the person typed in the application (the application could even have boxes to fill in to demonstrate genuine interest.

Granted that doesn't help directly in any way whatsoever for RoadTests, but just trying to think outside the box, how people show their inner collateral as it were! : )

You need to create a rating system. Take into account how long the participant has been registered in the community, how actively he participates in it, publishes articles, participates in road tests and in Project14, etc.

Indeed, there was a phase when applications had questions about whether someone had asked questions in the forum, attended webinars, did any Project14, etc. I think they got cut out of the application form because it was making too long. But the idea had been floated and taken on in the past. Perhaps it would be more honest if the staff checked user's profiles for their participation rather than relying on users to self-report - it is more work for them, perhaps they already do this?

- Gough

I like the idea of some kind of honour system.

Being chosen as a roadtester is indeed an honour, but it also gives the tester a chance to raise their own profile - and that has to be worth something.