Disappointed - My opinion

Hello All

As the person that does select challengers and runs Design Challenges, I am very invested in this thread. I appreciate you thinking about this on my behalf. You all have echoed conversations we have had within the e14 Team and within the Top Member circle.

To address some of your recommendations and questions, I can confirm that when checking applications we first check them for originality, and we do select the best applications based on a few sets criteria.

I personally do get disheartened when some people ghost on us... even more when their proposed application was fantastic.

I also understand that some people have better technical and practical skills than written English skills and try to keep an open mind on this when selecting challengers. Every challenger I accept represents a project I would like to see brought into the world and is most likely off the back of a convincing application.

We do track blog progress against the timeline and check in on stragglers, this is often when we spot the ones that have ghosted within a 3 weeks of launch. That is when we reach out to them, via what ever methods they have added to their applications. This 3 week period is where you could consider new challengers, but in challenges where we have a limited amount of kits, this is not an option.

We sanity check new challengers and new users but do also take a few chances, and will continue to do so, for example, 3 or 4 of the most recent design challenges have had new members as winners and many others as finisher prize winners.

These success stories are the reason I continue to take risks on new members as well as a mix of trustworthy members.

We find a lot of people are daunted by the size of design challenges so try to keep the barriers to entry a low as possible and also promote project 14 as something with easier access.

Now... lets discuss how we can weight these a little bit more to reduce drop numbers.

Originally a high quality application was seen to be enough of a deterrent to weed out most of the people testing their luck.

This no longer seems to be enough.

As for people building up reputation before being invited to join a challenge, my stance is that I will continue to take applications from new and old members, but I think there is a need for a skype call with applicants that could be classed as higher risk.

I will update the application page to help be more transparent in regards to what is required of a challenger.

I want to confirm my stance of keeping the barrier to entry a low as possible to new members but also confirm the intention of those joining.

Thank you all for your input and suggestions, please keep it coming!

-Phil Hutchinson

Great post Phil.

Are there any analytics on length of membership, participation and failure to complete? I'm thinking of predictors that your team can use to determine risk.

If the failures can be linked to membership length, then maybe a quarantine period with activity monitor is needed to ready potential candidates.

You invest in removing those individuals that are gaming the process. Some will over promise, under deliver and fail. That is an known learning path. The gamers (i.e. looking to manipulate the system) are the primary target.

Your teams task is not much different than the moderators for RoadTests. What is an acceptable failure to complete rate? Once you have reached that point any other effort provides diminished returns.

Sean

Hey Sean,

Ooof this is an old thread to suddenly resurrect! 

In response to some of your questions:

"failures can be linked to membership length" -

Not always, as above, sometimes we are let down by some long running members, they don't "ghost" us but life gets in the way, we can't hold that against people.  Finisher prizes and the application process are our main way of weeding people who are not fully committed.  My thought is that they have spent all the time to apply and come up with a project that resonates with me then give up, they are missing out on a finisher prize (which is usually quite good)

"You invest in removing those individuals that are gaming the process"

We certainly track and remember people who have taken advantage.

"What is an acceptable failure to complete rate?"

We try to not have one, although different kits cost different amounts, if we are giving away a £5 or a £500 Dev board we work to minimize people pulling out.  Every lost kit is a lost project and a lost story, never mind the money wasted.

Now for the good news:

We found a large contributor to the issue and seemed to have hit the nail on the head, and had much much better numbers over the past few challenges.  Firstly we changed a bunch of the copy/text and made challengers obligations more obvious, this has helped new and existing members to understand the challenges.

Then we found a problem with how we were advertising these challenges and the targeting used in our social media.

For those not aware of modern online marketing tools, you can choose a "type" of person (not individuals or specific groups), for example, Facebook, would push the advert to.

The way we had set up the targeting had brought lots of visits to the page, which sounds great, but the "quality" of person arriving on the page was very low, we were getting numbers and applicants and it really felt like a success until we went through the challenge and the audience were let down either early on or after we sent the kit.

I am happy to say we have changed the "audiences" and "targeting" for any external pushes for new challengers. These have brought us Challenge winners and new members who have stayed and contributed greatly.

I feel we are in a much better place and have a few shake ups incoming soon! Keep your eyes peeled for the next design challenge!

-Phil

Hey Sean,

Ooof this is an old thread to suddenly resurrect! 

In response to some of your questions:

"failures can be linked to membership length" -

Not always, as above, sometimes we are let down by some long running members, they don't "ghost" us but life gets in the way, we can't hold that against people.  Finisher prizes and the application process are our main way of weeding people who are not fully committed.  My thought is that they have spent all the time to apply and come up with a project that resonates with me then give up, they are missing out on a finisher prize (which is usually quite good)

"You invest in removing those individuals that are gaming the process"

We certainly track and remember people who have taken advantage.

"What is an acceptable failure to complete rate?"

We try to not have one, although different kits cost different amounts, if we are giving away a £5 or a £500 Dev board we work to minimize people pulling out.  Every lost kit is a lost project and a lost story, never mind the money wasted.

Now for the good news:

We found a large contributor to the issue and seemed to have hit the nail on the head, and had much much better numbers over the past few challenges.  Firstly we changed a bunch of the copy/text and made challengers obligations more obvious, this has helped new and existing members to understand the challenges.

Then we found a problem with how we were advertising these challenges and the targeting used in our social media.

For those not aware of modern online marketing tools, you can choose a "type" of person (not individuals or specific groups), for example, Facebook, would push the advert to.

The way we had set up the targeting had brought lots of visits to the page, which sounds great, but the "quality" of person arriving on the page was very low, we were getting numbers and applicants and it really felt like a success until we went through the challenge and the audience were let down either early on or after we sent the kit.

I am happy to say we have changed the "audiences" and "targeting" for any external pushes for new challengers. These have brought us Challenge winners and new members who have stayed and contributed greatly.

I feel we are in a much better place and have a few shake ups incoming soon! Keep your eyes peeled for the next design challenge!

-Phil