RE: Seeking Feedback on New Scoring System for Experimenting Challenges

Nice! Everybody likes tables of info, so this seems neat (if it's not too much overhead).

The intro blog looks like it will benefit everyone; it is a great way for contestants to quickly pick up points by writing a decent intro, and it will also be a great way for readers to immediately gain an interest in the project so they can follow all the remaining blogs easily, allowing for more interaction.

The weighting looks good too. Those who are in the zone or busy resolving problems with the project and cannot blog as much, can make up for it in the final blog, and also drop blogs along the way whenever they get the opportunity, without worrying they have not created content as frequently as they desire. Very handy if other life stuff gets in the way for a week or so. And those who can produce content, can benefit from a decent total of points for 7 weeks (and people get to read it at a nice rate).

It looks workable, but requires more administrative overhead. I like the idea that intermediate blogs count and I can see how they help keep the site content fresh. I gather you would like to see more continuous interesting content posted rather than a lump posting at the end. Perhaps this could be achieved by simply telling participants and judges that the judges will place value on continuous content postings. If you retroactively apply the above system to past Experimenting contests, would it significantly alter the winners circle? ie. if the new rules don't change the outcome, is there enough incentive for participants to do something differently and is the extra overhead worth it? Perhaps if judges were engaged at the start of the contest, they could be more aware of intermediate activity and its impact. Would leaderboard history provide a useful metric?  I don't have any strong feelings about any of this, just tossing some thoughts into the mix.

I would like to say that the Experimenting Challenges are already a great addition to the mix of contests.

The new scoring system sounds good. I have several minor notes:

• Blogs count minimum you need at least for finisher prizes. You can express it using score but I think it will still define minimum requirement on blog count. I recommend do not removing minimum of two blog posts. If you remove this limit, then you can face situation that someone post the only one (summary) blog in last week. In extreme case he/she can win with 200 points, but I think in this case single blog entry was not very useful contribution to the competition.
• Score must distinguish contestants. I recommend defining rule that in every category two contestants never receive exactly the same number of points. For example look 7 Ways to Leave Your Spartan-6: Contestant Standings. Look at First Project column. All has 0 points (did not complete project) or 50 points (completed project). If you take two projects from this table which received 50 points, then it means that both projects/blogs are totally equal in quality but in reality this is never true. There is always difference (at least very small difference) and scoring system should highlight it. You can for example give one project 45 and second project 44 points. Both contestant get very similar points for very similar quality of projects/blog but there is an order. The contestant with one point less can look at it and decide that he will post one additional intermediary blog for beating the originally slightly better contestant. Idea of this paragraph is that every time there must be the order. Giving all contestant 75 points for introductory blog does not make sense because all (except contestant who post nothing) will get them and there is no need to make any complex strategy in this case.
• Mentioned system rate blogs and project quality but do not count other contestant activities. Very frequently happen that someone ask on forum, someone answer question and help the other contestant to continue its journey or something like it. I think this should be graded and this kind of cooperation should be awarded by points, I think.
• In addition to previous point: forum activity should count. Both question and answers should count. In 7 ways to leave spartan there were category for this but in promoted system it is not. Contestant sometimes get into troubles and asking on forum is very effective way because at the same time there is another 9 contestant who have exactly the same hardware in the hands and together, they can resolve even very complicated issues.
• Carefully take in account amount of work needed to do on element14 side every week. We frequently see delays in selecting challengers, waiting for judges selection, postponing shipping dates and so on. I recommend changing “score updated on weekly basis” to “score updated every second week” and also remember to dedicate time slot every second Friday (or other day) in your calendar. If you will not have time for evaluating blogs every week, than space for complex strategies on contestant side decreases significantly. For allowing making strategies evaluation timing should be strictly followed.

I like the idea because I’d like to read of progress and outcome whilst on the journey.  Perhaps a little sadistically, it would be interesting to see the ‘theory’ for an experiment and the protocol for testing that before actually seeing the outcome so that nothing is hidden: very transparent and shows a good logical approach to the challenge.  All I could really add is to try it and see what happens.

Interesting points. For what it's worth, I think the "forum activity should count. Both question and answers should count." would be hard to deal with, because then judges need to go through all the comments as well as the blog content. Also, it could be open to abuse, by people just replying many times with not as much help as required (e.g. mundane comments instead of well thought out responses). i think the onus on getting answers to challenge issues should be on the contestants, they are best placed to do it, and if they cannot find the answer on the forum then they can ask a question anwhere, e.g. on their blog, on someone elses blog or other sites. Collecting up questions and answers in discussions seems (to me) difficult (unless I've misunderstood it). If people are not responding to questions asked of them, then it becomes clear they are not a team player, and you wouldn't get much of an answer from them even if there were (say) 10 points for a response.

Regarding ranking, sometimes content is just as good, so it would be nice if there was still a way to recognize that sometimes people do equally well in different ways. But I can see it would make it hard for contestants during the contest, who might prefer ranking (unless they are going to come second or third when their content may be equally as good! :) 

I like the idea that every score should be different, but that adds a lot of judging work. Maybe someone can come up with an easy way to achieve accurate differentiated scores without a huge amount of judging work.

I agree that it is lot of work and maybe this rule can be relaxed slightly. I mainly highlighted the situation when all received full points which eficiently decraded score to true/false ranking.

I did not take abuses in account. Of course if some start flooding too much questions, then he/she will receive 0 or very few points for it. Similarly you can abuse intermediary blogs. Some can split his text in a ways one blog -> one paragraf and then he will easily have tons of intermediate blogs but I gues it will not receive lot of points for it. Even worse, you can post 50 blogs of lorem ipsum with idea that 50 blogs can bring you up to 500 points! In this cases the magic words "up to" should apply and zero points still satisfy rule up to 10 points.

For the 7 Ways program, there were 25 participants. I was able to update the table and it didn't take that much time. I did have to spend a bit of time verifying if a participant completed a task. Actually, the quiz was the most time consuming because we have to go to a database and then search through some data. For the upcoming Experimenting challenges, I am looking at 7 to 10 applicants. So, I think it's manageable to do a table of this sort. The reports I got back from participants is that they really like the transparency. They knew were they stood, point wise.

How many of these are important in a scoring system and how would you rank their importance? Are there other important properties?

• technical content
• creativity
• entertainment and engagement value
• significant content published regularly throughout the contest period
• clarity & completeness
• effort expended