RE: Seeking Feedback on New Scoring System for Experimenting Challenges

This is my first thought on this system:

--If the supporting blog is value adding (better explains something, helps some understand better, is very relevant, etc.), it merits the full ten points

--If the supporting blog is only an unboxing that adds images and repeats info in data sheets, it merits 5 points.

--If the supporting blog is a blog just for the sake of writing a blog (ie., telling me he received the kit and photos of the kit and nothing more, it merits only 2 points

This needs clarification. Let's say for example Week 3 starts on Jan 22 and Week 4 starts on Jan 29th and week 5 starts on Feb 5th. In other words, three weeks. Each week you will get awarded a maximum of ten points for supporting blogs. Hence, for 3 weeks, that's 30 points. If people understand that, they are not going to write 50 blogs nor expect 500 points. Thus, I need to work on clarifying these instructions. 

You mention a lot of things. Thanks for taking the time to join this conversation. I'll address two of them now. 

Supporting blogs should not get all the same points. Yes, for the 7 ways program, most of the activities but not all of them I awarded with the same points. But this was different from Experimenting. If I asked them to write the differences between Spartan 6 and Spartan 7, the blogs would be similar since they were based on the same source info from the sponsor. Hence, similar point awards. For experimenting challenge, the supporting blogs would inherently be different. But to answer your question, yes they could be score differently. I am thinking of a standard such as the following:

If the supporting blog is value adding (better explains something, helps some understand better, is very relevant, etc.), it merits the full ten points

If the supporting blog is only an unboxing that adds images and repeats into in data sheets, it merits 5 points.

If the supporting blog is a blog just for the sake of writing a blog (ie., telling me he received the kit and phots of the kit and nothing more, it merits only 2 points. 

Your other suggestions concerned quantifying forum assistance. Very hard to track that in not only a democratic manner but also would not take much time. I would have to do that. I would not burden the judges with that task. 

Perhaps I could create a poll at the end of the program asking the community which participant do you feel has provided the most support? The top vote get would receive, say, a 20-point bonus. Just thinking out loud. 

Yes, there is more :"administrative overhead" with the table. As I told shabaz, 7 ways had 25 participants, while my upcoming Experimenting Challenge will have 7, due primarily due to budgeting limitations. So, it's less work. I'd also add the updating of the table into our project management tool. You ask some really great questions. I felt that the 7 Ways approach got more participation and it was a checklist for the participants. Would it alter the results? I can't be certain, but for one or two of them, it could have. I've really soul-searched about how much a burden I should be placing on the judges. I know some of them have full time jobs. I have been concerned about sending the judges 30 or 40 individual blogs to judge and then requiring them to rank the participants. I want the final decision to rest with the judges,yet at the same time foster participation throughout the program's timeline. 

How many of these are important in a scoring system and how would you rank their importance? Are there other important properties?

• technical content
• creativity
• entertainment and engagement value
• significant content published regularly throughout the contest period
• clarity & completeness
• effort expended

For the 7 Ways program, there were 25 participants. I was able to update the table and it didn't take that much time. I did have to spend a bit of time verifying if a participant completed a task. Actually, the quiz was the most time consuming because we have to go to a database and then search through some data. For the upcoming Experimenting challenges, I am looking at 7 to 10 applicants. So, I think it's manageable to do a table of this sort. The reports I got back from participants is that they really like the transparency. They knew were they stood, point wise.

I did not take abuses in account. Of course if some start flooding too much questions, then he/she will receive 0 or very few points for it. Similarly you can abuse intermediary blogs. Some can split his text in a ways one blog -> one paragraf and then he will easily have tons of intermediate blogs but I gues it will not receive lot of points for it. Even worse, you can post 50 blogs of lorem ipsum with idea that 50 blogs can bring you up to 500 points! In this cases the magic words "up to" should apply and zero points still satisfy rule up to 10 points.

I agree that it is lot of work and maybe this rule can be relaxed slightly. I mainly highlighted the situation when all received full points which eficiently decraded score to true/false ranking.

I like the idea that every score should be different, but that adds a lot of judging work. Maybe someone can come up with an easy way to achieve accurate differentiated scores without a huge amount of judging work.

Interesting points. For what it's worth, I think the "forum activity should count. Both question and answers should count." would be hard to deal with, because then judges need to go through all the comments as well as the blog content. Also, it could be open to abuse, by people just replying many times with not as much help as required (e.g. mundane comments instead of well thought out responses). i think the onus on getting answers to challenge issues should be on the contestants, they are best placed to do it, and if they cannot find the answer on the forum then they can ask a question anwhere, e.g. on their blog, on someone elses blog or other sites. Collecting up questions and answers in discussions seems (to me) difficult (unless I've misunderstood it). If people are not responding to questions asked of them, then it becomes clear they are not a team player, and you wouldn't get much of an answer from them even if there were (say) 10 points for a response.

Regarding ranking, sometimes content is just as good, so it would be nice if there was still a way to recognize that sometimes people do equally well in different ways. But I can see it would make it hard for contestants during the contest, who might prefer ranking (unless they are going to come second or third when their content may be equally as good! :)