Actions
Forum Thread Details
  • Replies 34 replies
  • Subscribers 12 subscribers
  • Views 4819 views
  • Users 0 members are here
Related

RE: Seeking Feedback on New Scoring System for Experimenting Challenges

rscasny

Note: this discussion covers only the Experimenting Design Challenge Competitions

Greetings to All,

I would like to introduce a blog scoring system in addition to the judges I use to determine the top prize winners of the experimenting with components challenges (e.g, capacitors, inductors, vibration sensors, etc.).

I would like to get feedback on this proposal by members of element14.

Before I detail the change, let me explain why I am suggesting it? There are a couple of reasons:

  • Transparency: The blog will be scored on a weekly basis and the score posted in a table similar to what I used in another one of my programs. Here is an example of the table I had used in that program. Click here to see.
  • Strategy: With participants seeing the point totals of all participants, they can plan a strategy for achieving a point total to win.
  • Ensuring Participation Throughout the Program Period: Sometimes participants post their blogs at the end of the 8 to 10 week period, not giving me the opportunity to promote the blogs through the program period. In addition, giving participants a reason to post supporting blogs throughout the active period allows them to pick up points along the way.
  • Making Judging More Convenient: Our judges will receive the final summary blog to judge, which I think will make it easier for them to manage. Since the final summary blog can earn the most points, the judges' decision will have the most weight in the final selections.

What is the Judging/Scoring System In Place Now?

There is a two-blog minimum required to successfully finish the competition. There is no scoring system in place now. Currently, at the conclusion of the competition (usually 8 to 10 weeks), I forward all the blogs of each participant to the judges. The judges rank the blogs and select the winners of the top prizes. At the end of the judging, I notify the winners and write a winner's announcement.

What is the Judging/Scoring System That I Would Like to Introduce?

There will be no blog minimum, but each participant needs to blog to accrue points to win the top prizes. There will be three scoring types, as follows:

  • Introductory Blog: participant describes in detail what he/she plans on doing. Worth up to 75 points.
  • Final Summary Report Blog: participant describes in details what he/she did and proposed in the introductory blog. What experiments were conducted. What results were obtained? What were the takeaways, that is, what was learned. Worth up to 200 points.
  • Intermediary Supporting Blogs: these are blogs that support their experimenting program but are not the Introductory or Final Summary Blogs. Each weekly blog is worth up to 10 points.

The overall point totals will be posted on the table on a weekly basis.

The typical program timeline will be:

  • The up-to-75-point Introductory blog: Needs to be posted in Week 1 and Week 2
  • The up-to-10-point Intermediary Supporting Blogs: Need to be posted each week in Weeks 3 through 9.
  • The up-to-200-point Final Summary Blog: Needs to be posted in week 10.

Element14 will assign points for the Introductory and Supporting Blogs.

The judges will assign points for the Final Summary blog.

The members with the most points at the end of the program period will win.

What do you think?

Randall
--element14 Team; Experimenting Challenge Program Manager

Parents

  • I'm all for the whole transparency and getting an idea of how things are going along the way. Sounds like a good plan.

    My main concerns really lie with the supporting blogs - is this done as "up to 10pts per week" or is it "up to 10pts per blog posted within the period"?

    I can see people "salami-slicing" their blogs to create more blogs, but less useful content, to earn an extra few lots of 10pts if it is the latter. If it is the former, then people with commitments may find it difficult to contribute regularly and be unfairly penalised if they have to take a week away.

    Another concern is the weighting of e14 versus the community judges - at present, it's something like 145 points to e14 and 200pts to community member judges - should this be closer to 50/50, or is the intention to still leave more of it in the community member judge's hands?

    - Gough

  • in reply to Gough Lui

    For the supporting blogs, the maximum award would be 10 points per week, not 10 points per blog.

    Over the course of the competition some could produce maybe 8 or 10 supporting blogs, thus, an extra 80 to 100 points--or about 45-50% of the total possible award of the final summary blog.. Hence, they have value. Conversely, if someone was perfect in every way and wrote superb intro and final blogs and were give the full points, that is, 275 points, they would be a good participant. A sizable amount and a possible winner. This would be a rare case, I think. 

    The judges would have the most weight in the final decision. That is my intent. However, the supporting blogs add into the fiNAL point total. Not every participant would get the full amount of points (200) for the final blog. 

Reply
  • in reply to Gough Lui

    For the supporting blogs, the maximum award would be 10 points per week, not 10 points per blog.

    Over the course of the competition some could produce maybe 8 or 10 supporting blogs, thus, an extra 80 to 100 points--or about 45-50% of the total possible award of the final summary blog.. Hence, they have value. Conversely, if someone was perfect in every way and wrote superb intro and final blogs and were give the full points, that is, 275 points, they would be a good participant. A sizable amount and a possible winner. This would be a rare case, I think. 

    The judges would have the most weight in the final decision. That is my intent. However, the supporting blogs add into the fiNAL point total. Not every participant would get the full amount of points (200) for the final blog. 

Children
  • in reply to rscasny

    Thanks for clarifying. As life can get in the way of producing a blog every week for some people (myself included), would it be better to consider judging every blog and taking the best 10 scores? A bit more work for the e14 team, but relaxes the need to contribute every week.

    If it's up to 10 points per week, can someone get a 10-score for a week if they post two blogs of 5pts each? Or do you take the best score for all the blogs of that week?

    - Gough

  • in reply to Gough Lui

    Most people have not written even 1 blog per week in the experimenting program and other programs I have managed.. And there have been a handful who have written a few blogs a week. For example, in the gesture sensors challenge, I believe there were 35 blogs written total. There were 7 participants. If each wrote 1 blog per week over 8 weeks that would be 56 blogs. (Now most of the participants wrote 2 blogs; one told me he had a busy schedule thus he wrote only 2 blogs over the course of the program.) We are no where near 56 blogs. So, credit for a blog a week seems reasonable approach to me. But keep in mind the management of all this content and the judging of it. I am not going to burden our judges with too much. This is why they will be responsible for judging the intro and the final blogs and that's why those things get the most points. 

  • in reply to rscasny
    rscasny said:
    Most people have not written even 1 blog per week in the experimenting program and other programs I have managed

    I think this might be historically true, but place the incentive in front of them, and you may well find a shift for the majority of participants ... in which case, I don't necessarily think it to be fair if someone averages one blog a week (but misses a week here and there, but puts out double in other weeks) is eligible for more points that one who puts out one blog consistently every week.

    Then again, perhaps it's putting the onus on the challenger to prepare their blogs ahead of time and release them in a scheduled fashion, rather than "as soon as possible".

    - Gough

  • in reply to Gough Lui

    So, the issue here is the timing of  the supporting blogs. And the concern of missing a week due to other commitments in one's life, while other weeks one would have the time to make up for the missed week. I think that's a fair concern for someone who wants to produce many supporting, mid-course blogs. Most people have not..This is how I think about it. Strategy--the most points are awarded for the intro and the final blog. So, those need to be emphasized. The supporting blogs add in the total points, call them extra points, and they take pressure off of needing to produce a fantastic final blog. But you make good points. It sounds like I need to revisit the supporting blogs and how they are credited. Good discussion. Thank you for your input. 

  • in reply to rscasny

    I expect the main scenario where supporting blog points would come into play is to break a "tie". In every other situation it would be tough to expect a judge to demote what they consider to be a much better project on the basis of fewer supporting blogs. In the case of a close call, considering supporting blogs will easily resolve as to which project did a better job of publishing "regular" interesting content, regardless of the precise schedule.

    Most winning projects already publish supporting blogs, but I think it is important to make clear what is desired - for example a weekly update blog could simply be a publicized guideline that judges consider, or it could be a hard criterion that requires a waiver to violate.

    When a contest indicates 2 blogs are needed, I try to restrict my blogs to 2 because I assume this requirement is to make it easier on the judges. Having to deal with 25 blogs on a single project is a nightmare for judges, and as a content consumer I personally do not like trying to read a project that is broken into 35 separate blogs.