Actions
Forum Thread Details
  • Replies 20 replies
  • Subscribers 182 subscribers
  • Views 2113 views
  • Users 0 members are here
Related

What's Your Ideal Way of Handling Judging and Mentoring?

tariq.ahmad

One of the hardest things to be asked in the community is to be asked to judge a competition.  For any given competition there might be five or six really impressive entries that are each worthy winners. You look really closely at each entry and narrow down your list as much as possible and go with the entry you believe is the strongest. Once a winner has been decided, unless there is one that clearly blows away everything else you see, you go with the candidate you believe embodies the spirit of the competition and hope you make the right decision. Often times, you second guess yourself but reassure yourself that you made a conscientious decision, and that any of the candidates you had to choose from, would be worthy winners.  At the same time, you want to reach out to the people that made your decision hard, as its their ideas or projects that made your decision hard. Having multiple prizes each month makes things easier but if this program is as successful as we hope than picking winners is going to be a very difficult job!

 

In an ideal world, its the community members, who the community is for, who would decide.  In reality, there are a lot of practical and logistical things you have to consider if you were to go that route. Like I mentioned, you don't want to undermine the integrity of the program by having the decision based on factors other than the work itself.  At the end of the day, whatever route we go with judging would have to respect the integrity of the program.

 

Several of you expressed concern, precisely over this. Project14 hopes to be a community member-centric program that's not about us but about you.  To ensure the integrity of the program the judging will be in part a responsibility on our end.  However, a Jury of Peers and Mentoring is a core feature of the program that makes it unique.

 

In the comments below tell us what your ideal way handling Judging and Mentoring for our new program!

  • Tariq, In my opinion the unicity of Judging + mentoring is really great and it is also the way for a honest judging. Most of the times I was part of a judging team the authors was unknown. In this case if a serious judge also seriously mentor it has the opportunity to collect all the aspect of the challenge process for every challenger not only supporting him but also giving an evaluation of his job in the most correct way.

     

    Enrico

  • How about having a points system?  You could have the winner determined by a 50/50 combination of popular vote from the community and a vote by a panel of selected judges.  Then the design competition manager (or a sponsor representative) could provide the tie-breaker in case of a tie.  That could keep the integrity of the program intact and allow the whole community to have a say in who wins the competition. 

  • in reply to pettitda

    As apparently your suggestion seems well balanced I think that unfortunately it is not. A panel of judges it is expected to go ahead in a politically correct direction, mentoring and judging. It is not rare that in the competitions where the community members (of any community, not necessarily Element14) arises their ranking depending on the number of friends they have, followers, contacts on other social that help him to gain votes.

     

    Enrico

  • Judging a competition is just like evaluating devices for a design.

     

    You first establish your requirements or desire outcome.

     

    You establish a list of attributes you want to use for comparison and rank them in importance.

     

    You then evaluate each device against the requirements using a fixed scale.  Sometimes we use Red, Yellow, Green, or Blue.  Red means does not meet, Yellow means partially meets, Green means meets and Blue means exceeds.

     

    After you compare all of the candidates you assess the scores.  If more than one candidate is all green, then you use the Blue values to decide based upon a ranking of the requirements before you begin the evaluation.

     

    In the end, it is usually pretty easy to figure out who did and who did not do the better job.

     

    DAB

  • in reply to balearicdynamics

    Agreed. Community polls should only account for a fraction of the total score, due to the ease of manipulation.

  • in reply to DAB

    Dab,

    as it is also involved the mentoring - and it is an advantage IMHO - also a human factor should be considered. How the job has been done, the approach, how the challenger has shared knowledge and efforts etc.

  • in reply to balearicdynamics

    Agreed.

     

    The Mentoring issues are much more difficult to measure.

     

    It depends upon the issues encountered, the skill of the person, their receptiveness to suggestions and their willingness to work with a mentor without expecting the mentor to do all of their work.

     

    In general the same type of process is used.

     

    Set up expectations and evaluate response.  The qualitative Red, Yellow, Green, Blue system works well for this approach as you do not have any "hard" targets, just perception of accomplishment.

     

    It still boils down to identifying your expectations and deciding if they are met.

     

    DAB

  • in reply to balearicdynamics

    My worry is that a panel of judges with community members would make this feel like a job if the same community members are volunteering.  Then it makes sense to rotate a different panel every month. What if you reach a situation where you're stuck finding volunteers in a competition that is always happening? Could advertising the judges create a bad blood in the community if people think they deserved to win but didn't?  At the same time I do like the idea of a serious judge + mentor but it seems like the best way to do this would be having their names be anonymous. I guess one route is to only allow judging by top members but then we don't want people to feel excluded as one of the core tenants of the program is to make it as inclusive for everyone.  Want to keep the focus on fun and the learning and not the prize you get for winning.   These are good ideas and I see what you thinking but tying this too closely to people's names may add a personal element which I worry would change the good mojo we have running in the community.

  • in reply to pettitda

    If this could be done in a gamified where it didn't turn into a simple popularity contest that would be great.  I don't know the mechanics of how gamification and badges work but I can ask about this as this is her area of expertise.  This is different from Design Challenges s because there is no sponsor and its up to the community to choose what components they want to use.  Design Challenges are great and they're not going anywhere but we felt that a new program that wasn't tied too manufacturers or a specific challenge (a general theme) instead seemed like something that would be worthy of its own program.

  • in reply to DAB

    This is good in theory but would all community members judge according to the same system?   Also, the community member staff that monitors these things is small so I don't know if we would have the bandwidth to follow the same system internally, month after month...  I'm intrigued by the idea though and its definitely worth exploring further. 

     

    We're still having volunteer community members + mentors but we're just trying to figure out mechanics that make more sense over what was initially proposed based on the feedback we've gotten so far.