Actions
Forum Thread Details
  • Replies 20 replies
  • Subscribers 206 subscribers
  • Views 2776 views
  • Users 0 members are here
Related

What's Your Ideal Way of Handling Judging and Mentoring?

tariq.ahmad

One of the hardest things to be asked in the community is to be asked to judge a competition.  For any given competition there might be five or six really impressive entries that are each worthy winners. You look really closely at each entry and narrow down your list as much as possible and go with the entry you believe is the strongest. Once a winner has been decided, unless there is one that clearly blows away everything else you see, you go with the candidate you believe embodies the spirit of the competition and hope you make the right decision. Often times, you second guess yourself but reassure yourself that you made a conscientious decision, and that any of the candidates you had to choose from, would be worthy winners.  At the same time, you want to reach out to the people that made your decision hard, as its their ideas or projects that made your decision hard. Having multiple prizes each month makes things easier but if this program is as successful as we hope than picking winners is going to be a very difficult job!

 

In an ideal world, its the community members, who the community is for, who would decide.  In reality, there are a lot of practical and logistical things you have to consider if you were to go that route. Like I mentioned, you don't want to undermine the integrity of the program by having the decision based on factors other than the work itself.  At the end of the day, whatever route we go with judging would have to respect the integrity of the program.

 

Several of you expressed concern, precisely over this. Project14 hopes to be a community member-centric program that's not about us but about you.  To ensure the integrity of the program the judging will be in part a responsibility on our end.  However, a Jury of Peers and Mentoring is a core feature of the program that makes it unique.

 

In the comments below tell us what your ideal way handling Judging and Mentoring for our new program!

  • in reply to tariq.ahmad

    Tariq,

    as far as I know, including the previous Element14 challenges, judges are never anonymous and in this case it is even more difficult as the judges are also - at least someone of them - mentor. Also keeping the information public I think that it is not a shame to involve only the TM for first then making some reports and discussions this mechanism will set by itself.

     

    About involving community members the judge role is just an invitation, IMHO, like the users to participate to a challenge. A judge is a volunteer, involved in the challenge as well as the challenger. Do you know the game Eleusis, illustrated around the mid of '60s (past century image ) by Martin Gardner on Scientific American? It is just our case.

    To say this in few words:

    • judges should mentor and the judges panel decide the winner and the challengers ranking based on the duality of the project results, and project conduction
    • Challengers will report their feel with judges and the judges panel will become a circulating group of volunteers that have their own ranking (and can be excluded with a too low value) based on the challengers reporting.
    • The judges are a fixed number every challenge but the subjects rotates (not necessarily all should change in two different challenges) depending on the number of volunteers judges respect the nudges panel (fixed number of persons)

    I think that with these rules what is granted is a self-balancing system.

     

    Enrico

  • in reply to tariq.ahmad

    Unfortunately, at it's heart, the gamification system operations much like a popularity contest (or perhaps more accurately, a participation contest). It can only track certain "tangible" actions (commenting, liking, viewing, etc.) in a basic "and/or, greater than or equal to" system. It couldn't even be used to track/award something that is "the most", i.e. "Out of these six blogs, <this one> received the most comments, likes, and views". That is something we would have to manually track and assign.

  • in reply to fvan

    The idea was floated of having the voting in polls determine the winner of the competition when monthly projects was initially floated.

     

    I was not involved at the time but had we gone that direction I would have felt uncomfortable with using polls to determine the winners.

     

    In the end we used the polls to determine the monthly themes.

  • in reply to e14megan

    I get what you're saying but I'd be less concerned with people gaming the system with gamification around project entries themselves than a picture of a project in a poll.

     

    From what I've heard internally, there was something before my time that involved polls and projects. It didn't work, which is why the idea was thrown out.

     

    Of course, you can't have a perfect system of judging but you can set up a system where the focus is on the work itself and that attracts the right audience.

     

    That's why blogs containing video, images, and compelling narratives (what makes things interesting); steps to show how you did it (learning and showing what's possible), are such necessary elements for building a program that's about the work itself (think Pikea table and Ben Heck builds) and not individuals.

     

    In my experience, in the year that I've been in the community, people love seeing cool projects and are often only too happy to share stuff that they've done on their own.

     

    The most popular content as far as things that generate likes, commenting, and views are about the work itself.

     

    That's what I find intriguing about going the gamification route, or at the very least setting up badges and the like.

     

    Real human judges will always be a part of the equation of course.

     

    You'll need people on our side to ensure that the integrity of the competition is upheld and you also have a great community that does a better job of self policing than can be expected almost anywhere else.

     

    We've got lots of options.   Its intriguing to hear everyone's ideas.

  • There are always risks with any judging system, but I think it would be very rare for members who are judges to purposely abuse their mandate and they would risk not being selected as judges again.

    The more we know about the judges the less likely they are to exhibit questionable judgement.

    If we know their names, already their reputations are at stake.

    If we see what scores each judge awards, (like Olympic diving) it is even less likely they would abuse their power,

    if we see their reasons for their scores, it is full transparency.

    On the other hand, anonymous secret ballots - where it is not known who voted, allow anyone to abuse the system without consequence.

    If you are really worried about it, you could eliminate the low and high judge's scores from each contestant's total. But I don't favour this as it can eliminate all the hard work of a judge.

    I suspect mentoring will work fine without rewards - success of mentee projects will be enough.

    Judge rewards, I'm not so sure, it can be a thankless job. They should at least get a special judge's badge.

  • in reply to balearicdynamics

    Hey ,

     

    One idea that was discussed internally was a jury of peers and a system for signing up.

     

    Don't think we're ready for that as there are still some concerns.

     

    A concern would be that this may over complicate the process and become burdensome if the unexpected happens and there isn't a lot of interest around a particular theme, or there's a time of the year where people are occupied with family, vacation, etc. like over the winter.

     

    Don't want the judging to feel like work or get too bureaucratic.

     

    An honor system with top members, at least initially, is very sensible and should work.

     

    We will be reaching out further to top members for help with the judging and those that volunteered will be contacted once projects start coming in.

  • in reply to dougw

    Yeah, gamification and badges would definitely need to be included!

     

    I believe you could award a badge for mentoring or judging after the event and keep it anonymous while the competition is happening... 

     

    Will have to circle back to as I'm still cloudy on the mechanics, etc of how all that works.

     

    We would know internally who the judges are but as far as announcing externally, that's where there's room for debate.

  • in reply to tariq.ahmad

    The idea was floated of having the voting in polls determine the winner of the competition when monthly projects was initially floated.

    It could be made to work.

     

    One idea is for the poll to be hidden and only selected people given access.

    It may be that not everyone invited actually votes, but once the voting ends, then the top 5 or so are looked at by element14 and the winner/winners decided.

     

    You might have 10, 20 asked to vote.

    You can decide who you ask, which may include a cross section of the community.

     

    The criteria for voting can also be included in the 'invite' and might include must have a video of it working, etc.

     

     

    Mark

  • As a teacher, I already get to judge enough as it is. I know my students, I know what they are capable of, and I can "judge" them accordingly, but it isn't a nice thing to do. I hate giving out marks, since sometimes I have to give a bad mark to someone who handed in a great idea, but didn't quite get the code right, or forgot some elemental piece. What I do like on the other hand is pointing out things (constructively) that could be changed. Maybe this would be better, maybe doing this might be easier. That's the 20+ years experience talking.

     

    Judging which project is the "best" isn't too hard. Mentoring depends on specific people, and the relationship (students want to be taught, some adults don't). Marking, on the other hand, is hard...

  • in reply to tariq.ahmad

    Thank you for your details Tariq. Despite the suggested rules I mentioned (maybe the list makes things similar to a sort of "must") the idea around the Eleusis games was just to create a balanced system between judges and challengers. In a free and friendly way. I perfectly agree that the entire process by both sides (judging and challenging) should never feel like a sort of job (also a volunteering job) but should feel mostly a nice activity, a cooperation oriented to knowledge exchange.

     

    I have considered TMs as the starting point, then it is needed to assume the risk of the first months to experiment and calibrate the best methodology.