Judging

Big Ideas@Berkeley believes that selecting appropriate judges and crafting appropriate judging criteria are critical to the Contest’s success: Without both of these components, Big Ideas would be unable to identify from the large pool of applicants those with the most innovative and impactful ideas.

Judge Qualifications
Each year, Big Ideas@Berkeley selects a pool of judges (see Judge Recruitment sub-section for detailed information on recruitment strategies) who have content expertise within a particular category. Judges in the Global Poverty Alleviation category, for example, typically are faculty or professionals with expertise in international development studies who have spent a significant portion of their careers working to end poverty domestically and abroad. Judges in the Creative Expression for Social Justice category, similarly, may be faculty or professionals with expertise in the use of dramatic, performing, or visual arts to effect social change. Put differently, judges are selected on the basis of their credentials and their fit with the Contest categories.

Number of Judges
Each Pre-proposal and Full Proposal is read by a minimum of four judges, in order to provide an average score that is a reliable and valid indicator of each proposal’s strength. Given the large number of Pre-proposals received (sometimes up to 40 in one category), Pre-proposal judges cannot read every proposal submitted within their assigned category. As a result, each judge is expected to read and score only a subset of the Pre-proposals submitted in their category—usually between six and eight. In contrast, Full Proposal judges are expected to read all of the proposals submitted in their category (between three and ten, depending on the category).

It is worth noting here that each year Pre-proposal judges remark that it is difficult to score the Pre-proposals assigned to them without knowing something about the entire pool of Pre-proposals in their category. In other words, it is difficult to rank each proposal without knowing its relative strength compared to all other proposals. As a result, although Pre-proposal judges are only assigned a subset of Pre-proposals in their category to read, they are given paragraph long summaries of all of the proposals within their category.

In the 2012-13 Contest year, Big Ideas received 160 Pre-proposal applications, and successfully recruited 87 Pre-proposal judges. From this pool 54 finalists were selected and submitted Full Proposal applications, and Big Ideas recruited 32 Full Proposal judges to review them.

Judge Training and Judging Timeline
After judges have been recruited for each round of the Contest, a training is held by Big Ideas@Berkeley staff (typically during the lunch hour, with lunch provided to incentivize participation) to familiarize judges with the mission and goals of Big Ideas and the proposal judging process. This training highlights differences between Big Ideas and other innovation or business plan competitions, provides detailed explanations on evaluating various elements in the proposals, and walks judges through how to submit their scores via the online Contest platform (see Online Platform).

The information provided during this training session is provided in a Judging Handbook that is available to judges on the Big Ideas website and is sent via email to all judges. Additionally, the judges’ training session is made available via a live webcast and a recorded version is made available to judges who are unable to attend the in-person training.

After the training session, judges are given approximately three weeks to submit their scores.

Judging Criteria
As detailed in the Contest Structure section, the judging criteria from the Pre-proposal round focuses on the extent to which the teams propose a creative solution to a social problem and the project’s intended social impact, while Full Proposals are judged primarily on the project’s potential social impact and the viability of the project plans. That said, the items on the judging scorecard for each round largely do not change from the Pre-proposal to the Full Proposal round; instead, the largest change between the Pre-proposal and Full proposal scorecards are the relative weights of each judging item. The following table indicates the broad areas assessed by judges, and their relative weights in each round:

Judge Recruitment
The best resource to help recruit judges are the departments and organizations that sponsor particular categories. For instance, the Center for Information Technology Research in the Interest of Society (CITRIS), which sponsors the annual IT for Society category, pulls from its broad network of faculty and industry professionals to assist Big Ideas staff in finding judges. As noted in the Funding & Category Development section, on a limited basis Big Ideas also has offered in-kind sponsorships to organizations and centers with close ties to particular categories in exchange for their support in recruiting judges and mentors, and promotional support. These types of in-kind collaborations helped tremendously with judges’ recruitment and reduced the administrative burden associated with recruitment on Big Ideas staff.

The incentive for judges to participate varies, but many choose to participate for one of three reasons: a) the opportunity to give back while also getting a first-hand look at some of the most innovative student ideas as they are being developed, b) the opportunity to build their own professional networks by attending Big Ideas events and mixers where they can meet other judges, professional mentors, faculty and students, or c) a professional courtesy to the category sponsor. It is worth noting that, generally speaking, judges who are incentivized by “giving back” and “professional development/networking” tend to be more committed and reliable judges.

In addition to working with category sponsors, the following strategies were utilized to recruit judges during the 2012-13 Contest year:

  • Judges from previous Contest years were emailed and asked if they would like to serve again as a judge for the Contest.
  • Big Ideas@Berkeley staff conducted online research, looking up organizations and departments on the UC Berkeley campus that might know of individuals to serve as judges.
  • Big Ideas staff utilized their professional networks and personal connections to identify potential judges.
  • The Big Ideas@Berkeley webpage advertised the opportunity to volunteer as a judge and contained a very short webform that interested individuals could fill out.
  • Big Ideas maintains a listserve of email addresses of all past applicants, mentors, judges, sponsors, event attendees, and individuals who have at one time expressed interest in learning more about Big Ideas. Big Ideas, at regular intervals, sends out announcements and newsletters about the Contest, and included in these emails are short messages that link to the Judge Application Form on our website.

Tips

  • Emphasize written feedback: During the judge recruitment and the review stage, stress the importance of providing comprehensive qualitative feedback to the applicants. Along with mentors, this feedback is cited by applicants as one of the most valuable resources provided by the Contest.
  • Follow-up and build a deadline cushion: Judges are busy professionals and so it is imperative to send constant reminders and follow-up emails throughout the judging period. Even with frequent reminders, there will inevitably be judges who need an extension to complete their reviews, so build a cushion into the review timeline.

Tools