Big Ideas believes that selecting appropriate judges and crafting appropriate judging criteria are critical to the Contest’s success: Without both of these components, Big Ideas would be unable to identify those with the most innovative and impactful ideas from the large pool of applications received each year.
Each year, Big Ideas selects a pool of judges (see the Mentor & Judge Recruitment section in the Mentorship chapter for detailed information on recruitment strategies) who have content expertise within a particular category. Judges in the Global Health category, for example, typically are faculty or professionals with expertise who have spent a significant portion of their careers working to address health-related issues domestically and abroad. Judges in the Art & Social Change category, similarly, may be faculty or professionals with expertise in the use of dramatic, performing, or visual arts to effect social change. Put differently, judges are selected on the basis of their credentials and their fit with the Contest categories.
In the 2015-2016 contest, Big Ideas recruited 270 judges for the Pre-proposal round (266 Pre-proposals) and 121 judges for the Final round (54 Full Proposals). It is generally recommended that 1 judge is recruited for every anticipated Pre-proposal submission, and 2 judges recruited for each Full Proposal submission to ensure that each proposal is read by a minimum of 6 different judges in each round.
The incentives for judges to participate vary, but many choose to participate for one of three reasons: 1) the opportunity to give back while also getting a first-hand look at some of the most innovative student ideas as they are being developed; 2) the opportunity to build their own professional networks by attending Big Ideas events and mixers where they can meet other judges, professional mentors, faculty and students; or 3) a professional courtesy to the category sponsor. It is worth noting that, generally speaking, judges who are incentivized by “giving back” and “professional development/networking” tend to be more committed and reliable judges. Due to the similarities in outreach strategies between Judge and Mentor recruitment, the process was integrated in 2015 into one large outreach effort in the beginning of the year (see the section entitled Mentor & Judge Outreach Strategies in the Mentorship chapter for a full overview on the types of approaches used).
Each Pre-proposal and Full Proposal is reviewed by a minimum of six judges, in order to provide an average score that is a reliable and valid indicator of each proposal’s strength. Given the large number of Pre-proposals received (sometimes up to 70 in one category), Pre-proposal judges cannot read every proposal submitted within their assigned category. As a result, each judge is expected to read and score only a subset of the Pre-proposals submitted in their category—usually between six and eight. In contrast, with the exception of a few larger categories like Global Health, Full Proposal judges are expected to read all of the proposals submitted in their category so they have a complete picture of the projects as they score each proposal (between five and ten, depending on the category).
It is worth noting here that Pre-proposal judges sometimes remark that it is difficult to rank each proposal without knowing its relative strength compared to all other proposals, since they only read a subset in the first round. Big Ideas thus provides Pre-proposal judges with paragraph-long summaries of all of the proposals within their category so they have a better idea of the types of innovations being proposed across the entire category (see the Pre-proposal Summaries Example – Food Systems in the Tools section).
Judge Training and Judging Timeline
After judges have been recruited for each round of the Contest, a training is held by Big Ideas staff to familiarize judges with the mission and goals of Big Ideas and the proposal judging process. This training highlights differences between Big Ideas and other innovation or business plan competitions, provides detailed explanations on evaluating various elements in the proposals, and walks judges through how to submit their scores via the online Contest platform (see the Judge Training Prezi and Judge Training Recording in the Tools section for more information).
The information provided during this training session is provided in a Judging Handbook that is available to judges on the Big Ideas website and is sent via email to all judges (see the Pre-proposal and Full Proposal Judging Handbooks in the Tools section). Additionally, the judge training is made available via a live webcast and a recorded version is made available to judges who are unable to attend the in-person training. After the training session, judges are given approximately three weeks to submit their scores and provide written feedback.
The judging criteria from the Pre-proposal round focuses on the extent to which the teams propose a creative solution to a social problem and the project’s intended social impact, while Full Proposals are judged primarily on the project’s potential social impact and the viability of the project plans (for a detailed description on the criteria used to assess each proposal, see the Judging Criteria example in the Tools section).
During both the Pre-proposal and Full Proposal judging rounds, judges are asked to provide both quantitative feedback (i.e., Likert scale rankings of the proposals’ strength in particular areas) and comprehensive qualitative feedback (i.e., feedback written to the team about areas of strength and areas for improvement in their proposals). At the end of each round, all applicants (finalists and non-finalists, winners and non-winners) are given the qualitative feedback so that they may use the judges’ advice to strengthen their proposals and/or project ideas in the future. Students report in surveys and informal conversations with Big Ideas staff that providing judges’ feedback is among the most useful resources provided through the competition. They say the feedback is helpful and informative to their team and that the level of detail in the feedback is rarely offered in other competitions. Furthermore, providing comprehensive written feedback to all teams ensures that both finalist and non-finalists receive a key takeaway from the Pre-proposal round. Numerous non-finalists have used the Pre-proposal feedback as a resource for further developing and refining their project for other funding opportunities and for resubmission to Big Ideas the following year.
- Emphasize high-quality written feedback. During the judge recruitment and the review stage, stress the importance of providing comprehensive qualitative feedback to the applicants. Along with mentors, this feedback is cited by applicants as one of the most valuable resources provided by the Contest.
- Follow-up and build in a deadline cushion. Judges are busy professionals and so it is imperative to send constant reminders and follow-up emails throughout the judging period. Even with frequent reminders, there will inevitably be judges who need an extension to complete their reviews, so allow some flexibility with the judging deadlines– typically 3 to 5 days and if possible, a weekend.
- Suggest that judges be critical, but constructive, in their feedback. Judges will often be too complimentary in their comments, and other times too harsh. It is important to communicate to judges that their comments should be written as if they were directly addressed to the team. The most effective feedback raises questions and considerations that the team has not addressed in their proposal, without demoralizing the team from pursuing a solution.
- 2015-2016 Winners Summaries
- Full Proposal Judge Training Prezi
- Full Proposal Judging Handbook
- Full Proposal Judging Scorecard
- Judge & Mentor Application
- Judge Training Invitation Emails
- Judging Deadline Reminder Email
- Pre-proposal Judge Training Prezi
- Pre-proposal Judging Handbook
- Pre-proposal Judging Scorecard
- Pre-proposal Summaries Example – Food Systems
- Recruitment Materials – Judge & Mentor Email
- Recruitment Materials – Judges FAQs
- Recruitment Materials – Judges Overview