Evaluation and Feedback

Big Ideas@Berkeley firmly believes that rigorous program evaluation is key to understanding whether or not the Contest is meeting its goals. As a result, Big Ideas@Berkeley conducts both formal and informal evaluations each Contest year.

Formal Evaluation
Each year, Big Ideas@Berkeley surveys its applicants to better understand a) the extent to which students developed skills that Big Ideas believes are critical to project management and entrepreneurial success (e.g., grant-writing skills, project management skills, leadership skills) and b) the extent to which the process of participating in the Contest was easy, straightforward, and supportive for students. In other words, formal evaluation seeks to understand skill development and provides Big Ideas staff with Contest management feedback from students.

In the 2011-2012 Contest year, both applicant teams selected as finalists and those not selected as finalists were surveyed in the spring semester before final decisions were made. Surveys were created using Google Forms and the links to the forms were sent to applicants via email.

Despite sending multiple reminder emails, the surveys were completed by only a small portion of finalists and non-finalists (less than 20%), and Big Ideas staff were therefore unable to analyze results from a representative sample. As a result, Big Ideas@Berkeley offered $5 Amazon.com gift cards as an incentive for participation, and although this improved the response rate, only about half of finalists and even fewer applicants who were not chosen as finalists completed the survey.

Given these difficulties with response rate, during the 2012-13 year, Big Ideas@Berkeley embedded the applicant survey into the online Contest entry form. As a result, each applicant was required to answer survey questions in order to submit their application. To ensure that applicant’s would respond as candidly as possible, they were assured that survey answers would be de-identified and would have no impact on their proposals’ scores. In addition, although Big Ideas@Berkeley believes that all students who enter the Contest (even those not selected as finalists) benefit from participation, surveys focused only on finalists in the 2012-13 Contest, as this group of applicants was expected to experience greater educational gains.

Survey questions were developed by Big Ideas@Berkeley staff and are refined each year to ensure that they provide accurate measurements of skill development and provide opportunities for feedback on Contest components. See the Tools section for the 2012-13 survey questions and an example evaluation report.

In addition to surveying applicants, each year, Big Ideas also surveys judges and mentors to better understand their experiences and to improve training and support for judges and mentors in future years. Typically, only a small fraction of judges and mentors participate in these surveys, but their feedback (largely provided in response to open-ended questions) is used to make changes to the judging criteria, judge training, and mentorship programs each year.

Informal Evaluation
Although formal evaluation provides useful information on skill development and feedback on the application process, Big Ideas@Berkeley also evaluates the extent to which teams continue to work on their Big Ideas projects and the impact that those teams are making.

To assess project progress and their impact, Big Ideas@Berkeley first created a LinkedIn group to connect past winners. Big Ideas@Berkeley staff hoped that the group would provide a forum for past winners to share their accomplishments with each other and with staff, but the LinkedIn group has proven relatively inactive, and has therefore not been a particularly effective informal evaluation tool.

In the fall of 2011, Big Ideas@Berkeley began conducting follow-up phone calls with the group of 2010 winners. During these phone calls, Big Ideas staff asked past winners if they were still working on their projects, what progress they had made to date, if they had won any additional contests or grants, and their plans for future work. These phone calls allowed Big Ideas to keep up-to-date with winners’ stories, which have been used in Big Ideas@Berkeley newsletters, in pitching Big Ideas@Berkeley to potential category sponsors, and as informal evidence of the impact of the Contest in grant proposals. Keeping in touch with past winners has also allowed staff to develop a greater sense of connection to and commitment from past winners to the Contest.