Overall Framework

Throughout the duration of the program, a comprehensive evaluation framework has been employed. The overall approach has been utilization focussed evaluation involving primary stakeholders for the program such as management, staff, funders, students and teachers at each stage of the process. During the initial phases of the program in 2002/2003 an evaluability assessment and developmental evaluation were conducted during which primary stakeholder provided insights into the best uses of the power of peers, and established a rigorous, and testable program logic model. Formative evaluations have been conducted each time new materials have been developed ensuring that core messages remain strong and relevant to our target audience. Process evaluations have been conducted on each year’s training sessions, and subsequent performance by teams in each school to monitor the ongoing implementation of the program as designed, as well as ensuring continuous quality improvement. Short term impacts and long-term outcomes have been assessed through a mixture of quantitative and qualitative methods, focussing on a number of areas including: benefits to primary participants (youth leaders, staff advisors and core team members); benefits to all students at participating schools; other benefits to the participating schools; and benefits to the larger communities within which the schools reside.

Impacts and Outcomes

Short term impacts of the program have been assessed during the pilot phase of the program through surveys of samples drawn from the student bodies of participating schools. These surveys showed significant reach through awareness of the program’s presence, recognition of interventions run, and identification of messages provided during the course of student led activities in the schools. Outcomes for both primary participants and the student body as a whole have been assessed in the pilot schools (and in most subsequent No Regrets schools) using pre-post surveys of knowledge, attitudes and behaviours.

In general the findings have been consistent:

• Primary participants gain knowledge of the magnitude of the injury issue and SMARTRISK’s core messages for prevention. They show shifts in attitude toward increased personal responsibility for risk taking and injury prevention, and shifts in behavioural intention around a few key preventive behaviours.
• All of these gains are maintained, with some decay, when follow-up surveys are conducted after a delay of six months or more. Importantly, students report that they have followed through with at least some positive behavioural intentions, by actually adopting some preventive behaviours.
• Similar, though somewhat smaller gains are seen in samples surveyed from the entire student population, in participating schools.
• In the pilot phase, a sixth school was unable to participate in the program but agreed to take the surveys and thus served as a non-equivalent comparison group. This school did not show significant changes in knowledge, attitudes or behaviour over the initial pilot phase.
• In the most recent surveys administered to a sample of school populations in 2006/07 a new set of questions asking for students’ experiences of being injured was introduced. The results indicated a 17% reduction in self reported injuries over a six month period, when the post-test was administered (over one year into the program) compared with the pre-test (administered prior to the first no regrets activities in the schools).

Additional school level benefits as well as community level benefits have been assessed in a number of ways including key informant interviews with school administrators, staff advisors and in recent years with community partners. Examples of the benefits to schools noted by these key informants included:

• Provision of interesting assemblies and events that held the attention of the majority of students (e.g. Snowbirds, workplace safety speaker, avalanche expert, Dr. Robert Conn);
• Actual use of the five SMARTRISK injury prevention strategies and the “Stupid Line” concept in student conversations and class discussions heard around the school;
• Actual behaviour changes being observed among students who are now practicing some of the five SMARTRISK injury prevention strategies (e.g.; Wear the Gear, Buckle Up);
• Students approaching No Regrets team members to ask questions and discuss injury prevention issues and some students are now looking forward to No Regrets events and are requesting that more No Regrets events be staged;
• Engagement of some sub populations of the schools that did not participate in other school activities/events;
• Increased awareness as to the severity of the injury issue for Canadian youth and wide dissemination of injury prevention knowledge and strategies to students and staff in the schools;
• A positive view of injury prevention and risk management;
• The enhancement of the schools’ reputations in their communities and in their school boards/districts (e.g., “the local media is paying more ‘positive attention’ to our schools”);
• The positive gains in reputation that resulted from the identification of the schools as partners with a nationally-renowned injury prevention organization (e.g., “having information on our school posted on the SMARTRISK Navigator website”);
• The facilitation of linkages with other schools in the community for the purposes of promoting injury prevention and delivering injury prevention activities and events;
• The facilitation of linkages with local injury prevention organizations and coalitions;
• The opportunity to link with local business organizations in order to sponsor school-initiated activities

Examples of the benefits to communities noted by these key informants included:

• participation of students from other schools in the SMARTRISK Heroes performances;
• publication of information on positive aspects of the project in the local media;
• direct assistance to other schools in their communities in promoting injury prevention and risk awareness;
• direct delivery of injury prevention activities to other schools in their areas; and, complementing the activities of the student councils and other injury prevention groups on campus (such as Students Against Drunk Driving).
• Furthermore, they believed that other high schools in their communities would benefit from participation in the project at some point in the future.

Dissemination of Results

In addition to profiling the SMARTRISK No Regrets program in several articles on the SMARTRISK website and in our publications, SMARTRISK presented on topics related to the SMARTRISK No Regrets program at the following conferences: Canadian Injury Prevention and Safety Promotion Conference, Halifax (2005); Canadian Public Health Association Conference, Ottawa (2005); Ontario Injury Prevention Conference, Toronto (2006); Ontario Injury Prevention Conference (2007), Canadian Injury Prevention Conference (2007). Papers for publication are in production. Full evaluation reports are available on request.