Assignment #1 ECUR 809- Looking at Program Evaluation

Mid-Term Evaluation- First Nations SchoolNet Program:

Final Report

http://www.ic.gc.ca/eic/site/ae-ve.nsf/vwapj/FirstNationsSchoolNetFinalReport.pdf/$file/FirstNationsSchoolNetFinalReport.pdf

The First Nations SchoolNet Program was an initiative created by the Federal government for the purpose of connecting federally funded First Nations schools through the Internet and providing the skills in Information and Communications Technologies (ICT). The program initiative began in 2000 by first connecting the school with Internet by providing the necessary infrastructure.  From 2004-2006, the program was granted $15 million a year to increase connectivity with high speed internet, as well as computers and ICT.  The program was facilitated by various organizations on a regional level who were to provide reports to Industry Canada (Federal Government). 

Malatest and Associates Ltd. was the firm hired by Industry Canada to conduct the evaluation of the First Nations SchoolNet Program.  The model they used in their evaluation was Scriven’s model.  The evaluation was conducted as a mid-evaluation to determine whether or not the initiative conducted by the government was going as planned and what modifications needed to be made so the necessary improvements could be made.  Therefore, the approach taken was formative. 

A strength in this evaluation is the simplicity of the discourse used.  Being an educator, I have some previous background knowledge in the field so this gives me some foundation in understanding the context of the initiative.  However, I found that reading the report was fairly easy to understand.  It didn’t contain a bunch of legal jargon or language that was too difficult to comprehend.  As a result, I was able to gain a good understanding of the initiative itself, as well as the process used in evaluating the program.

Another strength in the evaluation was the thoroughness of the development and pretesting of the questionnaire.  The consultant developed necessary questions that pertained to each of the stakeholders involved making the questionnaire relevant to them.  The questionnaires were then sent to a committee for finalization and then pretested. Once the questionnaires were passed in the pre-testing field, the Consultant translated the questionnaires for two stakeholders into French.  The Consultant also provided an introductory letter to explain the questionnaire as well as frequently asked questions.  To me this demonstrates that all factors are taken into account, so that the results can be obtained in a most convenient and efficient manner.   Evaluating a program is a big job, so it makes sense that the one tool that potentially could bring in the most valuable information requires much attention in the development stage.

Although I don’t have any experience in determining the strengths and weaknesses of programs that are evaluated, I do think that I identified a weakness in this particular evaluation.  I don’t think that the Consultant interviewed enough participants- particularly those who are most closely connected to the program.  The report indicated that there are 567 schools that are receiving the program, each of which are run by administrators.  In the evaluation, the Consultant was identified as interviewing a total of 19 administrators, 11 of whom were principals, four technical administrators and four directors of education.  I found that when reading the results, the Consultant was provided with the most valuable information, and yet of the 55 people interviewed, only 19 were directly connected to the schools.  If it were me conducting the interview, I would interview more people directly connected to the schools, such as teachers, students and community members.  The Consultant did look at two case studies, where 16 people including teachers, administrators and suppliers were interviewed.  Although lots of relevant information was collected, it was at two schools in similar regions (Manitoba and Ontario).  I believe that if an agency is looking to see whether or not the goals of the program they are implementing are effective, as many stakeholders possible should be interviewed.  Therefore additional case studies should have been examined.  I also think that they are missing the component of student voice.  They are the reason that the program is being implemented in the first place, so their opinions about the effectiveness should be included.  Because Industry Canada was looking to gain a better understanding of the progress and challenges of the program, more people should have been interviewed. 

Overall, I feel that the Consultant did a good job in providing the information that Industry Canada was looking for.  The format and language used was clear and could be understood by a variety of audiences.  The Consultant demonstrated professionalism in making sure the questionnaires were relevant and could provide as much information as possible.  Although I believe more stakeholders could have been interviewed, the Consultant was still able to provide many relevant recommendations to improve the management, delivery, progress and challenges of the program.