The Peace Operations Training Institute Monitoring and Evaluation (M&E) Programme examines the effectiveness of the Institute’s programmes. POTI staff collect data over a nine-month period both before and immediately after students complete a course through the Institute, three and nine months following the completion of the course, and through focus groups.
POTI Course Development and Pedagogical Architecture
The Institute develops courses by recruiting recognized experts as authors and partnering with distinguished organizations, UN offices and agencies, and National Peacekeeping Training Centres (NPTCs). Courses developed through such cooperation include three regional courses on Security Council resolution 1325 and the Women, Peace, and Security agenda, produced in collaboration with UN Women; the course Ebola Virus Disease: Awareness and Precautions for Peacekeeping Personnel, produced in cooperation with the World Health Organization (WHO); Mine Action and Explosive Hazard Management, produced in cooperation with the UN Mine Action Service (UNMAS) and which is mandatory training for all incoming UNMAS and UN Office for Project Services staff members; Core Pre-deployment Training Materials, co-produced with the NPTCs of Australia, Chile, Germany, Ghana, and Sweden; and The Role of United Nations Police in Peace Operations, produced in collaboration with the Australian Federal Police.
Most POTI students either come from developing countries or serve on missions where uninterrupted broadband internet access is unavailable, unreliable, or expensive. To accommodate these technical constraints, the Institute delivers courses in a “high-tech/low-tech” manner, maximizing accessibility. To train through POTI, students only need access to the internet twice: once to enrol and download their course in PDF format, and once to take their End-of-Course Examination. This allows students to review course material without an internet connection until they are prepared to take the examination.
Each lesson begins with a set of lesson objectives, and many courses include videos of the course author providing an introduction and key points for each lesson. Students also are able to submit questions directly to authors through their online student classroom. Lessons generally range from 15 to 25 pages in length and conclude with an End-of-Lesson Quiz, usually consisting of 10 questions on information included in that lesson. A key in the course provides students with the correct answers to confirm their mastery of the material, and students can review any sections that require additional study. When students have completed all lessons in a course and feel sufficiently prepared, they may attempt the End-of-Course Examination. The Institute maintains item banks of approximately 100 questions for each course and draws 50 questions in random order for each attempt. This generates a unique exam for every student worldwide (for the Ebola Virus Disease course, the exam has fewer questions to reflect a shorter course). Students must earn a minimum final exam score of 75 per cent to receive a Certificate of Completion.
Methodology for Monitoring and Evaluation
The Institute maintains an ongoing Monitoring and Evaluation programme to measure the effectiveness of its programmes. POTI programmes must have a positive and measurable impact on the performance of individuals deployed on peace operations to be considered effective. Although the term “M&E” usually means “Monitoring and Evaluation”, the Institute also defines it as “Measurement of Effectiveness”, meaning measurement of the effectiveness of the training and trained personnel in the field.
The Institute’s M&E programme covers both definitions and allows students to evaluate their experiences with POTI as much as POTI evaluates its impact on students. Our M&E methodology uses seven points of data collection:
- Pre-Course Assessment: Before commencing the course, students complete a 10-question pre-test. This establishes a baseline of the student’s knowledge before training and allows for the comparison of pre- and post-course performance.
- End-of-Course Examination: After students have studied all lessons and completed the End-of-Lesson Quizzes, they attempt a 50-question (20-question for Ebola Virus Disease) End-of-Course Examination.
- Course Feedback: After successfully completing the End-of-Course Examination with a score of 75 per cent or better, students are prompted to fill out a 17-question feedback questionnaire to provide their evaluation of the course.
- Five-Star Course Review System: Once a student completes a course, they can rate that course through a five-star rating system accessed through the Institute’s website. Students also can post comments that others are able to see before enrolling. POTI does not edit or redact these comments.
- Three-month follow-up survey: Students receive a 13-question follow-up survey three months after completing their enrolment in a POTI course. It includes multiple choice, multiple selection, short-answer, and long-answer questions.
- Nine-month follow-up survey: The Institute sends a six-question survey to students nine months after they complete their enrolment.
- Focus groups: Selected students participate in a focus group conducted via Skype and respond to nine questions asked by POTI staff. Responses are recorded and analysed for trends.
The Five-Star Course Review System
The Institute uses the course rating system to make potential students aware of the opinions and written reviews of past students. Once they complete a course, students have the opportunity to submit a review of that course by rating their experience on a scale of one to five stars and may also provide written comments. These are automatically published on the Institute’s website as course reviews. POTI does not censor or edit these reviews, but it does moderate its online community for inappropriate or irrelevant submissions. To date, course ratings submitted by students have been unproblematic and constructive contributions to our virtual community. To view the ratings for any course, visit the course page and select any course title. Average star ratings and all reviews are available from any course’s unique page.