The Monitoring and Evaluation (M&E) policy at the project level in UNDP/GEF has four objectives: - to monitor and evaluate results and impacts;
- to provide a basis for decision making on necessary amendments and improvements;
- to promote accountability for resource use;
- to document, provide feedback on, and disseminate lessons learned.
In accordance with UNDP/GEF M&E policies and procedures, all regular and medium-sized projects supported by the GEF should undergo a final evaluation upon completion of implementation. Final evaluations are intended to assess the relevance, performance and success of the project. It looks at early signs of potential impact and sustainability of results, including the contribution to capacity development and the achievement of global environmental goals. It will also identify/document lessons learned and make recommendations that might improve design and implementation of other UNDP/GEF projects. Brief project description The Polish Energy Efficient Motors Programme (PEMP) project was aimed to overcome barriers for increased market penetration of energy efficient motors and related efficiency improvements in the electric motor system (including variable speed drives), particularly, but not exclusively, in the manufacturing industry, the energy sector (heating), the utility sector (water supply and sewage treatment) and mining. Under the programme four components were developed and implemented during 2004 - 2009: a) Capacity building by providing information and services related to energy efficient electric motors; b) Implementation of four demonstration projects for efficient motor systems in different key sectors; c) Market transformation, using a financial incentive mechanism coupled with awareness raising; and d)Development of industrial energy efficiency policy, particularly in the field of efficient motor systems. PEMP was expected to achieve has established the following specific targets: - Increase energy efficient motor sales in Poland to app. 15% of the total motor market as a direct result of PEMP during the duration of the programme (five years);
- Increase the efficiency of electric motor systems by increasing the penetration of variable speed drives in combination with energy efficient motors;
- Achieve a medium-term increase of energy efficient motors sales of 30% of the total motor market in the year 2012;
- Save electricity by promoting the optimization of electric motor systems, including the implementation of energy efficient motors and variable speed drives to a level of 55.7 GWh/year in 2007 and 231.6 GWh/year in 2012;
- Reduce domestic GHG emissions by 832 kton CO2 by 2007 (directly attributable to PEMP and cumulative over the project lifetime), and 3.7 Mton CO2 by 2012, including the medium-term impact (cumulative over the lifetime of the investments).
|
Objectives of the evaluation: The evaluation should assess: Project formulation: The evaluator will assess the project concept and design. He/she should review: - The problems addressed by the project and the project strategy, encompassing an assessment of the appropriateness of the objectives, outputs, activities and inputs as compared to cost-effective alternatives;
- The extent to which the project idea had its origin within national, sectoral and development plans and focuses on national environment and development interests;
- Information dissemination, consultation and stakeholder participation in design stages;
- The ways in which lessons and experience from the project were or are replicable or scaled up in the design and implementation of other projects.
Implementation The evaluation will assess the implementation of the project in terms of quality and timeliness of inputs and efficiency and effectiveness of activities carried out. He/she should review: - Implementation Approach, including an analyses of the project's logical framework, adaptation to changing conditions and overall project management;
- The quality and timeliness of monitoring and evaluation of the project;
- Stakeholder participation in the project, specially – information dissemination, NGOs and local resources users in the implementation, the establishment of partnerships and relationships developed by the project with local, national and international entities and involvement of governmental institutions;
- Financial Planning, including an assessment of the actual project cost by objectives, outputs, activities, the cost-effectiveness of achievements, financial management and co-financing;
- Sustainability – extent to which the benefits of the project will continue, within or outside the project area, after project closure;
- Execution and implementation modalities.
Project outputs, outcomes and impact The evaluation will assess the outputs, outcomes and impact achieved by the project as well as the likely sustainability of project results (including review of GHG emissions reductions calculations from the project ). This should encompass an assessment of the achievement of the immediate objectives and the contribution to attaining the overall objective of the project. The evaluation should also assess the extent to which the implementation of the project was inclusive of relevant stakeholders and to which it was able to create collaboration between different partners. The evaluation will also examine if the project has had significant unexpected effects, whether of beneficial or detrimental character. Evaluation product: The evaluator will produce an evaluation report with findings, recommendations, lessons learned, and rating on performance. The report (in English) should include: Executive summary: - Brief description of project;
- Context and purpose of the evaluation;
- Main conclusions, recommendations and lessons learned
Introduction: - Purpose of the evaluation;
- Key issues addressed;
- Methodology of the evaluation;
- Structure of the evaluation
The project(s) and its development context: - Project start and its duration;
- Problems that the project seek to address;
- Immediate and development objectives of the project;
- Main stakeholders;
- Results expected.
Findings and Conclusions: In addition to a descriptive assessment, all criteria marked with (R) should be rated using the following divisions (according to GEF Project Review Criteria): Rating: - HS-Highly Satisfactory
- S- Satisfactory
- MS-Marginally Satisfactory
- U- Unsatisfactory
- NA-Not applicable
Project formulation: - Conceptualization/Design (R) – it should assess the approach used in design and an appreciation of the appropriateness of problem conceptualization and whether the selected intervention strategy addressed the root causes and principal threats in the project area. It should also include an assessment of the logical framework and whether the different project components and activities proposed to achieve the objective are appropriate, viable and responded to contextual institutional, legal and regulatory settings of the project. It should also assess the indicators defined for guiding implementation and measurement of achievement and whether lessons from other relevant projects (e.g., same focal area) are incorporated into project design;
- Country ownership/Driveness – must assess the extent to which the project idea had its origin within national, sectoral and development plans and focuses on national environment and development interests;
- Stakeholder participation in the project (R), specially – information dissemination, NGOs and local resources users in the implementation, the establishment of partnerships and relationships developed by the project with local, national and international entities and involvement of governmental institutions;
- Replication approach – it would be crucial to determine the ways in which lessons and experiences coming out of the project were/are to be replicated or scaled up in the design and implementation of other projects;
- Cost-effectiveness;
- UNDP comparative advantage as IA of the project;
- Linkages between project and other interventions within the sector;
- Management arrangements.
Implementation: Implementation approach (R) – it should include assessment of: - The use of the logical framework as a management tool during implementation and any changes made to this as a response to changing conditions;
- Other elements that indicate adaptive management such as comprehensive and realistic work plans routinely developed that reflect adaptive management and/or; changes in management arrangements to enhance implementation;
- The project's use/establishment of electronic information technologies to support implementation, participation and monitoring, as well as other project activities;
- The general operational relationships between the institutions involved and others and how these relationships have contributed to effective implementation and achievement of project objectives;
- Technical capacities associated with the project and their role in project development, management and achievements.
Monitoring and evaluation (R) – especially their quality and timeliness; Stakeholder participation (R) – it should include: - The production and dissemination of information generated by the project;
- Local resource users and NGOs participation in project implementation and decision making and an analysis of the strengths and weaknesses of the approach adopted by the project in this arena;
- The establishment of partnerships and collaborative relationships developed by the project with local, national and international entities and the effects they have had on project implementation;
- Involvement of governmental institutions in project implementation, the extent of governmental support of the project.
- Financial Planning, including an assessment of the actual project cost by objectives, outputs, activities, the cost-effectiveness of achievements, financial management and co-financing and Leveraged Resources;
- Sustainability – the extent to which the benefits of the project will continue, within or outside the project domain, after it has come to an end;
- Execution and implementation modalities – it should consider the effectiveness of the UNDP counterpart and Project Co-ordination Unit participation in selection, recruitment, assignment of experts, consultants and national counterpart staff members and in the definition of tasks and responsibilities.
Results: - Attainment of Outcomes/Achievement of objectives (R);
- Sustainability – appreciation of the extent to which the benefits of the project will continue, within or outside the project area, after project closure;
- Contribution to upgrading skills of the national staff.
Recommendations: - Corrective actions for the design, implementation; monitoring and evaluation of the project;
- Actions to follow up or reinforce initial benefits from the project;
- Proposals for future directions.
Lessons learned: - Best and worst practices in producing outputs, linking them to outcomes and using partnerships strategically
- Annexes:
- TOR
- Itinerary
- List of persons interviewed
- Summary of field visits
- List of documents reviewed
- Questionnaire used and summary of results
- Co-financing and Leveraged Resources
- Manual for calculating GHG benefits of GEF projects
Evaluation methodology: The evaluation will be based on the study of documents and interviews with the key persons involved in the project, i.e. representatives of the implementing agency, UNDP project staff, the Project Coordinator and other involved municipalities, the Steering Committee members, as well as other partners, stakeholders and beneficiaries. The evaluators will be provided with basic documentation related to the project, including the project document, summary records of Steering Committee and project reports. The evaluation will be carried out by the Consultant with support from local project staff. The evaluator selected should not have participated in the project preparation and/or implementation and should not have conflict of interest with project related activities. The Consultant will be responsible for preparing and submitting the evaluation report to UNDP. Implementation arrangements: The evaluation mission will be conducted in April, with the following steps: - Desk review - gathering of data, review of documentation (project document, project revisions, reports, and other relevant project documentation);
- Organization of the mission - conducted by the Implementing Agency (scheduling of meetings as agreed with the Consultant, confirming facilities and logistical arrangements) with the assistance of UNDP;
- Mission to Poland - will be conducted by the Consultant. Debriefing meetings for the Expert with UNDP representatives shall be organized on the first and last day of the mission;
- Preparation of the report - the initial findings of the evaluation should be presented as a debriefing to UNDP Poland and the Project Coordinator on the final day of the mission.
Time frame: The evaluation mission will take place in April 2009. The first draft of the evaluation report shall be submitted by 15 May to allow for comments from UNDP and the Project Director. Upon receipt of these comments, the Consultant shall submit the final report by 30 May. The work will require a total of 10 days, comprised of a 5 days visit to Poland and 5 days for preparation and drafting of the report. |
