• B/H/S
  • ENG
Objave

TERMS OF REFERENCES – BEEPAF-P165405-CQ-11-CS-20-FBIH

11.09.2020.

BOSNIA AND HERZEGOVINA
FEDERATION OF BOSNIA AND HERZEGOVINA
ADDITIONAL FINANCING FOR ENERGY EFFICIENCY PROJECT–ID P165405

TERMS OF REFERENCES

SOCIAL MONITORING AND EVALUATION
(INCLUDING SATISFACTION SURVEYS AND AWARENESS SURVEYS) FOR THE IMPLEMENTATION OF THE ENERGY EFFICIENCY IMPROVEMENTS IN PUBLIC BUILDINGS

Reference No.: BEEPAF-P165405-CQ-11-CS-20-FBIH

  1. Background:

The Government of Bosnia and Herzegovina (BH) has recognized the importance of energy efficiency (EE) to support sustainable economic growth and move towards EU accession and has received financing for the Additional Financing for the Bosnia and Herzegovina Energy Efficiency Project (BEEP) from the International Bank for Reconstruction and Development (IBRD) credit funds. The project development objective is to demonstrate the benefits of energy efficiency improvements in public sector buildings and support the development of scalable energy efficiency financing models. The AF BEEP became effective March 2020.

The project is supported by a US$32 million IBRD credit for BH, which is made available to the two entities, with US$ 19.23 million allocated to the Federation of Bosnia and Herzegovina (FBH). The project consists of three components implemented separately in each entity:

Component 1: Energy efficiency investments in public facilities
Component 2: Support for the development of scalable financing mechanisms and capacity building
Component 3: Project Management

The project implementation unit (PIU) established within the Federal Ministry of Physical Planning (FMPP) is responsible for the preparation, coordination, management and implementation of the project in the Federation of Bosnia and Herzegovina, including procurement, contracting, and payments of all goods, works and services related to the project.

  1. Objectives

The Federal Ministry of Physical Planning (FMPP) as the Client will contract the Consultant(s), who will perform a technical assistance for Social Monitoring and Evaluation (SM&E) related to the outcomes achieved through implementation of the energy efficiency works of selected buildings from the annual list prepared by the PIU and adopted by Project Steering Committee (PSC). The Social Monitoring, including inter alia surveys conducted before and after retrofits should cover a sample of the total number of buildings to be financed through the AF BEEP including 16 buildings) in a period up to 4 (four) years. The selected sample of buildings should be representative in terms of: use, number/type of users, climatic conditions and geographical location. Beside Social Monitoring that includes satisfaction and awareness surveys, the Consultant(s) will also design and perform two performance evaluations (mid-term and exit). The Social Monitoring and the Performance Evaluations are independent tasks

The mid-term evaluation, intended to assess the AF BEEP’s obstacles and lessons learned from the first two years of the Project implementation, implementation progress towards specified results, and to provide recommendations on the operational logic and implementation processes adjustments, if needed. The mid-term performance evaluation will identify immediate consequences produced by the project after the second year of its implementation, such as demonstration effect of energy efficiency improvements, identification of reforms needed for improvement of the BiH energy financial sector and is the Project on the right track to ensure scalability of energy efficiency financial models in BiH,
The Federal Ministry of Physical Planning (FMPP) will use evaluation findings, conclusions, and recommendations to reassess the achievements of the Project and readjust the Project implementation, if needed.
The exit performance evaluation will be conducted at the end of the project, in order to:

The data for the performance evaluations will be collected through:

The Consultant is expected to design detailed work plans for the two performance evaluations (mid-term and exit). that will at minimum describe:

Performance evaluations should provide an insight into following: how the project has been implemented against the assumed operational logic; what  are the project’s achieved results related to improved energy efficiency in the public sector buildings, increased scale of the energy efficiency financing models, identification of reforms needed to improve the BiH’s energy financial sector, and contribution of the Project to increase of the overall level of energy efficiency in BiH; how it is perceived and valued by the beneficiaries and stakeholders; and other questions that are pertinent to design, management, and operational decision-making. The intention is to provide information that is credible and useful, enabling lessons learned to be incorporated into the decision-making process of PIU/FMPP. Evaluation should take a broader view of an intervention asking if progress toward the target of explicit results is caused by the intervention or there is some other explanation.

The Consultant(s) will refer to the Federal Ministry of Physical Planning (FMPP) for all issues that could appear in the execution of the task, and for decisions that would have to be made in reference to the present contract. Federal Ministry of Physical Planning (FMPP) will be considered as the Client, even if some actions and/or decisions will have to be coordinated with the relevant Ministries and with the local administration, schools, hospitals, Cantons and Municipalities.

Social Monitoring should be designed in way to clearly indicate:

This work should allow to compare changes ‘pre-implementation’ (i.e., before the works on energy efficiency measures are carried out) and ‘post-implementation’ (i.e., after the works on energy efficiency measures are carried out).

Performance evaluations (mid-term and exit) should be designed in way to clearly indicate:

Performance evaluations should provide an insight into following: what a particular project or program has achieved (at any point during or after implementation); how it is being implemented; how it is perceived and valued; whether expected results are occurring; and other questions that are pertinent to design, management, and operational decision-making. The intention is to provide information that is credible and useful, enabling lessons learned to be incorporated into the decision-making process of PIU/FMPP. Evaluation should take a broader view of an intervention asking if progress toward the target of explicit results is caused by the intervention or there is some other explanation.

The following overall evaluations questions are likely to address specific demands for information related to the following broad areas of inquiry:

  1. Have the right things been done? ( relevance, effectiveness)
  2. Have things been done well? (efficiency, effectiveness)
  3. What results have been achieved? (effectiveness, impact, cost/effectiveness)
  4. How do the results compare with an alternative intervention to achieve the same objective? (relative effectiveness, impact, cost/effectiveness)
  5. How could things be done better in the future?
  6. Are the results sustainable?

The tentative set of evaluation questions have been drafted and are included with this ToR (Annex 1). This annex also includes the draft list of key informants/stakeholder.  The Consultant(s) should amend, develop and complete specific evaluation questions as part of the Evaluation Design and Work Plan.

  1.  Description and Scope of Services:

The Consultant will be responsible for the following:

  1. Social Monitoring
  1. Performance Evaluation

 

 

  1. Preparing periodic reports as needed

As requested by PIU/FMPP the Consultant(s) should prepare different period reports as needed for different type of stakeholders.

  1. Output/Deliverables:

I. Social Monitoring

Report No Deliverables Recipient Deadline
1 Draft Survey and Questionnaire Design FMPP Contract signing + 20 days
2 Inception Report including Final Methodology and Social Monitoring Plan (SMP) FMPP Contract signing + 1 month
3 Field Survey Report (FSR) for each visited building [to be determined later] before and after investment, which will include, but not be limited to: sample type, number of respondents, initial findings etc. FMPP Contract signing + 2 months

Contract signing + 1 month after works completion (preferably in winter season)

4 Draft Social Monitoring & Evaluation (Social Monitoring) Report, summarizing outputs/deliverables 2&3, findings of quantitative and qualitative analysis, statistical annexes etc FMPP Contract signing + 1 month after acceptance of report 3
5 Final Social Monitoring Report FMPP Contract signing + 1 month after acceptance of  report 4

Output 1 – Draft Survey and Questionnaire Design: Before the start of Social Monitoring the Consultant will ensure understanding of the research required through discussions with the representative of the Federal Ministry of Physical Planning (FMPP) and make a review of relevant background materials. The discussions, ahead of preparing the Inception Report, will include:

  1. survey design issues (goals of the research, ultimate use of the results, research approach (cross-sectional, longitudinal)
  2. survey instrument to use (e.g., self-complete questionnaire, interview, focus groups, etc.)
  3. survey administration (e.g., telephone, mail, personal interview, group)
  4. kind of respondents being targeted (ordinary citizens, beneficiaries/ communities of service users, “qualified respondents”, i.e. professionals, public officials etc.)
  5. sample design issues (sample frame, weighting, repetition in future);

To deliver Output 1, the Consultant will draft and submit for approval of the Federal Ministry of Physical Planning (FMPP):

  1. survey design, including the use of focus groups results to feed into survey questionnaire
  2. sample design
  3. questionnaire design
  4. description of fieldwork
  5. standard report formats
  6. agreed number and scale of additional custom reports

Output 2 – Inception Report: At the time of inception report agree with the Federal Ministry of Physical Planning (FMPP) sampling procedure, the maximum number and the actual sites, to the extent possible. The sampling size will cover 25% of the total number of buildings to be financed through the AF BEEP in a period up to 4 (four) years. Incorporation of factors, such us climatic and geographical conditions are obligatory.
To deliver Output 2, the Consultant will conduct the following:

Output 3 – Field Survey Reports (for each visited building before and after investment): To deliver Output 3, the Consultant will do the following:

Output 4 – Draft Social Monitoring Report: To deliver Output 4, the Consultant will:

  1. summary of the methods used including sample design, blank questionnaire, and fieldwork
  2. interpretations, qualitative analyze of the results of the survey
  3. interpretations, quantitative results of the survey, including a section with gender, age, income, education-disaggregated data and analysis. Level of statistical correlation between answers and these dimensions should be reported

Note: Beyond data description, all findings (quantitative and quantitative) must be interpreted in line with the objectives of this Social Monitoring.
Output 5 – Final Social Monitoring Report: To deliver Output 5, the Consultant will:

II. Performance evaluations (mid-term and exit)

Report No Deliverables Recipient Deadline
1. Evaluation Design and Work Plan FMPP Contract signing + 1 month
2 Presentation of preliminary findings FMPP Contract signing + 26 months
Contract signing + 45 months
3 Draft evaluation report FMPP Contract signing + 27
Contract signing + 46 months
4 Final evaluation report FMPP Contract signing + 28 months
Contract signing + 47 months

Output 1 – Evaluation Design and Work Plan
The evaluation design will include: (1) specific evaluation questions; (2) a detailed evaluation design matrix (including the key questions, methods, and data sources to address each question and the data analysis plan for each question); (3) draft questionnaires and other data collection instruments or their main features; (4) the list of potential interviewees and sites to be visited; (5) known limitations to the evaluation design (6) a dissemination plan. The work plan will also include the anticipated schedule. PIU/FMPP are asked to take up to one week to review and consolidate comments. Once the Consultant(s) receives the consolidated comments on the initial evaluation design and work plan, they are expected to return with a revised evaluation design and work plan within 5 days.
Key informant for interviews mid-term performance evaluation will commence two (2) years after the AF BEEP start date. Focus groups, online surveys and other data collection activities will be carried out during the same period.
Key informant for interviews exit performance evaluation will commence three-and-a-half years (3.5) years after the AF BEEP start date. Focus groups, online surveys and other data collection activities will be carried out during the same period.

Output 2 – Presentation of preliminary findings
Presentation of preliminary findings to PIU/FMPP shall discuss the summary of preliminary findings and recommendations to PIU/FMPP.

Output 3 – Draft evaluation report
Draft evaluation report will be consistent with the World Bank Group Evaluation Principles and will be submitted no later than 9 weeks after the start of key informant interviews. Once the initial draft evaluation report is submitted, PIU/FMPP will have 10 calendar days in which to review and comment on the initial draft, and submit the consolidated comments to the Consultant(s). The Consultant(s) will then be asked to submit a revised final draft report in 10 calendar days hence, and again the PIU/FMPP will review and send comments on this final draft report within 5 calendar days of its submission.

Output 4 – Final Evaluation report
The Consultant(s) will be asked to take no more than 10 calendar days to respond/incorporate the final comments from PIU/FMPP and submit the final report. Prepare and present a final Social Monitoring report, considering comments of the FMPP.
The final reports will be prepared in local language and English. They will be forwarded in both three (3) printed copies and an electronic file each suitable to the Federal Ministry of Physical Planning (FMPP) (i.e. Microsoft Word); All statistical tables both those used in the final report and those used to present working or additional results will be forwarded as electronic files suitable to the Federal Ministry of Physical Planning (FMPP), in MS Excel program files. The database of tabulated results will be forwarded in electronic file suitable to the Federal Ministry of Physical Planning (FMPP), exported to MS Excel program files.

  1. Preparing periodic reports as needed

As requested by PIU/FMPP the Consultant(s) should prepare different period reports as needed for different type of stakeholders. The reports will be submitted no later than 3 calendar days after the received request.

5. Time Schedule

The Consultant is expected to start with the activities no later than September2020. with duration of approximately up to 4 (four) years.

6. Qualification requirements

Interested consultants must provide information indicating that they are qualified to perform the services by fulfilling following requirements:

 

  1. Consultant for the Social Monitoring – Team Lead

Duties and Responsibilities
Under the guidance of the Project implementation unit (PIU), the Consultant will be responsible to carry out the following tasks:

The Consultant is expected to travel across BiH during the course of the assignment.

Required Skills, Experiences, and Competencies

 

  1. Consultant for the Social Monitoring – Team Member

Duties and Responsibilities
Under the guidance of the Project implementation unit (PIU), the Consultant will be responsible to carry out the following tasks:

The Consultant is expected to travel across BiH during the course of the assignment.

Required Skills, Experiences, and Competencies

 

  1. Consultant for the Performance Evaluations – Evaluation Team Lead (mid-term and exit)

Duties and Responsibilities
Under the guidance of the Project implementation unit (PIU), the Consultant will be responsible to carry out the following tasks:

The Consultant is expected to travel across BiH during the course of the assignment.

Required Skills, Experiences, and Competencies

 

  1. Consultant for the Performance Evaluations – Evaluation Team Member (mid-term and exit)

Duties and Responsibilities
Under the guidance of the Evaluation Team Lead, the Consultant will be responsible to carry out the following tasks:

The Consultant is expected to travel across BiH during the course of the assignment.

Required Skills, Experiences, and Competencies

 

Annex 1: Draft Evaluation Questions & Draft list of key stakeholders

Draft Evaluation Questions
EQ1:  To what extent has AF BEEP succeeded in measurable way in demonstrating the benefits of energy efficiency improvements in public sector buildings?
EQ2: To what extent has AF BEEP succeeded in developing and disseminating the new scalable energy efficiency financing models?
EQ3: What were the main obstacles in the implementation of the AF BEEP and how were they addressed / the lessons learned?
EQ4: How well is AF BEEP’s implementation coordinated/synchronized with other activities aimed at helping BiH increase the overall level of energy efficiency?
EQ5: Based upon the expert opinions of AF BEEP counterparts and key stakeholders, what priority reforms in the BiH energy financial sector (specifically energy efficiency area) need to be addressed?

Draft list of key stakeholders
List of key informants: