Kosovo - Reliable Energy Landscape Project Evaluation

Metadata Updated: March 10, 2021

The objective of RELP is to reduce the gap between energy demand and supply, through three main activities: subsidies for energy efficiency in Kosovo (SEEK), district heating and metering (DHM), and independent power producers (IPPs). On the demand side, RELP will address two barriers to investments in energy efficiency: a general lack of consumer awareness about energy saving measures and their benefits, including the benefits of metering and other energy saving devices, and the inability of poor households to pay for energy efficiency measures. These objectives will be achieved by lowering energy use through piloting household investments in energy efficiency, metering existing district heating consumers, and switching new consumers to cost-effective district heating. On the supply side, RELP will aim to reduce barriers to renewable energy entrants to the market by stimulating the business environment. In the original design of RELP, the women in energy efficiency (WEE) activity was a sub-activity of SEEK. As of July 2020, it was decided to treat WEE as a stand-alone activity given its objective of promoting female employment in the energy sector.

The evaluation of RELP is organized around the MCC RELP theory of change whose overarching objective is to reduce the gap between energy demand and supply. The evaluation will test the causal links in the theory of change using qualitative and quantitative data to determine whether the activities and inputs outlined in the RELP theory of change affect short- and medium-term beneficiary-level outcomes. In order to understand if the main objective-level outcome was achieved, the RELP evaluation will include descriptive analysis of broad trends and demographics of project beneficiaries and a combination of performance and impact evaluations. The proposed evaluation design incorporates findings from evaluator's (IMPAQ) scoping mission and subsequent conversations with all relevant stakeholders. Quantitative components of both performance and impact evaluations will use primary data, administrative data, and monitoring and evaluation data. Qualitative components will incorporate information from document reviews, key informant interviews (KIIs), and focus group discussions (FGDs) and will provide additional context to understand the quantitative impacts. Furthermore, as part of qualitative data collection, IMPAQ will conduct women-only focus groups and ensure that other key marginalized groups are well represented to understand their experiences. Finally, the evaluation will also include the evaluation of the WEE activity, which is not linked to the main objective-level outcome of reducing electricity demand and supply, but instead focuses on increasing female employment in the energy sector. The research questions that will guide the evaluation of RELP activities are briefly outlined below:

  1. Were the activities implemented as designed and were the outputs outlined in the project logic achieved?
  2. Did the SEEK activity result in reduced electricity consumption in the intervention areas? What was the contribution of various components of the program toward any changes in energy consumption (technical assistance to key institutions, BC&O activities, installation of energy efficient equipment)? Were energy savings previously envisioned during preliminary baseline studies and energy audits achieved, and why?
  3. Did the SEEK activity identify a cost-effective model for reducing household energy use that could be scaled up nationwide and what is the overall expected energy reduction?
  4. Did the DHM activity result in a change in energy consumption in the intervention areas? What was the contribution of various components of the program toward any changes in energy consumption (technical assistance to key institutions, BC&O activities, and implementation of consumption-based billing)? How many new households shifted to district heating as a result of the DHM activity?
  5. Was the IPP activity successful in reducing the perception of financial risk for investors and commercial lenders in the energy sector, and why? Did the IPP activity contribute to increasing the electricity supply in Kosovo? If so, by how much and why?
  6. How effective were program activities at increasing women's employment in the energy sector and at increasing investments in energy efficiency among women entrepreneurs?
  7. Did the program meet the stated objective of reducing the gap between supply and demand of electricity? How sustainable are critical outcomes of the program, and why?

We briefly detail below the evaluation approach for each activity under RELP:

  1. SEEK (Subsidies for Energy Efficiency in Kosovo) Activity

HER (Household Energy Retrofits) sub-activity The primary quantitative approach to estimate the causal impact of the HER sub-activity on electricity consumption using the quasi-experimental ITS method. This method will involve analysis of high-frequency (monthly) household-level administrative data on electricity consumption from the Kosovo Energy Distribution Services (KEDS). We will complement the ITS method with a pre-post approach using baseline and endline primary quantitative survey data to provide additional analyses on consumption from other energy sources. The evaluation will also use qualitative data through FGDs and KIIs to provide an in-depth understanding of behavior change. In addition to baseline and endline, we will collect qualitative data in the interim for continuous monitoring and to develop a deeper understanding of effects.

AER (Apartment Energy Retrofits) sub-activity Similar to the HER sub-activity, we will use a combination of and ITS approach and a pre-post approach to understand the effect of this sub-activity on energy consumption in individual apartment units and the common areas of MABs. We will use an ITS approach to understand the impact on the main variable of interest: electricity consumption. The pre-post approach will complement the ITS analysis and provide evidence on the mechanisms of change. All quantitative findings in the AER sub-activity will be complemented with a qualitative case study approach, to provide more nuanced understanding of changes in energy consumption at the MAB level. Qualitative data will be collected at baseline and endline, in addition to three rounds of interim data for continuous monitoring. Finally, we will also conduct a CEA for AER, measuring cost-efficiency at the MAB/apartment level.

  1. DHM (District Heating Metering) Activity We propose to estimate the causal impact of the DHM activity on the key outcome of interest-district heating consumption-through the ITS method, using high-frequency (monthly) administrative data for each beneficiary household from Termokos. Additionally, we will use a pre-post approach that will utilize primary survey data collected at baseline and endline to understand the impact of DHM on consumption of energy from other sources, including electricity. Quantitative data collection will be accompanied with qualitative data collection at baseline, endline, and interim periods to understand how different program components affect district heating consumption after the installation of meters and thermal valves. Qualitative data will also help contextualize the quantitative findings.

  2. IPP (Independent Power Producer) Activity We will implement a process-tracing approach to assess the impact of the IPP activity. Our approach will be primarily qualitative, analyzing data collected through (a) document review and monitoring and evaluation data; and (b) semi-structured KIIs and small group interviews. We will track the progress, successes, and challenges of the major implementation steps. The timeframe of our evaluation is too short to examine whether the IPP activity ultimately contributes to increasing the electricity supply in Kosovo. However, we will assess whether the IPP activity was successful in reducing the perception of financial risk for investors and commercial lenders in the energy sector. We will also qualitatively assess whether investors and lenders are more willing and able to support IPP project development.

  3. WEE (Women in Energy Efficiency) Activity The implementation and effectiveness of this sub-activity will be analyzed primarily through qualitative data from semi-structured KIIs and FGDs. Semi-structured KIIs will include a short quantitative component as well. We will also use document review and monitoring and evaluation data. Implementation will be assessed by analyzing whether program targets and outputs have been achieved as per the initial plan, and if any identified challenges were overcome. Through semi-structured KIIs and FGDs with beneficiaries, the effectiveness of program activities will be assessed. Additionally, we will use secondary data to assess the program's influence on women's academic and employment goals and access to job opportunities in the energy sector. In the case of entrepreneur grants, we will determine whether grants led to more efficient use of energy, lowered energy costs, and improved profit margins.

Access & Use Information

Public: This dataset is intended for public access and use. License: See this page for license information.

Downloads & Resources

Dates

Metadata Created Date March 10, 2021
Metadata Updated Date March 10, 2021

Metadata Source

Harvested from MCC Data.json

Additional Metadata

Resource Type Dataset
Metadata Created Date March 10, 2021
Metadata Updated Date March 10, 2021
Publisher Millennium Challenge Corporation
Unique Identifier Unknown
Maintainer
Identifier DDI-MCC-KOS-IMPAQ-RELP-2020-V01
Data Last Modified 2021-03-09
Public Access Level public
Bureau Code 184:03
Metadata Context https://project-open-data.cio.gov/v1.1/schema/catalog.jsonld
Schema Version https://project-open-data.cio.gov/v1.1/schema
Catalog Describedby https://project-open-data.cio.gov/v1.1/schema/catalog.json
Homepage URL https://data.mcc.gov/evaluations/index.php/catalog/273
License https://data.mcc.gov/terms-and-conditions.php
Program Code 184:000
Source Datajson Identifier True
Source Hash 372569b5cdcc919fc85fd42805acef3f3b715b63
Source Schema Version 1.1

Didn't find what you're looking for? Suggest a dataset here.