#1THE WORLD BANK IMPACT EVALUATION Evaluating Impact: Turning Promises into Evidence Third Pakistan Poverty Alleviation Fund Project Tanvir Hussain/PPAF, Champak Prasad Pokharel/NES, Syed Hassan Akbar/PPAF Christel Vermeersch/World Bank Kevin Crockford, Imtiaz Alvi/World Bank Kathmandu, Nepal February 2010#2IMPACT 1. Context EVALUATION □ Nine years of Pakistan Poverty Alleviation Fund (PPAF) operations (total US$900 mill). Third 5- year project started in 2009 □ Strategy and approaches continue to evolve over time □ Strong demand from Government of Pakistan, PPAF management and WB to show results not just project outputs □ Increasing number of donors and sources of financing 2#32. Background to the Project IMPACT EVALUATION □ Objective: To empower the poor with increased incomes, improved productive capacity and access to services to achieve sustainable livelihoods □ Components: ■ Social Mobilization and Institutional Building ■ Livelihood Enhancement and Protection ■ Micro-credit Access ■ Basic Services and Infrastructure □ Outreach: 126 districts, 51,000+ villages □ Partners: 82 partner organizations 3#43. Results Chain Inputs IMPACT EVALUATION Activities Outputs Outcomes Longterm Outcomes • Project funds •Technical Identification and appraisal of partner organization •Revitalized and new Community Poor community institutions organizations empowered assistance •Capacity Following Evaluation of • 200 development at livelihood proposals community and PPAF Staff .82 from partners and financing partners Technical capacity building of partners Monitoring and evaluation opened and renovated individual levels •No. of loans available •Productive asset transfers to poor •Community physical infrastructure built •Community health centers and schools assets for the poor increased: • Financial Social • Physical • Human • Natural empowered with increased incomes, improved productive capacity and access to services to achieve sustainable livelihoods 4#5IMPACT 4. Primary Research Questions EVALUATION □ Has the project been able to affect household income, consumption and production? □ Has the project succeeded in enhancing the following assets of its target beneficiaries: Financial, Physical, Human, Social, Natural □ Has the project been able to increase access to basic services and infrastructure? □ Has the project been able to improve community level institutions in terms of their social capital, empowerment, and governance? □ Has the project been able to reduce vulnerability and improve food security and resilience to external shocks? 5 UT#6IMPACT 5. Outcome Indicators EVALUATION □ Household income, consumption, food consumption and household assets □ Agricultural production and non-farm enterprises (gender disaggregated) □ Access to drinking water, roads, sanitation, irrigation schemes, and education and health facilities □ Linkages and access to other public and private service providers □ Participation of women in community institutions □ Participation of poor in community institutions This exercise will focus on Research Question 1 and the first two clusters of outcome indicators Existing evaluation looking at infrastructure and social mobilisation 6#7IMPACT 6. Identification Strategy/Method □ Basic sampling unit: Settlement EVALUATION □ Preferred Option: Randomly selected treatment and control settlements, based on phasing. □ Option 2: Treatment settlements pre-determined and matching control villages to be identified (propensity score matching). □ Several baseline and impact evaluations staggered over 8 cohorts, households interviewed at 3 points of time. □ Difference-in-difference method for analysis, use of IV for any possible contamination. □ Results augmented with monitoring reports and data. □ Poverty scorecard administered in each settlement. 7#8IMPACT 7. Sample and data EVALUATION □ A representative sample taking care of geographic spread, partner diversity, financial weight of interventions, and multitude of components and interventions is a challenge. □ Sample to be stratified to ensure adequate representation of each cateogry. □ Sample is expected to provide statistically valid results at the project level. □ Based on final stratificaton, 95% confidence level, and ±3% error margin, a representative sample of settlements and households to be selected. □ Based on simulation, an estimated 4,000 treatment (and 4,000 control) households will be required. 8#98. Time Frame/Work Plan □ Timeline of evaluation surveys IMPACT EVALUATION Baseline 1 Baseline 2 Baseline 3 Baseline 4 Baseline 5 Baseline 6 Baseline 7 Baseline 8 Mid-Term 1 Mid-Term 2 Start FY 2009-10 FY 2010-11 Mid-Term FY 2011-12 FY 2012-13 FY 2013-14 End FY 2014-15 01 02 03 04 01 02 03 04 01 02 03 04 01 02 03 04 01 02 03 04 01 End-of-Project 1 End-of-Projea 2 Mid-Term 3 End-of-Projea3 Mid-Term 4 End-of-Project 4 Mid-Term 5 End-of-Projeas Mid-Term 6 End-of-Project 6 Mid-Term 7 End-of-Project 7 Mid-Term & End-of-Project 8 Treatment and control groups for each cohort of baseline, mid- term and end-of-project evaluation, cohort fixed effects. □ Case studies to be conducted at various points of time. □ User/beneficiary surveys to be conducted at the end of each financial year. 6#10IMPACT 9. Sources of Financing EVALUATION □ A portion of the financing from project funds □ Seek remaining funds from national, regional, and global sources Existing MoU with premier academic institutions. □ Similar linkages to be established with international institutions seeking partnership 10