IWPR терміново шукає Final Project Evaluation Consultant, Ukraine

The Institute for War & Peace Reporting (IWPR) is an independent not-for-profit organisation working to empower local voices on the frontline to drive positive change in areas of conflict, crisis or transition around the world. For over 30 years IWPR has been promoting reliable information and public debate to build more stable, just and inclusive societies.  

The EU-funded Boosting Investigative Journalism in Ukraine project (December 2021 – December 2023) had been initially focused on capacity building of the regional investigative journalism centers in Ukraine, but with the outbreak of the full-scale war of Russia against Ukraine it has been partially repurposed to providing emergency assistance to independent media to strengthen their resilience and support vital content production and outreach.

While the overall objective of the project has been to contribute towards greater access of war-affected citizens to reliable information by strengthening editorial and/or organisational capacity of the independent regional media, the specific objectives of the project have been to increase resilience of independent regional investigative journalism centers and media; to increase critical issues coverage by them; and to facilitate their collaboration contributing to individual and organisational capacity and editorial performance.

Objective of the Evaluation

The overall objective of this evaluation is to provide the final review of the Boosting Investigative Journalism in Ukraine project by assessing its performance according to the OECD-DAC criteria and other specific criteria. This final evaluation has three main objectives: 

• To assess and analyse the achievements after 2 years of project implementation, compared to the expected results and in line with the theory of change that underpinned the design and ambitions of the project. This will involve determining the current level of achievement of the project’s indicators (for project objectives and results) and comparing the final achievements with the baseline situation. As for the theory of change, the aim will be to examine the hypotheses of the causal links between actions and results and to confirm or disconfirm the explicit or implicit causal steps of the theory of change.         

• Capture observable changes in practices, behaviour and attitudes adopted by individual actors in the media and civil society sector, by targeted stakeholders including communities, state actors, internationals bodies and any other individuals and structures that are reached directly or indirectly in the activities through direct implementation and sub-granting; highlight the relevant achievements and lessons, with a view to informing the continuation of the project and/or the development of other following projects.
• To assess objectively, quantitatively and qualitatively, the relevance, efficiency, and effectiveness of the project interventions during this time, the sustainability of the results achieved, following the international evaluation criteria of the OECD/DAC as well as the internal and external management and coordination of the project.
• To identify and document good practices / project successes that are recommended for replication  to enable scale up/out into new initiatives.

Evaluation Methodology

The evaluation methodology will be proposed by the consultant in his/her technical proposal. The consultant will have to propose two types of robust methodology to: 

• Test the validity of the project logic (theory of change statement and result framework) and its underlying assumptions, including the analysis of results and outcomes achieved and the logic chain linking interventions to results; 
• Assess the impact, relevance, efficiency, effectiveness, and sustainability of the project interventions, as well as the management of the project including the coherence and coordination within project partners and complementary projects.

Baseline and progress MEL data have been collected and reported quarterly by the project staff in accordance with the donor-approved logframe with relevant indicators. They include periodic project reports, event reports, subgrantee reports with monitoring data, content production lists and outreach data, mentors’ reports etc. IWPR has also piloted a new self-assessment tool IJO Sustainability Index. Please note that  more extended reporting data is available in Ukrainian language. 

The consultant will combine quantitative and qualitative approaches as necessary to adequately address the evaluation objectives and answer the different evaluation questions mentioned below. The evaluation should also assess the gender sensitivity of the project and whether the different needs of women and men were taken into consideration, this includes the collection of sex-disaggregated data. Preferred qualitative evaluation methodologies, largely utilised by IWPR are: 

• Outcome harvesting – where the OH can’t be applied fully, the consultants will be asked to develop outcome statements for significant results achieved.
• Impact stories of change for a selected number of project beneficiaries
• Stakeholders engagement – had developed two stakeholder engagement indices  to measure changes in engagement among stakeholders or in a network. 

The consultant(s) will be asked to update a qualitative evaluation database, store relevant impacts/outcomes, lessons learned, recommendations. The database will be available to conduct qualitative database analysis, consultants are welcomed to use it. 

APPLY 

The proposed methodology should include the following phases:

Preparatory phase: review of all project documentation, updated analysis of the project implementation context, preparation of a work plan jointly with the project team; outlining the evaluation questions kickoff call with staff, donor (as appropriate), propose the evaluation methodology and the development of data collection tools and a sampling methodology. As a result of this phase the consultant will have to submit a plan/inception report, outlining the approach. Upon sign of the report the consultant(s) may initiate the work. 

Fieldwork: Recruit and train  local data collectors (if needed). Field test questionnaires with the different target groups through a pilot or pre-survey; conduct data collection, consolidate, validate, triangulate project existing data. Attend projects online/in person meetings/workshops to gather data as appropriate. Note any travel requirement should be outlined in the plan and in the agreed overall budget and will be allowed only after appropriate security checks. 

Analysis of qualitative and quantitative data: triangulated and, if contradictions appear, explore and present hypotheses as to why such contradictions might exist; preliminary findings will be presented to the project team and the evaluation committee, and the consultant will need to take into consideration the project team’s feedback. 

Deliverables

Some key deliverables must be submitted in draft form to IWPR and the evaluation committee before being finalised as indicated in the table below. All deliverables and data from the evaluation will be owned by IWPR and may be used for internal and external reporting and communications.

• Deliverable 1 Plan Proposed plan and methodologies for the evaluation and case studies development, to be approved by the project manager. 
• Deliverable 2: Conduct online meetings, workshops with IWPR directors, program manager, field staff and other key stakeholders (beneficiaries, donor) to capture, document, triangulate results. 
• Deliverable 3: Draft evaluation: submit to the evaluation committee a draft evaluation and related annexes that outline the main findings with related evidence, including recommendations and lessons learned.Visualisations and case studies are encouraged.
• Deliverable 4 Final evaluation report and summary: in Word format, to be produced in English, should be a maximum of 30 pages (excluding annexes/attachments) and should include the following elements. Summary of the evaluation findings: 2 pages max with overview of overall achievements/performance and key actionable recommendations.
• Deliverable 5 Evaluation dataset A database (excel) with list of outcomes, significance, contribution and related categorization used for the analysis.
• Deliverable 6 Case studies Up to two case studies (2 pages max each) to support “scaling up of success”. Throughout the evaluation the consultant(s) will identify up to 2 successes (either good practices, either techniques, results etc) and, upon IWPR team confirmation, the consultant will document them, providing sufficient evidence and guidance to support uptake in new programming.  Each case study should be 2 pages max. 

Reporting

A proposed full evaluation structure is outlined in the Annex 1. The final full evaluation report should provide with following information and analysis:

1. Overall assessment of the level of success of the project as per DAC criteria: coherence, efficiency, impact, effectiveness, relevance, sustainability. 
2. Beneficiaries reach: overview of beneficiaries reached directly and indirectly: number, type, gender, country and any other relevant disaggregation and involvement in project intervention. 
3. List of significant outcomes achieved, intended as observable changes in actions, relationships, policy or practice of both beneficiaries and external actors linked to the project activities, categorise them using existing and unplanned performance indicators, by DAC criteria and other relevant categories (e.g. gender, type of beneficiary, date etc).
4. Outline significance and relevance of each outcome in light of implementation context at national, regional and international level. 
5. Outline contribution of the project for each outcome.
6. Provide updated key performance indicators (KPIs) using the following examples. Thanks to ZYX, XXX media outlets reached demonstrated greater sustainability or resilience; Thanks to XYZ, XXX citizens were exposed to media content about XX (topic e.g. investigations), and XX audience engaged with XY (theme of content); thanks to ZYX, XXX media outlets received a grant between XX and XX EUR in 2022, and XXX media outlets received training and advice on XXX, and XX attended networking meetings.
7. Outline key actionable recommendations for future programming in the light of current needs (national/regional context), project goal and project successes, good practices and lessons learned. 
8. Outline significant challenges and/or negative changes that have limited or affected achievement of success, analyse effectiveness of mitigation strategies applied (if any) and level of impact of those challenges. Outline lessons learned for future implementation.
9. Update progress as per project performance matrix/M&E framework (baseline, targets and year by year progress)
10. Include visualisations (graphs/tables) and quotes as appropriate to support deeper understanding of evidence and analysis.

Timeframe and Budget

Assignment should be completed between 1 November 2023 and February 29, 2024. 

Signing of contract with consultant(s)

1 – 10 November 2023: Consultant(s)

Activity: Literature review of the project;

• Preparation of the Inception Report, including:
• A detailed methodology note,
• A data quality assurance plan
• Work plans, data collection and data analysis
• Evaluation protocol

Activity: Development of data collection tools (questionnaires, interview guide, Outcome Harvesting form or other proposed methodologies)

Deliverables:

• Draft inception report;
• Draft data collection tools.

10-15 November 2023-Evaluation committee

Activity: Revision and resubmission and validation of final inception report

Deliverables:

• Final inception report; 
• Finalised data collection tools.

15 November 2023-Consultant(s)

Activity: Field data collection (qualitative and quantitative)

Deliverables:

• Detailed workplan and operational plan for data collection

15 November – 15 December 2023- Consultant(s), Evaluation committee

Activity: Preliminary analysis of data and presentation of preliminary findings

Deliverables:

• PowerPoint presentation of preliminary results

15 January 2024- Consultant(s)

Activity: Submission of first draft of the evaluation report in English

Deliverables:

• Draft final evaluation report

22 January 2024- IWPR, Evaluation committee

Activity: Review of first draft of the evaluation report

1 February, 2024- Consultant(s)

Activity: Integration of inputs in 1st draft and submission of 2nd draft

15 February 2024- Consultant(s)

Activity: Submission of final report in English with all annexes

Deliverable:

• Final evaluation report in English

Activity: Approval of evaluation report by donor

Ethics and Data Protection

The assessment will be conducted in accordance with IWPR  policies on non-discrimination, conflict sensitivity, data protection, safeguarding and ethics. All relevant policies will be communicated to the consultant(s) with their contract. It is expected that the consultant(s) will demonstrate in their proposed methodology how they will address non-discrimination, data protection, safeguarding and ethics, and conflict and gender sensitivity.

Quality Assurance 

Evaluation Committee: An evaluation committee will be established consisting of IWPR staff (IWPR MEL manager, program Director and Manager). Implementing partners and donors will be involved by the committee as appropriate at the kickoff and completion of the effort to ensure validity and utilisation. The committee will accompany the process and ensure the quality of the analysis and methodology proposed in the inception report, and its implementation after submission of the preliminary results and a draft of the final report. 

Data quality: Quality data are data that are not systematically biased and do not distort representativeness or coverage. A data quality assurance plan proposed by the consultant will take into account anything that might go wrong with the data collection in advance and develop a strategy to prevent these problems. The plan should be shared in the inception report. This data quality assurance plan will focus on three elements: data collection tool/survey design, field management of data, and high frequency monitoring.

Knowledge, Skills and Abilities

We are looking for a consultant/team of consultants/international or national consultancy firm meeting the following requirements:

• A minimum of a Master’s degree in a relevant field (political science, sociology, international relations, conflict analysis and management and similar disciplines);
• At least 8 years of professional experience in the above-mentioned fields with proven experience in conducting quantitative and qualitative research, in particular in conducting evaluations of complex, multi-stakeholder projects in the areas of peacebuilding, media, conflict and/or organisational governance and theory-based evaluations;
• Experience with quantitative data analysis software or the ability to subcontract work (if quantitative analysis method is proposed);
• Excellent writing skills in and English. Knowledge of Ukrainian will be considered an advantage;
• Excellent report writing skills;
• Knowledge of Ukrainian socio-cultural, economic and political context and related international geopolitics;
• Knowledge and experience of gender and conflict issues and application of suitable evaluation methodologies.

Position: Final Project Evaluation-Boosting Investigative Journalism in Ukraine project

Contract type: Consultant 

Location: Remote 

Start date: ASAP

Duration: Assignment should be completed the end of January 2024

APPLY