Short-term consultant to conduct endline assessment of UCCCT project/ Короткостроковий консультант для проведення підсумкової оцінки проекту
Global Communities is seeking a qualified short-term consultant for the Ukrainian Communities Countering Corruption Together Project (UCCCT):
POSITION: Short-term consultant to conduct endline assessment of the project
PERIOD OF PERFORMANCE: 4 months
UCCCT is a two-year project implemented by Global Communities, in partnership with the Odesa branch of the Committee of Voters of Ukraine (Odesa CVU), CASE Ukraine, and selected communication consultants. UCCCT aims to engage citizens in local governance, building civic and government understanding and capacity for anti-corruption efforts. The program goal is to reduce corruption at the community level, with a focus on enabling improved services in healthcare and education as the two key sectors of citizen concern in the incidence and prevalence of corruption, pursuing two objectives:
- Objective 1: Build citizen awareness of corruption risks and effective mechanisms supporting local government transparency and integrity; and
- Objective 2: Communities establish social accountability and oversight mechanisms to counter corruption in local government.
UCCCT project development hypothesis is the following:
If citizen awareness and understanding of anti-corruption measures is built, and more civic activism and scrutiny of government processes is introduced in CCs, and public pressure is brought to bear on local government to change, then local government will be more transparent for citizens.
For the UCCCT development hypothesis the underlying assumptions, as well as external factors that may influence program performance, are as follows:
Assumptions: CCs’ local government are committed to work with UCCCT and introduce positive changes to higher accountability and transparency. Local governments are open to give access to public funds data. CCs citizens are willing to raise their civic activism and participate in public resources oversight.
External Factors: GOU passes the final legislation changes that perform education and medical reform. War with Russia remains contained in the eastern Donbas Oblasts. Political and economic conditions in Ukraine remain stable after the 2019 presidential and parliamentary elections. Duration of quarantine measures to combat СOVID-19.
The chain of results from activities to outputs and outcomes that illustrate what UCCCT aims to achieve under given assumptions and external factors defined, is shown in Appendix I: Logic Model of UCCCT.
UCCCT’s approach to local governance strengthening emphasizes cooperation between citizens and government to strengthen mutual understanding and increase transparency, ultimately creating a platform for constructive cooperation across the citizen-government divide on issues of anti-corruption. The program combines training, coaching and awareness-raising for citizens and officials with opportunities for learning-by-doing, such as through advocacy campaigns conducted in coordination with local authorities.
The primary outcome-level indicators of interest the end line will use to evaluate the overall performance of the program at the end of the performance period are the following:
1.1: Proportion of government officials provided with USG-supported anti-corruption training that are aware of the available tools/resources to reduce corruption and how to use them.
1.2: Proportion of people affiliated with non-governmental organizations provided with USG-supported anti-corruption training that are aware of the available tools/resources to reduce corruption and how to use them.
2.5: Number of anti-corruption successes achieved by external oversight, investigations or other anti-corruption interventions as a result of USG assistance.
In addition, the end line study will seek to gain a deeper understanding of the motivations for engaging in anticorruption among public service users and public officials who participated in the project.
Objectives and Scope of the Assessment
The objectives for the project end line assessment are:
- To understand to what extent UCCCT achieved its outcomes;
- To learn about some of the effective ways in which UCCCT achieved those results.
- Which anticorruption tools introduced by the project were most effective and why?
- What is the proportion of trained officials who can provide at least one example of an anti-corruption tool s/he used, how s/he used it and for what result?
- What is the proportion of NGOs affiliated and trained by the project whose representatives can provide at least one example of an anti-corruption tool they used, how they used it, and for what result?
1. What are the main motivations of program participants to become engaged against corruption and in support of local government transparency?
2. For what reasons did public officials of communities or facilities targeted by the project become involved in working to prevent corruption?
3. For what reasons did service users of facilities targeted by the program become involved in working to prevent corruption?
4. What were the most significant intended and unintended results achieved by this project?
5. Which results did government officials see as significant in the field of anti-corruption?
6. Which results most influenced citizen trust and confidence in government?
7. Which results most influenced the confidence of service users of the facilities target by the project in government?
The proposed methods of data collection and analysis should include:
- Desk review and analysis of the project reports, monitoring data and annual work plans;
- Key informant interviews by telephone (skype, zoom, etc.) or face to face, with beneficiaries and stakeholders among public officials, service users and non-governmental organizations.
We will use a case-based approach to illustrate changes and successful project results, including rich in detailed descriptions of what were UCCCT’s specific ways for achieving the successful results and important context details. This ‘how we learned to do it’ focus in presenting project results will be helpful for future scaling up of the project experience in Ukraine. The assignment will entail qualitative data analysis to uncover the themes relevant to the research questions. As there are a variety of ways to approach qualitative data analysis, the technqiue used under this assignment will be determined by the consultant in discussion with GC. Any cost related to subscription or use of the software would be considered to be covered by (not in addition to) the consultancy fees.
Work Plan, Deliverables and Timeline
The following steps are foreseen with required deliverables and proposed timelines
- Desk Review
Estimated LOE: 3 working days
Desk review of background documents (familiarization with the project, its objectives, beneficiaries, partners and progress reports, project documents, project monitoring data). Information and necessary documents will be provided by the technical experts of the Kyiv office of Global Communities.
- Inception Report
Estimated LOE: 5 working days
Develop an Inception Report detailing the strategies to collect and analyze data against project targets and key indicators from the proposed evaluation methodology. The inception report will need to be approved by the UCCCT staff, and must include data collection tools (questionnaires, outline for focus group discussion, interview questions etc.) Questionnaires should be designed for three categories of interviewees: for community activists, CC management, and program partners. The inception report should propose an appropriate sample size using a summary of the data provided by GC. The inception report should detail how data (particularly personal identifying information) will be protected and secured all stages -from collection to reporting. The report should also include reference to any relevant Ukraine laws or norms regulating research involving human subjects and how the measures being applied in this assignment to comply accordingly.
Estimated LOE: 20 working days
Key informant interviews by telephone (by skype, zoom, etc.) or in-person, in-person focus groups with beneficiaries and stakeholders among public officials, service users and non-governmental organizations. Conduct a survey of size and quality able to provide a meaningful understanding of progress toward targets and changes in the indicators being measured. Keep a log of interviews, a database with details for each contact, time of scheduled interview, emails sent. Keep a transcript of each interview that has a matching identifier with the database. . Share and discuss progress and any challenges or concerns with GC Ukraine staff on a regular basis. Preparation of a report on the results of the deliverable.
- Focus group discussions with beneficiaries and stakeholders among public officials, service users and non-governmental organizations to stratify the needs of different segments of the target population, assess how UCCCT interventions and implementation pathways were perceived in the target Focus group discussions will be stratified by geography and sector (health or education). The CC and contact lists will be provided by UCCCT management in collaboration with Odessa CVU. At least one focus group is planned in one of the CCs.
- In-depth interviews and consultations with local activists and CC management to understand the project implementation process and quality, as well as identify best practices, barriers and challenges in promoting anti-corruption tools in project areas. The consultant will collect information to assess knowledge, behaviors and skills of the target groups. At least 5 interviews are planned for each group (community activists, CC leadership). Selected communities are expected to be from different regions and different in size).
- In-depth interviews and consultations with key implementers of the project (coaches, communication consultants, CASE Ukraine) to understand the project implementation process and quality, as well as identify best practices, barriers and challenges in promoting anti-corruption tools in project areas. At least 5 interviews are planned for coaches, 1 for CASE Ukraine, and 3 for communications consultants.
Estimated LOE: 4 working days
Once data are collected and prepared, the consultant will analyze it. Data will be analyzed to obtain answers to the main questions of the research for further formation of the draft report.
- Draft Report
Estimated LOE: 8 working days
Preparation of the first draft of the report, introduction of changes, revision with the Kyiv and Headquarters offices of Global Communities. Based on the analysis conducted in the previous step, produce an end line report that highlights significant, important or interesting findings of the project according to the methodology of the end line assessment. The report should look like this:
Table of Contents
Including boxes, figures, tables, and annexes with page references.
List of acronyms and abbreviations
Executive Summary (up to two pages)
A stand-alone section that should:
- Briefly describe the intervention (the project(s) that was evaluated).
- Explain the purpose and objectives of the evaluation, including the audience for the evaluation and the intended uses.
- Describe key aspect of the evaluation approach and methods.
- Summary of the key findings, conclusions, and recommendations.
Introduction (up to five pages)
This section will:
- Provide a brief explanation of why the evaluation was conducted, why the intervention is being evaluated at this point in time, and why it addressed the questions it did.
- Identify the primary audience or users of the evaluation, what they wanted to learn from the evaluation and why and how they are expected to use the evaluation results.Identify the intervention of the project.
- Acquaint the reader with the structure and contents of the report and how the information contained in the report will meet the purposes of evaluation and satisfy the information needs of the report’s intended users, such as Global Communities, donors representatives, implementing partners, etc.
Description of the Intervention (up to eight pages)
This section will provide the basis for report users to understand the logic and assess the merits of the end line assessment methodology and understand the applicability of the evaluation results. In particular, the section will:
- Describe what is being evaluated, who seeks to benefit, and the problem or issue it seeks to address.
- Explain the expected results map or results framework, implementation strategies, and the key assumptions underlying the strategy.
- Identify any significant changes (plans, strategies, logical frameworks) that have occurred over time and explain the implications of those changes for the evaluation.
- Identify and describe the key partners involved in the implementation and their roles.
- Describe the context of the social, political, economic, and institutional factors, and the geographical landscape within which the intervention operates and explain the effects (challenges and opportunities) those factors present for its implementation and outcomes.
- Point out implementation constraints (e.g., resource limitations).
Evaluation Scope and Objectives (up to 10 pages)
This section of the report will provide an explanation of the evaluation’s scope, primary objectives and main questions.
- Evaluation scope and objectives – define the parameters of the evaluation, for example, the time period, the geographic area included, and which outputs or outcomes were and were not assessed.
- Evaluation questions – the evaluation questions will define the information that the evaluation will generate. The report will detail the main evaluation questions addressed by the evaluation and explain how the answers to these questions address the information needs of users.
The description of methodology will include discussion of each of the following:
- Data sources – sources of information (documents reviewed), the rationale for their selection and how the information obtained addressed the evaluation questions.Data collection procedures and instruments – methods or procedures used to collect data, including discussion of data collection instruments (e.g., interview protocols), their appropriateness for the data source and evidence of their reliability and validity.
- Stakeholder engagement – how the level of involvement contributed to the credibility of the evaluation and the results.
- Data analysis – procedures used to analyze the data collected to answer the evaluation questions. This will detail the various steps and stages of analysis that will be carried out, including the steps to confirm the accuracy of data and the results. The report will discuss the appropriateness of the analysis to the evaluation questions. Potential weaknesses in the data analysis and gaps or limitations of the data should be discussed, including their possible influence on the way findings may be interpreted and conclusions drawn.
Lessons Learned and Success Stories (up to five pages)
The report will include discussion of lessons learned from the evaluation, including new knowledge gained from the particular circumstances (intervention, context outcomes). Particular attention will be paid to the success stories of the use of anti-corruption tools in communities, as well as the opportunities for their dissemination outside the project. It is planned that at least one success story will be identified in each community.
Findings and Conclusions (up to three pages)
This section will present the evaluation findings based on the analysis and conclusions drawn from the findings. In particular,
Findings: This section will present findings as statements of fact that are based on analysis of the data. The evaluation findings will be structured around the evaluation criteria and questions so that report users can readily make the connection between what was asked and what was found. Variances between planned and actual results will be explained, as well as factors affecting the achievement of intended results. The assumptions or risks in the project design that subsequently affected implementation will also be discussed.
Conclusions: This section will be comprehensive and balanced and highlight the strengths, weaknesses and outcomes of the intervention. The conclusion section will be substantiated by the evidence and logically connected to the evaluation findings. The conclusion will also respond to key evaluation questions and provide insights into the identification of and/or solutions to important problems or issues pertinent to decision-making.
Recommendations: The evaluation will seek to provide very practical, feasible recommendations. The recommendations will be specifically supported by the evidence and linked to the findings and conclusions around key questions addressed by the evaluation. This recommendation can be used to have better achieved the project’s goals in the future or in future programming, or in similar projects.
The Annex section will include documents to provide the report reader with supplemental background and methodological details that enhance the credibility of the report.
- Internal Validation Workshop
Estimated LOE: 2 working days
Facilitate a half-day online validation workshop (via Zoom, Skype, Microsoft Teams, or an equivalent software), in which UCCCT program and technical staff will provide the consultant with additional input that will serve to validate initial findings and recommendations before they are made final. The workshop will also be open to Odesa CVU coaches for participation.
- Final Report
Estimated LOE: 8 working days
The first draft will be reviewed by the management of the Kyiv and Headquarters offices of Global Communities. The consultant will finalize all recommendations and make the necessary changes to the document and the final report will be refined to reflect the recommendations made.
The consultant is expected to complete all the work by August 30, 2020.
The Selection Process
Global Communities will select candidates based on meeting the following criteria:
- At least a Master’s degree (sociology, public policy, economics, and other related areas);
- At least 2 years’ experience as an evaluator, designing and conducting assessments and baseline surveys including qualitative data collection;
- Excellent knowledge and understanding of research methodologies and processes;
- Experience in gathering and systematizing qualitative data;
- Experience and knowledge of local government management and decentralization reform;
- Previous experience (or ability to demonstrate the capacity) of working with the CCs, facilitating public meetings in the community;
- Knowledge of English at the work level is an asset;
- Ability to work in a team according to established terms, good skills of time distribution;
- Creativity, initiative, focus on results, and responsibility.
To apply please send your CV, examples of work (optional) and cover letter with summary of your experience as described above to [email protected] indicating “UCCCT project Consultant” in the subject line by COB June 5, 2020.
Only applicants selected for interviews will be contacted. No telephone inquiries will be accepted.
Please note that in the event of a prolongation of quarantine in Ukraine, in connection with COVID-19, all interviews will be conducted via Skype.