Hello,
Exciting news- following a series of learning events in London on DFID’s Beneficiary Feedback Mechanisms pilot programmes, a Linked In group has been set up to allow us all to share resources, ideas, experiences….
Come and join https://www.linkedin.com/groups/7053295/profile
You can find case studies for the pilots here
and a really excellent film that is great for advocacy/ training purposes here
Irene Guijt and myself are doing a blog series on Better Evaluation for June and July. You can find our first blog here
http://betterevaluation.org/blog/four_reflections_on_participation_in_evaluation
Keystone recently guest edited the Alliance magazine special feature on feedback – Beyond Accountability: Feedback as Transformation – read the lead article here.
Do please share any other good and relevant reads with me and I can add them. Continuing on from the bibliography that you can find here.
Thank you to everyone for comments, inputs and support. The report is now out, although still to be launched. You can find it attached here.
Please do feel free to print out the checklist and analytical frameworks that I posted below. Do use them, revise them and share your improved versions….DfID Beneficiary Feedback, Feb 2015 – Final
Also attached is the ppt presentation that I shared at the UK Evaluation Society Conference last week.Beneficiary Feedback in Evaluation_UKES Methods Workshop
Attached are a set of checklists that I developed for
1. Evaluation Commissioners
2. Evaluation Practitioners
to help integrate a beneficiary feedback approach to evaluation.
Please feel free to pilot, use and feedback on what was and wasn’t useful. This is a live tool that I will happily keep reviewing based on feedback. Let me know how it goes….
So it has been a busy month finalising the report on Beneficiary Feedback for DFID and we already have people up for piloting/ testing the approach: Two forthcoming evaluations in the Africa Region, one joint monitoring mission between DFID and the World Bank and hopefully another evaluation of a DFID funded NGO programme. The draft report has been presented to DFID, to the UK Learning Group on Beneficiary Feedback and will be presented to Interaction next week. Positive responses so far. World Bank are keen to review also, particularly in view of their commitment to 100% beneficiary feedback. Hopefully this paper and this new paper from Interaction will help with the “How To” do this practically, meaningfully and ethically in an evaluation context.
But I would love to get some discussion going here.
Are you trying to engage a beneficiary feedback approach to evaluation? What are your successes and challenges?
Have you read Interaction’s new paper and what do you think?
As for my paper, it is currently being formatted and will be published as a DFID working paper in the next few weeks. I will share then.
I have designed what I hope is a really simple tool to help map out what types of beneficiary feedback might be most appropriate at different evaluation stages. Would anyone be interested in testing out this tool in any evaluations that they are commissioning/ tendering for/ designing?
As you can see below, it is a straightforward matrix that requires you to consider and map out the type of feedback that you think would be most appropriate at each stage of your evaluation. You can then use this to consider which methods might be most useful as well as what resources you might require. Sometimes there may be no extra cost, sometimes there may be.
You can copy the matrix from the below or I have attached it so that you can (hopefully) print it out and use it. Beneficiary Feedback in Evaluation Matrix
Please note that different types of feedback may be more appropriate at different stages.
| One-way feedback to beneficiaries | One-way feedback from beneficiaries | Two-way feedback – inter active conversation between beneficiaries and evaluators | Two-way feedback through participatory evaluation | |
| Evaluation Design | ||||
| Data Collection | ||||
| Validation and Analysis | ||||
| Dissemination and Communication |
Do let us know how it goes! What were your challenges? Which approaches/ methods worked best? This website has received over 1100 views so far from all over the world so I hope we can share our learning together.
Literature review, interviews done, I have established in my mind that feedback can mean everything and pretty much nothing. And it definitely does not need to fall under “participation” (topic for the next blog post). Specificity is everything in this case. I have therefore developed the following simple typology to help me to distinguish and analyse different types of feedback:
Neither of these is “good” or “bad”. Each has value in and of itself- choices will be defined by your context.
To illustrate, I have tried to map examples of each of these types of feedback to the evaluation stages that I discussed in my previous post. This is very much early thinking so feel free to add your thoughts…
Matrix 1: Illustrative examples of beneficiary feedback at different stages of the evaluation process
| One-way feedback to beneficiaries | One-way feedback from beneficiaries | Two-way feedback – inter active conversation between beneficiaries and evaluators | Two-way feedback through participatory evaluation | |
| Evaluation Design | Dissemination of evaluation protocol to enhance informed consent and meaningful engagement with content and process of evaluation | Views of beneficiaries sought on evaluation questions/ protocol | Joint discussion on evaluation protocol/ questions. This could be at different stages of design. Ultimate design decisions rest with evaluation team | Beneficiaries input into evaluation design as evaluation team members |
| Data Collection | Beneficiaries informed of data collection processes, including with other stakeholders | Views of beneficiaries sought alongside other data collection methods. This could be through survey, interview, focus group discussion where evaluators extract data with no return of/ joint discussion on data | Beneficiaries question each other and evaluators as data is collected through e.g. interactive focus group discussions or discussion of survey findings in order to challenge own and evaluators’ assumptions and interpretations | Beneficiaries engaged in data collection as members of team |
| Validation and Analysis | Sharing of early findings and/ or analysis with beneficiaries | Beneficiaries share their views of early findings with evaluation team- questioning or validating these | Beneficiaries and evaluation team discuss and refine findings together, jointly discussing recommendations. Ultimate decisions rest with evaluation team. | Beneficiaries engaged as part of evaluation team in validation and analysis of evaluation findings |
| Dissemination and Communication | Dissemination of relevant evaluation findings in appropriate format (poster/ document/ radio/ video/ face to face) | Beneficiaries share their views on materials shared: this can be on content or methodology | Beneficiaries and evaluation team or programme team (depending on dissemination and communication plan) discuss findings together, learning jointly from process | Beneficiaries, as part of evaluation team, design and support dissemination and communication activities |
Feedback for me is about a conversation. This is what distinguishes it from simple data collection. In evaluation, this conversation may occur during the data collection phase. However, what about the other stages of the evaluation process? What about a conversation with beneficiaries during design? This could be as part of participatory design or it could be about respecting basic evaluation ethics to ensure that consent to participate in the evaluation is informed. Rather than evaluators turning up only to find that they participants of a focus group discussion, for example, are quite unclear as to why we/they are there. What about a conversation to ensure that our provisional evaluation findings, including judgements, are on track? That they haven’t been subject to our world view to the point that we may have missed crucial cues or been unable to break through entrenched power relations? What about ensuring that findings, including lessons learned, are shared to ensure that beneficiary groups involved in a global programme can learn from and adapt successful practice in other parts of the world? This isn’t just about manners, ethics, respect. It is also about ensuring we have robust evaluation findings.
Could we expect evaluators to consider a four part process for engaging beneficiary feedback? The inability to conduct any of the steps would need to be argued in the inception phase- it may be a lack of resources, including time, for example. But the expectation would be that due consideration be given to these steps, in the same way as it would be to developing evaluation questions, for example.
Here are the four steps:
1. Feedback as part of evaluation design: e.g. Sharing of/ consultation on/ participatory design of evaluation
2. Feedback as part of data collection: Could be extractive/ interactive/ participatory collection of information
3. Feedback as part of joint validation and or/ analysis of information: Could be extractive or participatory
4. Feedback on end product/ response and/or follow up: Could be simple dissemination or participatory engagement for future actions.
Would this be reasonable?
NB: Appropriate methods would need to be selected in response to the evaluation questions. They may be participatory or they may be extractive. Each evaluation design will need to select what is most appropriate to answer the given question.
Thank you to all those who have emailed me, discussed on Pelican or MandE groups. It is a privilege to be part of this community.
What strikes me so far:
1. We still haven’t got it with the language. “Feedback” has meant something different to pretty much everyone I have spoken with and we are still where we were decades ago in terms of trying to re-define the word beneficiary. The word had theoretically disappeared forever with the advent of rights based approaches to development. Gone. Or so we thought. Yet, somehow, it has slipped back into the mainstream of development discourse. How did this happen? What does this mean for rights based approaches? Are they slipping away too?
My worry? Language is important and we must have these discussions and I really hope someone finds the perfect word. I am trying but failing myself.
BUT as various organisations I have discussed with have told me “We keep having the discussion and then getting stuck on language. This paralyses us”. Is this why I am finding so few examples of closed loop feedback? Are we getting so lost in the words that the actions are not following? Or do the words highlight just how difficult it is to truly implement a rights based approach (which must include closed loop feedback) in the current context? Or maybe it is like gender equality, we must never assume we are there. We must constantly shine the torch on the fact that it makes sense to ensure the on-going (closed loop) participation of those in whose name we work in shaping what we do.
2. We are in a period of growth in terms of the tool box available for extracting data from people to feed into programme decision-making. With technology and speedier travel, we can do this more quickly and solicit the views of more people than ever before- we can do it in extractive or in participatory ways. VOTO has a system of “Interactive voice response (IVR) (which) allows a real person to record a powerful and moving message using all verbal influence tools (e.g., tone, energy level, personality of voice) in a user’s local language.” The Listening Project listened to 6,000 aid recipients about their experiences.Breadth – tick. What about depth? Well, Reality Checks methodology have taken anthropological approaches into evaluation. The University of Bath has been testing its Qualitative Impact Assessment Protocol, which in a similar way to Goal Free Evaluation, in that informants don’t know what they are giving feedback on. The idea is to ‘blind’ everyone involved in the process to the projects being assessed to try and illicit less biased and more wide-ranging information. Reflective Learning have worked with young people to develop one minute questionnaires using touch screen apps: the evaluation questions are designed by young people to reflect what they feel they need to know.
BUT
3. We haven’t cracked it in any way when it comes to closing the loop in our evaluations. We extract data: we can do it without even talking to people, they can send us an SMS, or we can do it using participatory methods. However, very few of us are i) validating our evaluation findings with our informants or the wider group that they represent and/ or ii) feeding back our observations/ conclusions/ management responses to those who have so generously given of their time to give us the information that we ask for.
Why?
Have you got an interesting experience of beneficiary feedback in evaluation that you would like to share? If so, I would love to hear from you.
I am developing 4 case studies to pull out the added value/ challenges/ practicalities of beneficiary feedback in evaluation.
While the case studies would help my research, they will support your own organisational learning. I would also be very happy to provide feedback to your organisation on the findings if that would be of interest.
Don’t hesitate to contact me for more information (lesliegroves@yahoo.com)
I look forward to hearing from you!
Beneficiary feedback in evaluation 