4 Steps to Beneficiary Feedback in Evaluation

Feedback for me is about a conversation. This is what distinguishes it from simple data collection. In evaluation, this conversation may occur during the data collection phase. However, what about the other stages of the evaluation process? What about a conversation with beneficiaries during design? This could be as part of participatory design or it could be about respecting basic evaluation ethics to ensure that consent to participate in the evaluation is informed. Rather than evaluators turning up only to find that they participants of a focus group discussion, for example, are quite unclear as to why we/they are there. What about a conversation to ensure that our provisional evaluation findings, including judgements, are on track? That they haven’t been subject to our world view to the point that we may have missed crucial cues or been unable to break through entrenched power relations? What about ensuring that findings, including lessons learned, are shared to ensure that beneficiary groups involved in a global programme can learn from and adapt successful practice in other parts of the world? This isn’t just about manners, ethics, respect. It is also about ensuring we have robust evaluation findings.

Could we expect evaluators to consider a four part process for engaging beneficiary feedback? The inability to conduct any of the steps would need to be argued in the inception phase- it may be a lack of resources, including time, for example. But the expectation would be that due consideration be given to these steps, in the same way as it would be to developing evaluation questions, for example.

Here are the four steps:

1. Feedback as part of evaluation design: e.g. Sharing of/ consultation on/ participatory design of evaluation

2. Feedback as part of data collection: Could be extractive/ interactive/ participatory collection of information
3. Feedback as part of joint validation and or/ analysis of information: Could be extractive or participatory
4. Feedback on end product/ response and/or follow up: Could be simple dissemination or participatory engagement for future actions.

Would this be reasonable?

NB: Appropriate methods would need to be selected in response to the evaluation questions. They may be participatory or they may be extractive. Each evaluation design will need to select what is most appropriate to answer the given question.

Advertisements

Moving from one way “upwards” feedback to closing the loop

Thank you to all those who have emailed me, discussed on Pelican or MandE groups. It is a privilege to be part of this community.

What strikes me so far:

1. We still haven’t got it with the language. “Feedback” has meant something different to pretty much everyone I have spoken with and we are still where we were decades ago in terms of trying to re-define the word beneficiary. The word had theoretically disappeared forever with the advent of rights based approaches to development. Gone. Or so we thought. Yet, somehow, it has slipped back into the mainstream of development discourse. How did this happen? What does this mean for rights based approaches? Are they slipping away too?

My worry? Language is important and we must have these discussions and I really hope someone finds the perfect word. I am trying but failing myself.

BUT as various organisations I have discussed with have told me “We keep having the discussion and then getting stuck on language. This paralyses us”. Is this why I am finding so few examples of closed loop feedback? Are we getting so lost in the words that the actions are not following? Or do the words highlight just how difficult it is to truly implement a rights based approach (which must include closed loop feedback) in the current context? Or maybe it is like gender equality, we must never assume we are there. We must constantly shine the torch on the fact that it makes sense to ensure the on-going (closed loop) participation of those in whose name we work in shaping what we do.

2. We are in a period of growth in terms of the tool box available for extracting data from people to feed into programme decision-making. With technology and speedier travel, we can do this more quickly and solicit the views of more people than ever before- we can do it in extractive or in participatory ways. VOTO has a system of “Interactive voice response (IVR) (which) allows a real person to record a powerful and moving message using all verbal influence tools (e.g., tone, energy level, personality of voice) in a user’s local language.”  The Listening Project listened to 6,000 aid recipients about their experiences.Breadth – tick. What about depth? Well, Reality Checks methodology have taken anthropological approaches into evaluation. The University of Bath has been testing its Qualitative Impact Assessment Protocol, which in a similar way to Goal Free Evaluation, in that informants don’t know what they are giving feedback on. The idea is to ‘blind’ everyone involved in the process to the projects being assessed to try and illicit less biased and more wide-ranging information. Reflective Learning have worked with young people to develop one minute questionnaires using touch screen apps: the evaluation questions are designed by young people to reflect what they feel they need to know.

BUT

3. We haven’t cracked it in any way when it comes to closing the loop in our evaluations. We extract data: we can do it without even talking to people, they can send us an SMS, or we can do it using participatory methods. However, very few of us are i) validating our evaluation findings with our informants or the wider group that they represent and/ or ii) feeding back our observations/ conclusions/ management responses to those who have so generously given of their time to give us the information that we ask for.

Why? 

Looking for 4 case studies of beneficiary feedback in evaluation

Have you got an interesting experience of beneficiary feedback in evaluation that you would like to share? If so, I would love to hear from you.

I am developing 4 case studies to pull out the added value/ challenges/ practicalities of beneficiary feedback in evaluation.

While the case studies would help my research, they will support your own organisational learning. I would also be very happy to provide feedback to your organisation on the findings if that would be of interest.

Don’t hesitate to contact me for more information (lesliegroves@yahoo.com)

I look forward to hearing from you!

Welcome to this workspace on beneficiary feedback in evaluation and research

“Beneficiary Feedback” is a term that has entered the anglophone development and humanitarian workers’ vocabulary. perhaps surprisingly for those of us who thought that the term “beneficiary” had been dropped with the advent of rights based approaches. But here we are. The term is out there, being discussed/ blogged about/ rejected/ embraced. For evaluation and research, it seems the term brings up more questions that anyone has yet answered: is this just a re-hashing of participatory approaches to evaluation and research? Is it something different? Could this be the antidote we need to bring people and power back into the forefront of our discussions on evaluation and research? What does it mean in practical terms?

To help think through some of these implications, I have been commissioned to conduct research to:

  • Summarise the literature on Beneficiary Feedback, including definitional issues and lessons from practice;
  • Analyse the current and potential use of Beneficiary Feedback within evaluation and research; and,
  • Provide recommendations for future DFID guidance on incorporating Beneficiary Feedback within evaluation and research programmes.

I have set up this workspace to solicit views, experiences, recommendations from the wider evaluation and research in development community.  Please note that this space is not an official DFID approved website. Any opinions expressed here are my own and shared in the interests of open learning and reflection.

My Terms of Reference are here.Terms of Reference Beneficiary Feedback in Evaluation and Research

To start us off, I have posted a poll below and would love to hear your thoughts….

I would also really like to hear from anyone who is engaging “beneficiary feedback” in evaluation or research. I am looking for case studies to help us with our learning. Please use the contact form below and I will get back to you directly.

Also please feel free to post a comment/ question below and get the discussion rolling….

Thank you.