Evaluating across cultures

As monitoring and evaluation practices become more commonplace within organisations and project plans, it becomes even more important to capture the lessons we learn from the different monitoring and evaluation (M&E) methods. At the same time, ensuring that all relevant stakeholders participate is a widely shared goal of organisations working in the social justice and development fields. Participatory M&E methods can bring these two aims together.

Participatory M&E methods can be defined as any method which ensures broad, meaningful stakeholder (both internal and external) contribution to an organisation’s M&E practice. Methods can range from including a diverse sample of stakeholders in an ex-post evaluation, to involving various stakeholders in the design process of the M&E framework.

Beyond the scope of participation, it is crucial that participation is actually meaningful. Our experience evaluating NGOs across cultures is that contextualising participatory evaluation tools to both the culture of organisation itself, which includes structure, people and ways of working developed over time, and that of the society in which that organisation operates are key to making participation meaningful in any given context. Democracy in organisations is associated with higher levels of responsibility and self-expression. It is therefore interesting to look at how participatory M&E works in these organisations. In this essay, we focus specifically on evaluations of democratic organisations, and share two main lessons on contextualising participatory evaluation tools.

Firstly, it is important to identify the societal norms which may limit the democratic space provided by an organisation and adjust participatory tools accordingly.

While some organisations have a culture that promotes self-expression, societal cultural values may present elements that influence the way in which staff members/stakeholders express themselves. For example, from conducting participatory evaluations in different cultures, we have experienced several cultures that put emphasis on the respect for older persons. In a workplace setting, this leads to some senior managers finding it a challenge to provide negative feedback to older people in more junior positions, even when they are given the space and authority to do so. Where age is not a factor, some societies still show a general reticence to confront figures of authority in spite of the organisation’s efforts to encourage self-expression.

In such cases, it is important for the evaluator to understand that this dynamic is likely to manifest in the participatory evaluation process as well. As such, participatory evaluation tools need to be tailored to overcome this dynamic. Tools that include diverse focus groups may be less helpful in these situations and one might opt for tools that involve more one-to-one private communication. Questionnaires and one-to-one interviews, in which confidentiality and anonymity are emphasised, for example, can facilitate discretion. If one opts to use focus groups, the selection of participants can be based on purposive sampling that aims to overcome this dynamic.

In addition to these steps, it is important that the evaluator actively listens to and observes the group or individual dynamics. For example, the evaluator may notice that some respondents may not be particularly assertive on issues about which they feel uncomfortable, such as providing negative feedback to respected people. It is important that the evaluator recognises this, and tailors the way in which constructive criticism is probed. This can be done by paying attention to the use of language, for example, asking for suggestions for improvement as opposed to asking respondents to point out weaknesses.

Secondly, participatory tools in democratic organisations operating in non-democratic societies need to be encouraging, non-threatening and empowering.

Organisations that maintain a democratic culture while operating in non-democratic societies present an opportunity for their staff to express themselves. When conducting evaluations in Least Developed Countries where the space given to civil society is minimal, we have found that staff openly shows appreciation for contextualised participatory evaluation. The process allows them to feel that they have been given the opportunity to share their thoughts and contribute to learning within the organisation. Staff generally find this experience rewarding and motivating. They also perceive it as an opportunity to explore themselves and their ability to evaluate a project.

However, when conducting participatory evaluation in such contexts, it cannot be taken for granted that staff will seize the opportunity to participate without hesitation or suspicion. In addition to being participative, the tools used need to be particularly encouraging of participation, non-threatening, and empowering. Questionnaires for example may need to be personalised to take into account the particular experience of the participant so as not be overwhelming. This also applies to one-to-one interviews. Focus group participants would need to be carefully selected, in full consideration of what would make participants most comfortable to openly engage. The evaluator may consider separate focus groups for donors, managers and junior staff, and if necessary further split these groups based on other relevant criteria. In all cases, the evaluator would need to clearly articulate the purpose of the data collection, what happens to the information and any specifics on confidentiality, anonymity or risks. The evaluator may also invite participants to ask questions or express their concerns before the discussions. Dealing with these at the beginning goes a long way towards making participants feel comfortable.

Evaluation methodologies have benefits that stretch beyond the short-term purpose of evaluating impact. For democratic organisations in which societal norms limit self-expression as in the examples provided above, well selected participatory evaluation tools help to gather meaningful information to support learning and bolster planning. Beyond this, the evaluation process can leave participants with examples of alternative ways of self-expression that may be more culturally acceptable.

For democratic organisations operating in non-democratic societies, well selected evaluation tools may showcase to stakeholders, the organisation’s commitment to openness and communicate to them that their participation is valued. This can increase motivation, thereby contributing to impact in the medium- to long-term.  Participatory evaluation therefore needs to look beyond achieving participation in terms of scope and invest effort in making participation meaningful so that its full potential can be realised.

In any given context, there will be organisational and socio- cultural elements to consider. For the evaluator, this means investing more time in understanding the context in which one is conducting the evaluation before going into the field. Depending on the type of organisation/project/programme being evaluated, it also means tailoring evaluation tools to the experiences/roles of the stakeholders. While the process of contextualising participatory evaluation tools takes time and effort, our experience has been that it increases the quality of the evaluation and equips organisations with tools or experiences that contribute to longer term capacity, confidence and motivation particularly amongst those who have been engaged.