Developing Outcome Indicators

Cathy Sharp draws lessons from her experience of developing indicators of capacity building outcomes for an evaluation of a community project in Fife.

To paraphrase the philosopher Richard Rorty; “The point of research is to talk to each other about what we ought to be doing.”

For those involved in voluntary and community sector organisations, evaluation can provide just such a chance for this type of conversation. Rather than something ‘done to them’, evaluation should be something that organisations are fully involved in. Even if an organisation believes that they are doing is worthwhile and important work, there is always scope to ask “how are we doing?” and “how could we improve what we are doing?” This suggests that evaluation ought to be something that is done continuously and that is built into everyday practice. In keeping with the principles of community development, it should also involve all people that have an interest in a project or programme.

My experience of developing indicators of capacity building outcomes for an evaluation of a community project in Fife, Scotland, highlights the importance of developing approaches to evaluation in keeping with the principles of community development. There are lessons here for: (i) the measurement of “softer” outcomes; (ii) making evaluation an integral part of everyday practice; and (iii) the use of similar approaches in other contexts.

Evaluating the Crosshill Project
The Crosshill Project is a voluntary sector project run by Fife CVS. It aims to build community capacity and good governance structures through a process of organisational development with small community groups. To be eligible for support groups have to have incomes of less than £15k a year. Groups work on a variety of community development issues relevant to social inclusion.

The project provides training events and seminars on topics such as conflict resolution, better meetings, constitutions, financial management, committee development, roles and responsibilities and funding options. They organise ‘social suppers’ which are networking events so that group members can meet people from other groups and issue regular newsletters and other publicity.

It is unusual for evaluation to be planned from the very start of a project and my work in Fife started about one year into the three year funding cycle. The approach I took was designed to maximise the opportunities for learning by engaging the project staff and community organisations in the process. The approach was flexible and used different methods to encourage participation.

The project had an annual target of working with a certain number of new groups each year. The monitoring data that they had collected gave information on outputs: basically the things that they could count, such as the number of training events held, and the number of people/groups that had attended. They had not explicitly identified outcomes and so were not capturing large elements of their work and not doing themselves justice.

Research methods for inclusion and dialogue
One of the problems with using evaluation toolkits and frameworks is that whilst they help people think through some of the issues and explain the jargon, they do not always point people in the right direction in terms of what methods to use to gather the data they need. Here my goal was to try something that would get them talking; discussing and clarifying how they thought their project might build community capacity and what that would look like.

First we did a review of project documents and exercises with project staff to explore their assumptions, processes and expected outcomes. In the jargon, we developed the ‘theory of change’ of the project; the links between the everyday tasks and the larger outcomes.

An effective technique was to get the team to think about the project “as if it were a tree”: the roots represent the origins of the project; the trunk its heart or core values; the branches and leaves its activities; and the fruits its outcomes. Understandably, this idea was received with some initial scepticism, but the process ultimately provided valuable insights. The team realised that project outcomes were not being measured and they spelt out what the goal of “active communities” would look like, so that groups would be: effective; efficient; independent; skilled; open to learning; able to apply for funding; democratic; inclusive; confident; empowered; able to act in their sphere; and supportive of peers.

Although we identified outcomes in this way, they still seemed rather abstract and difficult to measure and we needed to develop some indicators for them. So the next phase was an exercise with the community groups themselves. We wanted to test whether we had identified the right outcomes from the groups’ perspective and wanted them to assess their own capacities or progress towards those goals.

Making it visual
We conducted a kind of participatory appraisal exercise with fifteen people from six different groups that attended a social supper. Twenty-six different capacities or skills were identified in advance, grouped broadly into four headings:
· communication
· governance
· financial and administrative
· networks, action and other

Each skill or capacity was written out in large print on a separate piece of paper and stuck on the wall. Each person was given a supply of coloured stickers and asked to indicate their views about the degree to which the group of which they are a member had the capacities or skills indicated. They had four options:

a) Green sticker - We have this now
b) Amber sticker - We’re not sure, but think it might be important
c) Red sticker -We don’t have this now, but we’d definitely like to develop it
d) No sticker - Not relevant

They were then asked to go round again and list their top five priorities. There was an immediate visual impression of their responses. It was a lively and engaging exercise. There were no omissions of skills and most were of importance to most of the groups.
The exercise demonstrated the value of a participatory technique in this setting. It built in an element of self-appraisal and allowed groups to prioritise different outcomes. It developed mutual understanding of what “capacity building” might mean in practice, in everyday language. Importantly, it was also an enjoyable technique that gave ‘equal voice’ to all and helped the groups “find a way to talk about complex issues”.

Lessons
This approach recognised the assets, skills and capacities (rather than deficits) that already exist amongst groups. It allows groups to decide for themselves which skills and capacities they need to develop in order to achieve their goals. The self-appraisal element is important and in keeping with the ethos of developing autonomy. It is essential where such “softer” outcomes are involved. Although twenty-six might seem rather a lot of indicators, it is important that a “family” of indicators is developed that encompasses the breadth of the vision of the project.

Such an approach could be used in a more explicitly developmental way to assess the “distance travelled” of groups over time, whilst also generating evaluation data throughout the process. Here the intention was to develop the indicators, not establish an overall baseline. It certainly got them talking to each other about what they ought to be doing.

Cathy Sharp works for Research for Real. Contact info@research-for-real.fsnet.co.uk
Edited - S. Simanowitztop