In my last post I wrote about why we embrace standards and universal metrics, but cautiously. Since then I’ve seen a number of posts that echo my sentiments and also contain some more detailed explanations of why a veteran of the public sector would be wary of more big indicator databases and measurement requirements.

Alex Jacobs of the NGO Performance blog wrote a great post back in August, Tools Not Indicators. It’s not as rash as it sounds.

The inimitable and invaluable social thought blogger Duncan Green, of From Poverty to Power fame, also wrote addressing the question of randomized trials which would be required for real tracking of quantitative indicators on a large scale. (If you work in the international social sector, you really need to be following Duncan’s blog.)

What posts like these highlight is twofold.

First, that in terms of tracking social impact, the public sphere is light years ahead of the private, for-profit sphere. And that makes sense. The for-profit sphere is used to tracking profits. They are new to social change. It is in all of our interest for them to catch up as quickly as possible. We can’t afford to make the same mistakes twice.

Second, that we do already have ways to track impact, and that we are tracking impact. It’s really a question of putting our money where our mouth is (putting resources into tracking impact) and following through with the tough management decisions that are required to make impact measurement useful.

But I think that before we go ahead with that, we need to look at what it means to balance standards, requirements, participation and staying on the edge of chaos in the real world.

So how does this work in practice?

I have to admit that I use my philosophy training here. First we analyze, and then we synthesize. We start with the smallest parts and look at the individual pieces, in other words, break apart the whole vision of social change into the individual changes people want to see in their lives.

We start by asking the end-users of each organization what their idea of change is. Then we work with the organization to ensure that the stated and actual vision of the staff and management are aligned with those whom they serve. At this point, we can formalize some of the things we want to measure and decide how we are going to measure them (who, what, where, when). This is the start of synthesis and comprises the first or second drafts of a performance management or monitoring and evaluation plan. It is a conversation: we ask, we listen, we repeat to clarify and confirm, and then listen some more. Sometimes, when working with well-developed organizations that have participatory practices ingrained in their cultures, the dialogue will be quick and formal with a little cross-checking. But for a start-up that is either weak on participation, customer feedback, or formal data and/or profit tracking mechanisms, there will be a lot more discussion to build up a framework for performance management.

The PM or M&E plans are formalized and checked for approval with the organization or venture in question. At this point, we get to the higher order of synthesis.

• We collate indicators that are similar or the same. For example, “% and number people in the catchment area within X meters of clean drinking water X times per day” may be an indicator for several enterprises that sell clean water at affordable prices. We ensure that these more universal indicators are in line with those monitored by GIIN and IRIS, the Sphere Standards, and the Millennium Development Goals, so that the data can be used on a larger level, and also to ensure that the standard meets minimum standards set by the UN and humanitarian agencies.
• We create category groups for diverse indicators that together indicate movement towards a social goal. For example, “# and % of women who have their own bank account” and “# and % of women that belong to a self-help savings group” can be put under an umbrella “women’s empowerment” indicator such as, “# and % of women that have a means to make, save, or borrow money with their own name”.*
• We may also have indicators that stay very context-specific, such as, “# of the parents which attend one non-mandatory event of the parent-teacher association”. These are useful because they help triangulate the more general and more universal indicators, and also because they bring a human element into performance management that is easy to forget when dealing with so much quantitative data. They also provide diversity of thought that presses us to think harder about the limits of what we are doing. They can be used as an indicator bank for others, and eventually could be categorized if that type of indicator proves useful.

Based on the indicators we develop from the bottom-up, and standardize with those set top-down, we can work with each enterprise or organization to create a plan for collecting and using the data. IFAD, the International Fund for Agricultural Development, has been working on income-generation for farmers for decades and has one of the most detailed guides to tracking progress and impact on the Internet. Their M&E Matrix is particularly informative for those new to development, philanthropy, or managing for impact. Because Insaan works with a variety of organizations and does not seek to micro-manage, we do not have our own template. Instead we work to optimize existing systems to minimize duplication and paperwork. But for those with little experience in M&E, starting with a template can be helpful.

Using in-person meetings and informal and traditional data-collection methods that are transferred to paper, and then electronic recordings fed into a database (this is increasingly direct), we end up with a very robust system to collect information on impact and the indicators of the process that will take us there.

But finally, and most importantly, we don’t monitor everything. We leave some of the monitoring and some of the data in the hands of informal mechanisms like village councils, or the gossip among teachers in the staff room. We leave room for informal reporting when we go to the field and by encouraging good management in which managers and owners spend time on the sales floor, so to speak. And we are not afraid to let go of what is not working and move to another, better indicator, or to look at something that appears to be working very well even though we cannot yet put our finger on just what is working.

Next week: Our impact metrics challenge. We’ll ask social enterprises and organizations to submit their plans and indicators for impact and link those to standards and provide feedback on how to optimize measurement and management for impact.

*Why not just ask everyone to use the less specific indicator? Most people do well when tracking concrete, specific indicators that they designed themselves. So long as the relationship between the category and the indicator remains clear, categorizing them remains more efficient.