Do you want to evaluate an initiative similar to this:

Major systems change initiative where the intervention initiative aims to “tip” a system in a major new direction…

Or maybe like this:

Many different agencies and project teams working collaboratively on the same problem with complicated interactions, impossible-to-attribute outcomes, diverse responses to unexpected events…the challenge is ongoing development of the collaborative effort and providing feedback about its effectiveness.

These are two of 10 scenarios that Michael Quinn Patton gives to illustrate where developmental evaluation is an appropriate evaluation approach.  These are in his new book Developmental Evaluation:  Applying Complexity Concepts to Enhance Innovation and Use.

Patton is father of the field of developmental evaluation (DE), as distinct from development evaluation.  DE is a burgeoning field working with complex adaptive systems, contrasting to the traditional log frame and other static, linear approaches that can cause havoc with change efforts.  Patton explains that DE “…facilitates ongoing innovation by helping those engaged in innovation examine the effects of their actions, shape and formulate hypotheses about what will result from their actions, and test their hypotheses about how to foment change in the face of uncertainty in situations characterized by complexity.” (p. 14)

The book is a wonderful overview not just of the approach, but of the many years of Patton’s work to develop it.  DE is not a methodology – it encompasses many different methods, including traditional ones such as surveys and participant observation.  I think of it as a stance: as in action research, the evaluator is a co-participant in the development of the initiative, actively working with others to draw out learnings and integrate them into actions with action-reflection-planning-action cycles being built into daily worklife.  DE is a practice, that aims to pragmatically guide workday actions and promote the type of leaderful culture so important for change networks.

This contrasts with formative evaluation that gets a program model ready (working out the bugs) and summative evaluations at the end of a project to assess “did it work?”  These use frameworks like SMART (Specific, Measurable, Achievable, Relevant, Time-bound).  The evaluator is typically thought of as outside of the project being evaluated, being a dis-interested observer and analyst who delivers periodic reports.

Single-loop, double-loop and triple-loop learning

Good questions and learning are foundations that unite all the evaluation approaches.  Patton brings up single-loop learning (asking questions within the established policies, structures and goals…eg: are we doing well at providing people fish to eat) and double-loop learning (asking questions about the policies, structures and goals…eg:  should we instead be teaching how to fish for people to feed themselves).  Oddly he does not raise triple-loop learning (asking questions about how we think about an issue…eg:  how do we understand the eco-systems-fish-consumption relationships).  This omission seems rather odd, since triple-loop learning is really about learning how to learn…and perhaps defines a limitation of Patton (triple-loop learning is a more recent concept) rather than of DE itself.  He places DE’s focus more at the double-loop level;  I would say that it moves into triple-loop as well.

The book provides a very good comprehensive discussion of fundamentals concepts behind DE, such as systems thinking, adaptive cycles and approaches to change.  He presents an additional useful take on the distinctions between simple-complicated-complex:  the key variables are the degree of agreement about what to do and the ability to define the impact of actions.    (Exhibit 4.5)

However, all this is similar to many other books.  The key value is Patton’s ability to give great examples.  He identifies 10 types of complex systems development, including the two described at the beginning of this blog.  Then he gives detailed analysis of how to address each, including key questions, time lines and design/methods options.  For each he also gives an applied example with commentary.

Patton likes the word “bricolage”, a French term he appropriates to describe “combining old things in new ways, including alternative and emergent forms of data collection and transformed evaluator-innovator relationships.”  Together with his descriptions of formative and summative evaluation, this makes me think there is great value in understanding how to combine these traditions with DE to create comprehensive approaches to the concept of evaluation.  After all, as Patton emphasizes the old traditions have their place – it just needs redefining along with DE.

Michael responds:

Thanks for your review of the Developmental Evaluation (DE) book.  I agree that the omission of triple-loop learning is an oversight on my part — and unfortunate.  While the opening chapter notes the distinction between single and double loop learning as one tradition on which DE draws, the actual processes of and interactions around developmental evaluation more often than not involve and include triple-loop learning.  This is part of what is covered in the concept of “process use,” the learning about evaluation-based learning that results from going through a developmental evaluation process, which is a form of learning quite distinct from findings-focused learning and findings use .  Process use (see the book’s index) is inherently triple-loop learning.  However, as you point out, the connection is only implicit in the book and should have been made explicit — and will be made so in the next edition.  Thank you for that.

If you are starting or working in a network, you should use new mapping technologies to “see the whole”.  Knowing who is working in your field and their relationships is key for good strategy.  In a previous blog, I briefly introduced several mapping technologies.  Now I’ll give more details about one of the easier and quicker ways to map:  using web crawls.   They give a view of the structure of the “virtual (digital) world”, that is becoming an increasingly good description of “real world” relationships as the internet develops.

“Hyper-links” embedded in organizations’ web-sites that link to another organization’s site can be gathered through web crawls of internet sites.  A map such as in the diagrams below can then be generated to describe organizations’ virtual relationships.

I did this with the Global Organizational Learning and Development Network (GOLDEN), using the Issue Crawler developed by Richard Rogers at the University of Amsterdam.  The mapping was driven by the GOLDEN goals in terms of key stakeholder groups.  It aims to bring together leading academic research centers and businesses to spur attainment of sustainability.  The issue arena can be labeled “academic-corporate interactions for corporate sustainable responsibility (CSuR)”. The founders speak in terms of engaging 50 research centers and 250 corporations within a short time.  “Community organizing” is not framed as a goal, but it is an implicit activity to realize the goal.

Network Development Rule 1

Rule number one in initiating a network is to understand that someone is always already working in the issue arena…and to identify them if possible.  As in most cases, some of the leaders in the issue arena are among the founders of the new network—although they’re all academic CSuR leaders.  And as is also true in most cases in global networks, they are mainly older white men (like me!).  To realize a global network with all the complexions that implies for the issue, mapping can help enormously.

Issue crawls begin by identifying key URLs – referred to as “seed URLs” – relevant to your issue arena. In this case, I identified networks of organizations of two major stakeholder groups that are working CSuR.  First to note is that the issue arena is already quite crowded:  I identified 9 existing academic-business CSuR networks including ABIS, GRLI and UNPRME.  Also I identified 14 business CSuR networks including Business for Social Responsibility, the International Business Leaders Forum and the World Business Council for Sustainable Development.

Using these 23 seeds to conduct crawls produces data about URL connections and maps that display connections visually.  Some notes on “reading” the maps:

• Always remember you are looking at the virtual (digital) world of relationships.  Although it increasingly aligns with the real world, particularly for global issues, idiosyncrasies remain.  For example, educational institutions are much more reticent to link to external URLs than, for example, NGOs.  And some simply have much less developed web-sites in this regard…EABIS, for example, does not have outlinks;  GRLI has outlinks, but not for its institutional partners (but note that importance of a URL is suggested by inlinks –  links directed at it from other URLs).
• Nodes are sized according to the number of inlinks .
• Nodes’ relative locations are the product of their links and those of others in the network (nodes they are “closest to”).
• The maps here are all co-link ones which means they show nodes with two or more links in the network.  As well, they display the top 200 nodes’ links for the seed URLs (there are actually only 112-118 nodes in these maps, since some nodes have multiple links).
• In these maps I’ve color-coded the seed URLs for each map:  red for the academic-business networks, green for the business CSuR networks, and purple for multi-stakeholder networks (explained below).

In Map 1 (click on the map to enlarge) only eight of the seed URLs are among the top 200 nodes.  The map suggests two centers (clustering of big nodes):  one around intergovernmental organizations like the UN and World Bank, and another around multi-stakeholder networks, in particular the Global Compact and the Global Reporting Initiative (GRI).  This leads me to do additional runs that:

1. Exclude the IGOs…I perceive them as “background noise” that is masking the relationships in the issue arena I want to describe;  and
2. Include as seeds the URLs of multi-stakeholder networks’ (Global Action Networks’…GANs’) that are clearly working in the issue arena:  the Global Compact, the Principles for Responsible Investment, the Global Reporting Initiative and Transparency International.

Map 2 is a run excluding the IGOs.  It shows the business CSuR (green) nodes as central, the academic-business CSUR (red) seeds as fewer and more peripheral (suggesting the importance for them of their linkage to IGOs rather than business CSuR networks), and reinforces the idea that the GANs should be included because of their centrality and size.

Map 3 also includes the GANs as seeds (purple). We can see that there are more academic-business (red) and business CSuR (green) network seeds (10), which also supports the decision to include GANs and exclude IGOs.  The seeds for the business CSuR networks and GANs group, which would be expected as they tend to link to each other and the same organizations.

The Map 3 academic-business CSuR networks (red) are comparatively small, non-central and dispersed; three are really part of an educational grouping that suggests their orientation towards educational institutions is significant stronger than towards businesses (if they were balanced, you’d expect to see them with the GANs);  the two Asian ones are quite different with Asian associations.

Each of these maps is accompanied by several types of data-base outputs summarized in this excel spreadsheet.  For example, Columns B-C list all the nodes in the network (I set the maximum at 600 nodes) by inlinks;  another data output even gives lists by web-page, to identify locations/people within large organizations that are relevant.

In a run using snowball analysis (rather than co-link) the crawl retains URLs with at least one link from seeds.  Run with the three stakeholder groups, this produced a list of 5317 URLs (Column D).  And other maps show these by geography which more helps identify, for example, research centers in China. GOLDEN is particularly interested in particular geographies, like China.  More runs can be done for China in particular, and using Chinese-language web-sites.

Web Issue Hyper-Link Crawls and Strategy

So here are some ways all this work helps strategically.  It gives:

1. A feel for the domain structure…the “ecology” of relationships.  Who is dominant?  What are the groupings that are important to understand for success?  The maps help create a network perspective, by giving a visual association with the abstract “space” in which an initiative is working.
2. Strategic options…who to partner with and “reweave” the nodes so they can realize a network’s goals.  Everyone has their own “network” of relationships, but how do we broaden this perspective and who should we prioritize to engage?  GOLDEN’s founding academics obviously want to partner with corporations.  But should they do this by weaving together existing networks?  By going directly to corporations?  Some combination…and if so what?  The data bases are particularly helpful in identifying potential partners.  They include many education institutions and corporations and their linkages, to help identify who would likely be most interested in becoming involved in GOLDEN, and who would come with the richest network.  The snowball crawl, for example, identifies 5317 potential partners…although only a small fraction of these will be of interest to GOLDEN, it is a universe of options that is much more comprehensive than traditional snowball methods.  Should the IGOs be actively considered as partners?
3. Strategic clarity…what would you like the issue arena/map to look like after your intervention?  What are the “holes” in the map where relationships should be forged?  Today the map does not have “GOLDEN”;  what would the map look like in three years time if GOLDEN is successful?  This has an “impact evaluation” suggestion, thinking that GOLDEN’s inlinkage placement on the map in a few years can be one way of assessing its success.  As well, this helps guide questions about partnering priorities and web-site development.
4. Communications…who are the gatekeepers? Who should GOLDEN “get on board” to facilitate communications?  Naturally enough, many nodes concern communications.  This includes social media web-sites, publishing ones and traditional media ones.  This is very useful for development of a communications strategy.
5. Other…there is much useful information.  For example, maps show foundations and resource-organizations working in the issue, to help define a funding strategy.

Of all the benefits, however, perhaps the greatest is simply helping people to think more in network terms.  Although not as helpful in this regard as something like value network analysis, web crawls are a great step forward.  And of course if you’re interested in me helping you apply these types of analyses to your situation, email me!

There are many different ways to approach impact measurement, but using the wrong methods can actually undermine a change network’s efforts. The value of appropriate impact measurement is that it not only helps explain to funders their return on investment, but it also is an important tool for priority-setting, decision-making and managing.

Traditional evaluation approaches come from an industrial “in-put/out-put” model.  This is fine for simple tasks, but it is inappropriate for complicated and complex tasks that are part and parcel of change networks.

Simple, Complicated and Complex Activities

Three key differences in these types of tasks in the Table reveal that a change network does all three activities.  However, these networks are distinguished by an over-arching mission that requires complex activities.  Therefore, although the networks need impact measurement methods that will address all three activities, their umbrella measurement method must accommodate complexity.

In change networks, the need for methods that can address complicated and complex activities is evidenced in a number of ways, such as:

• Methods for evaluating simple tasks can not address the complications of the interaction in network participants’ relationships.
• There is not one, but an emergent number of possible pathways that require exploration and development to address issues such as ending corruption, creating sustainable forestry and integrating triple bottom line imperatives into corporations.
• Change networks’ visions require a long time to realize.  With all the change in their operating environments over that time, adaptive strategies are required, although simple ones can be good for relatively short-term sub-initiatives.
• Change networks usually do not aim to “take credit” for the actual valued outcomes (such as healthy, happy people).  They aim for a back seat in favor of their participants’ being recognized for their work. This makes attribution, a cornerstone of traditional impact measurement, highly problematic.

The demands of complex systems are reflected in “developmental evaluation” (DE), both an approach and title of a book by Michael Quinn Patton about to be released.  Michael writes:

“Developmental evaluation supports innovation development to guide adaptation to emergent and dynamic realities in complex environments. …  Informed by systems thinking and sensitive to complex nonlinear dynamics, developmental evaluation supports social innovation and adaptive management.  Evaluation processes include asking evaluative questions, applying evaluation logic, and gathering real-time data to inform ongoing decision making and adaptations.”[1]

As in action research strategies, the evaluator is part of the development team from beginning to end, rather than someone who comes in at the end to simply do a post facto analysis.

Two Examples

Ricardo Wilson-Grau, a colleague who works with the DE approach, points out there is a number of methodologies that can be used under that heading.  He has, for example, practiced DE using Outcome Mapping with the Global Partnership for the Prevention of Armed Conflict (GPPAC) and the Global Water Partnership (GWP).

Ricardo explains that traditional evaluation poses questions such as:

• Were problems encountered in implementing the change strategy solved in a way that is faithful to the model?
• To what extent have the intervention model’s specified outcomes been achieved?
• What has been learned about how to fully and faithfully replicate the model?

DE, however, is more interested in answering other questions about the strategy as something in development. For example, the Global Platform for the Prevention of Armed Conflict (GPPAC) introduced an Outcome Mapping in 2007 as a planning tool. In 2009, Ricardo advised on:

• How can the success of Outcome Mapping be judged so far?
• How can Outcome Mapping now be developed as an evaluation/monitoring system vis-à-vis results?

The first question was a reflection on the system itself;  the second was about further development of the system.  Based on those findings, GPPAC is now further developing Outcome Mapping.

Another example is with the GWP.  GWP operates in a highly complex, dynamic environment. It has thousands of members who are constantly changing, grouped into 60-70 country water partnerships, whose actual number at any given moment is unknown.  These country partnerships are grouped  into 13 regional water partnerships with a global secretariat in Stockholm.concerns the approach to measurement. Over ten years they had placed the issue of integrated water resource management on the environmental agenda.

In traditional evaluation performance and success are measured against predetermined goals and SMART outcomes: specific, measurable, achievable, realistic, and time-bound. DE is quite different.  With Ricardo’s support GWP created a monitoring procedure to apply DE principles to  develop measures and tracking mechanisms as outcomes emerge. They introduced the procedure into one region and, according to what did and did not work, adjusted it for the next region. That is, the measures could change as the process unfolded. They tracked the forks in the road – specifically how different regions had to adjust the monitoring procedure – and used this information to point out the implications of key decisions as the innovative monitoring system evolved. Consequently, their donors are being informed of the governmental policy and practice changes that GWP – directly or indirectly, usually partially and often sometimes unintentionally – influences.  That’s simple, complicated and complex.

Ricardo is an independent evaluator and organizational consultant based in Brazil and the Netherlands. He can be reached at ricardo.wilson-grau@inter.nl.net.

---

[1] Patton, M. Q. (2010). Development Innovation:  Applying Complexity Concepts to Enhance Innovation and Use. New York, NY, USA, Guilford Press.