Both in INSITE and MD (EU sponsored research project close to INSITE) we are developing a new method of evaluating innovation projects, especially social innovation projects, called Dynamic Evaluation.
The key idea behind this method is that standard evaluation, based on a set of pre- defined outcome variables to be checked at the project’s conclusion, does not provide a valid measure of the real value of a social innovation project. Those who provide the financing want accountability, and in our society that accountability usually has the form of justifying project financing in terms of the project’s impact on parameters as growth and jobs. But these parameters so often miss or underestimate the very objectives of this kind of projects, and the real impacts they finally have, considering the social values and transformations they promote, not to talk about the complex ever-changing environment in which they are carried out and long term perspective considerations so often absent in standard evaluations.
Discovering new patterns of interaction, and the generative relationships to which they give rise, emerging potentials, unforeseen transformations in social organization and values, and similar unpredictable innovations must be a primary goal of evaluation – as they are a primary goal of such projects – and no a priori choice of measurement parameters can capture these emergent features. Even if the evaluation parameters are expressed in different terms than growth and jobs, such as measures of quality of life and social well-being, and even assuming a co-participation strategy by which all people involved in the project have been consulted in formulating the a priori evaluation criteria, the problems and constraints of a static evaluation remain unaltered: a static set of people and interests, knowable a priori, with a static set of values and opportunities that will not be augmented or transformed by the changes that follow in the wake of the project simply does not correspond to the distinctive, unpredictable, ever changing innovation processes.
Providing a representation of the processes whereby the project unfolds in time constitutes the project’s evaluation: what we mean by the representing the process is making visible the changes, opportunities and potentials that emerged during the project, the set of people involved as it vary over time, the transformations in relationships and attributions these people have experienced through their interactions with the project, and the values they assign to these transformations. A representation that makes these things visible has value not just for the organizations and agents who enable and lead the project – because it shows who is affected, how, and what it means for them, and thus can help them to act – but for all participants, and funders too, who can obtain a much richer, more nuanced and socially relevant picture of what the return on their money is, than the reductive and falsified accounting in terms of growth and jobs.
The research developed to carry out such a dynamic evaluation unfolds basically in two different directions. The first one, ethnographic, involves developing techniques and protocols to elicit the required information dynamically, as the project progresses from inception to implementation, following the cascades of changes it induces. The second is technological: to construct information tools to code, store and analyze the information collected and to represent it visually in such a way that it can be used by project participants to both evaluate the transformations the project is inducing and to act in ways that can help move these transformations in what participants regard as socially positive directions.
We are now implementing such a dynamic evaluation methodology in the study of Green Communities – a project financed by Italy government to better the quality of life, energy production, etc. in some Italian mountain communities, leading them to both ecological and economic sustainability. We are still in the middle of the process, but the results so far are really interesting, both for us as researchers as well as for the projects leaders. We will also start in September a dynamic evaluation to another European project – Educational Pioneers – lead by INSITE’s partner Kennisland in Amsterdam and oriented to improve quality educational system.