The METRONOME project (A Methodology for Evaluation of Project Impacts in the Field of Transport) was financed under the 7th Framework Programme (FP7) of the European Commission.
The project aimed to develop a methodology for evaluating project achievements supported in FP5 and FP6 with particular focus on:
- Strengthening industrial competitiveness (IndCo);
- Contributing to sustainable development (SuD);
- and Improving community and public policies (CPP).
The expected impact set for the project was to use the results and findings from the FP5 and FP6 projects to contribute to the definition of intermediate performance targets for FP7 and new research policy objectives.
The developed METRONOME screening, selection and evaluation methodology has three main phases:
- Identification of European transport research and policy objectives for Industrial Competitiveness, Sustainable Development, and community and Public Policies;
- Screening and selection of FP5 and FP6 themes and projects for the evaluation, and
- Evaluating project impacts through the METRONOME impact model, using a multifaceted approach.
The methodology takes a two-dimensional approach to project impact evaluation. On the one hand, it evaluates the projects' achievements in light of the FP5 and FP6 work programme objectives and targets set for IndCo, SuD and CPP themes. On the other hand, it evaluates, through the METRONOME impact model, the impacts of the FP research projects according to four impact groups and related indicators. The impact groups are:
- impacts on management and co-ordination;
- scientific impacts;
- customer/end user impacts;
- and societal impacts.
In general, the METRONOME evaluation methodology proved to be useful in producing information for the definition of intermediate performance targets for FP7 and of new research policy objectives from the perspectives of:
- achieving the FP5 and FP6 objectives and targets;
- FP5 and FP6 implementation and operational environment;
- and research project outcomes and impacts. Testing the methodology illustrated that different mixes of evaluation methods (both reviewing and surveying) are needed to gain in-depth information on both FP target achievement and potential project impacts over various time perspectives.
In the course of the METRONOME project, the developed methodology was applied to a sample of 100 FP5 and FP6 transport projects in the fields of industrial competitiveness, sustainable development and community and public policies. For each of the evaluation themes, a specific application with different mixes of evaluation methods was developed and applied.
The evaluation methodology used proved to be useful in producing information for definition of the intermediate performance targets for FP7 and for new research policy objectives from the perspectives of: achieving FP5 and FP6 objectives and targets, FP5 and FP6 implementation and operational environment, and research project outcomes and impacts.
The main difficulty encountered during the METRONOME methodological development was the availability of project result (e.g. Final Report) data. A structured, up-to-date FP project result database that is ready and available for the evaluators would enable more reliable, less time-consuming and less costly FP impact evaluations. Other major difficulties identified during the methodology development were the relatively low response rates in co-ordinator surveys and interpretation of the multi-level objective and target structures of the FPs as the basis for evaluation. As regards the future methodological development, only a repeated (and simultaneously elaborated) evaluation process can provide more detailed analysis of project or programme impacts. Further, and as a complement to the former, more emphasis and resources are needed in integrating such future elements into evaluation methodologies that can better support the future WP objective and target setting in the changing European transport and research environment.
Evaluation of FP5 and FP6
It seems that FP5 and FP6 have certainly played a significant role in the European science and technology agenda. For evaluation of the role of the FPs on the global map or on their contribution to EU research competitiveness at international level, the project sample does not give a representative insight.
Modern infrastructure and smart funding: A coherent funding framework