D.4. Comparison and evaluation
The objective of the session is to understand how to contribute effectively to the comparison and evaluation process.
The session is used during the comparison and evaluation, prior to developing evaluation criteria.
The session is intended for the evaluation stakeholders who will be involved in the definition of evaluation criteria.
This is a relatively informal introduction to the comparison and evaluation process. The initial parts of the schedule may seem a bit simplistic, or even a waste of time. However, they are intended to build a common understanding of the evaluation process amongst the group, and to get the group used to working together. They are an opportunity for the group to try out some of the techniques before the more serious work of defining evaluation criteria for real.
If a more “serious” schedule is required, remove steps 2 and 3, and shift step 4 to after step 6.
Introduction
Learning objective: understand the purpose and nature of the session.
Method: present that this session introduces the approach to evaluation. It is a light-hearted session, but an opportunity to make sure we all see it the same way.
Evaluation basics
Learning objective: understand the need for structure, criteria, measures and impacts.
Method: discussions based on “What do you look for in a car?” Bring out that we need some structure, well-defined criteria, pre-defined criteria, groups of criteria, defined grades, measures, impacts (both show stoppers and caveats).
Shared evaluation process
Learning objective: appreciate the need to share in the evaluation process.
Method: demonstration of giving out chocolates unfairly. Present the need to balance different interests, the need to be inclusive. Explain that we will use weighting and if necessary voting.
Effective criterion definition
Learning objective: define criteria effectively.
Method: exercise comparing criteria. Show criteria that are vague, ambiguous, trivial, obvious, off-topic, not relevant (is the car kept in the garage or on the road). Recap with summary of good criterion definitions for evaluation.
Process overview
Learning objective: understand the process of criterion definition, assessment, analysis and decision-making.
Method: presentation of the process.
Criterion definition process
Learning objective: understand how criteria will be defined.
Method: presentation of how criteria will be defined in a series of workshops, using system governance criteria as a base, amended and added to as necessary.
Analysis
Learning objective: understand how analysis will be applied.
Method: presentation of the analysis method, of scoring and of impact analysis.
Summary
Learning objective: summarise what has been learnt.
Method: summary sheet of process, and of criterion definition requirements.