C.6. Evaluation criteria development workshop
The evaluation criteria development workshop defines and agrees a set of criteria to use the for evaluation of options in a system selection.
The workshop takes place in the early stages of the comparison and evaluation process, to define the criteria to be used for the evaluation.
The workshop is lead by the system governance manager, or by someone appointed by them.
The workshop involves the evaluation stakeholders.
Before the meeting, a decision will have been made to extend the evaluation criteria beyond the standard set used for system governance. (If no more than the standard set is to be used, then there is no point in running a workshop.) A decision may have been made to include functional criteria (business requirements), in which case the evaluation stakeholders will include people who can represent the functional requirements.
The evaluation stakeholders will have been trained in the evaluation process. See Section D.4, Comparison and evaluation.
The workshop does not include the actual drafting of new and changed criteria, or the sign off of the criteria. The drafting should take place after the workshop, and the sign off by the evaluation owner and stakeholders should follow normal practice for your organisation.
The agenda is written as if the workshop will be run as a single session. In some cases this will be possible, because the workshop will only involve a few additions and revisions to the standard criteria. Often, however, a series of workshops will be required.
If a series of workshops is required, split the agenda as follows:
Include agenda items 1 to 3 in a single introductory meeting.
Run each iteration of item 4 in a separate workshop.
Plan to run one workshop for each of the high-level groups in the standard criteria, but modify this if the high level groupings change as a result of item 3.
Consider varying the attendees for the workshops according to the criteria. For example, evaluation stakeholders representing business requirements may have little interest in technical criteria, and vice versa.
Consider drafting changes to criteria after each workshop in preparation for the final review workshop.
Run a final review workshop to cover the topics in item 5. If criterion changes have been drafted after each of the item 4 workshops, then this final review can sign off the criteria.
![[Tip]](../files/sghandbook/tip.png) | Don't go for too much detail |
|---|---|
Well-defined criteria are critical to evaluation, but it is possible to get carried away and define too many criteria and too much detail. Within the workshop, make sure each criterion will contribute to the decision making process or identify significant issues. Do not just catalogue everything you can think of. You will need to select and rephrase the standard criteria to make them suitable for the evaluation, but do not change their meaning and gradings without clear reason. |
Introductions
Introduce the participants. Explain the objective of the workshop. Describe the agenda and subsequent processes.
Present existing material
Present an overview of the standard criteria to the group. This will typically be the criteria used for system governance in the organisation. If system governance has not yet been adopted, present the standard criteria available from Metrici.
Explain that some will need to be rephrased to be suitable for evaluation (from “explain how the system achieves X” to “explain how the solution could support X”), some criteria may not be relevant to the evaluation, and that additional criteria will be required.
Agree high-level groups
Review the high-level groupings in the standard criteria, and discuss whether these need to change. Consider whether groups need to be added or removed, and whether weightings should change.
To gain consensus on weighting the criterion groups, consider a simple voting system in which each participant weights each group, and then the weights are averaged to give the agreed weighting.
The discussions of groups and individual criteria can be rather circular, because the weighting of the groups will depend on the criteria they contain. However, it is worth establishing some high- level structure and weighting before considering the detail of the criteria.
Agree individual criteria
Consider each of the criterion groups in turn, and review each of the criteria in the group. Consider whether criteria need to be added or removed, whether wording should be changed, and whether grades and scoring need to change.
Focus on the main criteria for comparing options and uncovering significant issues. Do not get diverted into detailed discussions on less relevant criteria. Do not change standard criteria (other than rephrasing them to read properly in an evaluation) unless there is a clear need.
To gain consensus on weighting the criteria, consider a simple voting system in which each participant weights each criterion, and then the weights are averaged to give the agreed weighting, as in the criterion groups.
Close meeting
Summarise the progress made, and thank the participants for their work.
Explain the next steps in the process, especially any further review, follow-up or sign-off activity.