4.10. Comparison and evaluation
The comparison and evaluation process evaluates the options in a system selection against system governance and other criteria, to measure how well the options meet requirements and to ensure that the issues with each option are known. This can be used as part of a request for proposal (RFP) or invitation to tender (ITT).
The process is an input into the system selection decision making, to be considered alongside functional and commercial factors. The comparison and evaluation process informs the decision, but should not be the sole factor in making the decision.
The comparison and evaluation process can be used whenever the organisation is running a system selection. It is particularly suitable for large and complicated selections that require a high degree of transparency. It can also be run less formally to provide a quick evaluation of options.
Although the process benefits from an established system governance regime, this is not a precondition. The process can use standard materials from Metrici as a starting point.
The process is requested by someone who is in charge of the broader selection process, and this role is referred to as the evaluation owner. The evaluation is likely to need input from many sources, and this group is referred to as the evaluation stakeholders.
The evaluation is lead by the system governance manager. and their team.
The evaluation needs input from the providers of different options. The term “provider” has been used, though some organisations might term this “supplier” or “vendor”.
The process can be used to support a major selection process, providing a transparent and auditable decision-making process. It can also be used less formally, to provide a quick evaluation of competing options. Some judgement is required to pick a suitable level of formality.
The process uses a set of criteria to compare different system options, typically competing software products from different providers. The organisation needs to decide what criteria to use. There are four options.
- Use the standard system governance criteria used for other reviews. This is simple and quick, and is suitable for an informal assessment. However, many of the criteria will ask questions that can not reasonably be answered by providers because they relate to how the system will eventually be run. For example, it does not make sense to ask a software vendor whether the system is included in disaster recovery plans.
Use the system governance criteria, but filter and rewrite them to make sense to the providers.
If the organisation wants to run a standard evaluation for all system selections, choose this option. Create a standard questionnaire that relates back to the standard system governance criteria, and use it for every system selection.
- Extend the system governance criteria to cover all non-functional requirements (technical requirements). Filter and rewrite the normal system governance criteria as in the second option, and then elaborate or add to these to cover other non-functional requirements. For example, the standard criteria might ask general questions about security, but the selection process might have more specific requirements.
- Extend the system governance criteria to cover all requirements, including functional requirements (business requirements). This is the same as option three, but with the addition of functional requirements.
Many organisations use this type of process for functional requirements, and if the organisation requires formal evaluation against functional requirements the fourth option can provide an effective framework. It would not make sense to run a separate process for functional requirements, because it is more efficient to run both in a single process.
Evaluating business requirements solely using this approach can lead to evaluations that concentrate too much on features and individual functions, and not enough on broader capabilities and commercial fit. Do not let the rigour of this process divert attention from a broader evaluation of a provider's offer.
In general, the most effective use of this process is the third option, to extend the system governance criteria to cover all non-functional requirements, but not functional requirements. Functional requirements may be better evaluated by a more open-ended assessment of how different products would support the required business processes.
Evaluation stakeholders provide input to elaborate and extend the standard system governance criteria. If the standard criteria are used (options 1 and 2 above), no stakeholders are required. Steps 2 and 3 of the process are not required, possibly replaced by some work to rephrase the standard criteria to be more relevant to providers.
Initiate evaluation
- Responsible:
- Evaluation owner
- Involved:
System governance manager Evaluation stakeholders
Engage the system governance manager to run a comparison and evaluation. Provide them with background information, the options to be evaluated, and the required timescales.
Discuss with the system governance manager the scope and depth of the evaluation. Will it just use standard system governance criteria, or be extended with further non-functional (technical) requirements? Will it include functional (business) requirements?
If the standard system governance criteria are to be used as the basis for evaluation, skip the next two steps. If criteria are to be extended, recruit a suitable group of stakeholders.
Train evaluation stakeholders
- Responsible:
- System governance manager
- Involved:
Evaluation stakeholders
Train the stakeholders in the comparison and evaluation process, and their role in it. See Section D.4, Comparison and evaluation.
Develop evaluation criteria
- Responsible:
- System governance manager
- Involved:
Evaluation stakeholders
Run an evaluation criteria development workshop to develop a set of criteria for the evaluation.
Draft the text of the evaluation criteria, and develop a template specifically for this evaluation. Review and sign off the criteria with the evaluation stakeholders, according to usual organisational practices.
Assess options
- Responsible:
- System governance manager
- Involved:
Providers
Working with the providers of each option, complete the textual responses of the assessment and ensure that they agree that the responses are factually correct.
It will usually be appropriate to grade the responses as a separate exercise, and to withhold the grading from the provider.
This step could be carried out by the evaluation owner or members of their team, if they are suitably briefed.
Validate assessments
- Responsible:
- System governance manager
- Involved:
none
Validate each assessment, and address any issues that arise from the validation.
Prepare evaluation review
- Responsible:
- System governance manager
- Involved:
Evaluation owner Evaluation stakeholders
Analyse each of the assessments.
- Calculate an overall score for each option.
- Analyse the scores for each option by criterion group, to identify strengths and weaknesses.
- Collate the issues identified for each system, and their priority.
Draft an evaluation review report to document the analysis.
Review the report with the evaluation owner, and then circulate it to the evaluation stakeholders. Also circulate the assessment responses and gradings for each option.
Choose best option
- Responsible:
- Evaluation owner
- Involved:
Evaluation stakeholders System governance manager
In discussion with the evaluation stakeholders, decide on a preferred option. This decision should be informed by the evaluation review report, but not based solely on it.
Inform the system governance manager of the decision.
Feedback to providers
- Responsible:
- System governance manager
- Involved:
Providers
This is an optional step, carried out at the discretion of the evaluation owner.
Provide each of the providers with a detailed analysis of their fit to the evaluation criteria. Explain what overall score they got, and any issues that were found. Explain in general terms, within the constraints of commercial confidence, how their offer compared with the others.
This step could be carried out by the evaluation owner or members of their team, if they are suitably briefed.
Repurpose assessments
- Responsible:
- System governance manager
- Involved:
- none
This is an optional step. It makes sense where the criteria are largely based on the standard system governance criteria.
Using the evaluation assessment as a basis, assess the chosen option against the standard system governance criteria.
If the chosen option will now be the basis for an implementation project, and the implementation project will be subject to a project review, this assessment can be the basis of the first assessment in the project review.