Process code for software procurement

Contents

  1. Goals
  2. Actions
    1. Considering quality and long-term value, not just up-front cost
    2. Creating a good selection panel
    3. Conducting a qualitative, user-oriented evaluation
    4. Conducting a quantitative and technical evaluation
    5. Continuing to communicate
  3. Common challenges
    1. Cost eclipsing quality
    2. Fear of the unknown
    3. Business model pigeonholes
    4. Poor vendor interactions
  4. Checklist
  5. References

Contribute on GitHub

  1. Introduction
  2. Orientation
  3. Planning
  4. Assessment
  5. Implementation
← Vendor interactions Contracting →

8. Bid evaluation

Diagram showing who should read this step 8

This step is for organizations:

  • buying off the shelf software
  • hiring a vendor to customize existing open source software
  • buying custom software

If you are building software in-house, read 6. Agile development planning and resourcing or skip ahead to the Implementation phase.

Goals

  • To conduct an objective, fair, quantitative and qualitative assessment of bids.
  • To consider direct and indirect costs associated with full software lifetime.
  • To conduct thorough technical evaluation that mitigates downstream contract issues.

Actions

Considering quality and long-term value, not just up-front cost

Because you introduced multiple types of criteria into the RFP, you can evaluate proposals based on quality in addition to price.

  • Does the vendor align with municipal objectives such as economic development (for example by hiring locally)?
  • Does the city maintain ownership and control of data?
  • What is the long-term monetization model for the software?

Creating a good selection panel

The selection panel will review bids using the evaluation framework stated in the RFP. Each member will give their professional assessment of the quality of each bid, then work together to assign scores.

  • The panel should reflect the stakeholder group, including:
    • A representative from the primary team or business unit procuring
    • A technical expert
    • A representative from the group of actual users
    • Additional individuals who were not involved with writing the tender
  • Do not include people who have conflicts of interest (anyone who stands to benefit or lose, on a personal or professional level, from a particular outcome of the RFP award).
  • Panelists should be briefed on the evaluation metrics (and ideally on a summary of the discovery research)
  • Panelists should review all bids separately before comparing thoughts and agreeing on scores.

Conducting a qualitative, user-oriented evaluation

If you are comparing existing solutions, you can test how well they address your problem statement by involving the end users. The best evaluation will come from people who will eventually be using the software.

  • Ask vendors for access to the software. Run the potential software through the scenarios included in the RFP (literally or as a thought experiment).
  • Schedule demonstrations from all bidders. Invite all of the end users who contributed to the discovery research and project scoping.
  • Replicate the city’s digital environment outside the city firewall, so that vendors (or other developers) can build and test on top of it.

Conducting a quantitative and technical evaluation

It is important to conduct a full technical evaluation of software on offer. Does it match up to vendors’ claims? Will it work within the city’s digital environment? Is it secure? Not every municipality has technical staff who can perform such an analysis, and civil servants should not be responsible for full technical auditing expertise.

  • Consider forming a technical committee or hiring a consultant for independent technical review. Explore hidden costs and issues, and determine the truth of vendors’ claims.
  • Ask the IT department (or the future users of the software) to test the software and ensure that it is compatible with existing technical infrastructure at the city.
  • If the software can’t be tested out of the box, require vendors to demonstrate a clear integration and deployment plan.

Continuing to communicate

Building long-term relationships with the software development community is important for the success of future projects. Continued communications will foster trust and mutual respect with vendors.

  • Give all vendors the courtesy of a post-award debrief (sending every respondent a copy of the evaluation and the winning proposal, offering to do debrief calls). This is also a good way of holding yourself accountable for conducting a good process and evaluation.
  • Host regular ‘office hours’ with current, past, and potential vendors. Learn how they work and what their goals are — and share how you work and what the city’s goals are.

Common challenges

Cost eclipsing quality

Quality, user experience, and integrity are very difficult to quantify. In an effort to maintain objectivity, evaluation processes often default to cost-only comparison, without adequately accounting for the actual value that the solution will generate.

Fear of the unknown

Choosing legacy solutions can feel safer – they appear proven or dependable. These legacy systems may be easier to understand and evaluate, but may not be the best solution for the problem statement.

Business model pigeonholes

Procurement norms were established for awarding high-value contracts to private sector companies. But defaulting to private sector vendors, without adequately considering non-profit or open source solutions, may sacrifice cost, quality, or opportunities for value-creating solutions.

Poor vendor interactions

Ineffective or disrespectful communications can result in bad relationships and compromise future procurement.

Checklist

  • Fair and objective evaluation of all bids
  • Choice of a winning bid or vendor
  • Feedback to all bidders

References

Open and ethical procurement guide on engaging with the private sector

https://opennorth.ca/publications/2hvkzrlujufylsvxgf7li5_en

18F: Questions to ask

https://derisking-guide.18f.gov/state-field-guide/questions-to-ask/

CityMart: Defining the evaluation framework

https://medium.com/citymartinsights/unit-5-defining-the-evaluation-framework-3e131df69632