Federal Field Guide

Deciding what to buy

The following sections identify challenges and strategies to mitigate risk during the research and solicitation phase of custom technology projects in government.

Conduct modern market research

By Mark Hopson

Challenge: Agencies do not use industry best practices for market research.

Executive summary

• Market research is vitally important to ensure successful outcomes with any acquisition. Given the dynamic nature of software, this is especially true.

• If done well, market research can provide reliable, accurate information that will help shape requirements, competitors, and ultimately the final product. If done poorly, then the project will be plagued from the beginning, which could lead to delayed schedules, increased costs, and ultimately, dissatisfied users. More often than not, many of the major failures involving government IT projects started off on the wrong foot based on the conclusions reached in market research.

Recommendation

What is market research?

Market research is defined as "collecting and analyzing information about capabilities within the market to satisfy agency needs." ^[1] FAR Part 10, the section of the FAR for conducting market research, is only two pages. It provides extraordinarily broad discretion for how agencies may conduct market research. Because of such discretion, many agencies rely on outdated ways to learn about companies, technologies, and broader trends that may influence a government technology program.

Market research is a continuous, dynamic process of determining, collecting, and analyzing the availability of commercial goods and services. This ever-changing state requires keeping up-to-date with changes in the marketplace.

A government employee can easily spend one third of their time educating themselves about a good or service they may eventually acquire.

For this approach, market research has two distinct methods: market surveillance and market investigation:

• Market surveillance is an ongoing process to stay informed on industry trends, new technologies, and general information about the marketplace of offerings for the goods and services needed to fulfill the mission.

• Market investigation is a comprehensive, focused process on specific sources, materials, or potential competitors to fulfill a requirement usually done in order to complete a market research report for an active procurement.

The easiest way to keep the two clear is to think of market surveillance as being strategic in nature, whereas market investigation is more tactical.

For example, consider something that is very popular because it is essential for digital products and services: user experience (UX).

Market surveillance would involve a series of broader questions such as:

• What exactly is user experience?

• What does the practice of user experience design consist of?

• What makes for a good user experience?

• What makes for a good user experience provider?

• What kind of qualifications and experience do good user experience providers have?

With market investigation the questions become more pointed:

• Who is a good user experience provider?

• Where can I find a good user experience provider?

• Are there professional associations or conferences for user experience?

• Are there any trade publications or other information sources about user experience?

• What work have they done before? Have they done any work for government agencies before?

• Do any of these companies have existing contracts through an available Federal Supply Schedule (FSS) through GSA or some other Governmentwide Acquisition Contract (GWAC)?

• Are any of them under a recognized socioeconomic program or status such as the 8(a) program or Service-Disabled Veteran-Owned Small Business (SDVSOB)?

By considering market surveillance and market investigation separately, the advantages become apparent. By consistently conducting market surveillance, a government employee has a baseline level of information that is accurate, relevant, and timely. An agency can dive into specifics (market investigation) whenever the agency need arises. This makes it far easier to complete necessary tasks like writing a market research report or an acquisition plan, or even conducting a competition.

When considering an agency's potential software needs, good market research is crucial to determine whether or not commercial off-the-shelf (COTS) will suffice or whether a custom agile development effort is needed.

Often, an effort's success or failure is based on a decision made in this early step before any solicitation is issued or actual development takes place. If an agency makes the wrong decision based on poor information gathered in this step, it often leads to what 18F refers to as "unrecognizably modified off-the-shelf" (UMOTS).

How to conduct market research

Everyone is a consumer. People buy all kinds of stuff in their personal lives. However government officials have to take different considerations into account when acting as stewards of taxpayer dollars. Agency staff must consider themselves as a public's buyer. Agencies must do rigorous research across numerous sources to gather enough of the right information.

Keep these points in mind when you are conducting market research:

• Market research is meant to be a forecasting exercise, not a legally binding process. Market research cannot be used in place of the government's source selection or evaluation process to determine a contract award, nor can it favor a specific vendor or company that will eventually be awarded a contract. It's also very difficult for people to keep themselves from jumping to this step (we're all human). Keep those two procedures (surveillance and investigation) separate. Don't try to rush right into picking something when the real goal is to just see what is available.

• Market research shouldn't decide a preferred or specific manufacturer, model, or brand. This is prescribed in FAR Part 11.104 for a not-so-obvious reason: honing into a specific manufacturer, model, or brand eliminates all of a buyer's negotiating leverage.

  This is one of the less appreciated reasons for the legal mandate to compete an agency's requirements to the maximum extent practicable. It's to ensure you don't lose bargaining power.

• Expect things to change. Agencies will rarely know the exact source to satisfy their requirements up-front. That's ok. Often needs will change by the end of a market research process, which will impact acquisition strategy and/or contract vehicle.

One big mistake people make when it comes to government acquisitions is thinking of every single step as a strictly linear, sequential process. If the process is mapped out, it's a common mistake to assume that market surveillance always comes first, followed by market investigation.

But this isn't really the case in practice. A good market researcher will conduct both market surveillance and market investigation as the mission needs dictate. Usually they occur in parallel, because one will inform the other and vice versa. To do that well, agencies need to think about how to generally gather reliable information regardless of the formal title for any given step.

In the Evaluate contractor proposals based on industry best practices section of this handbook, we provide a list of what to look for in a quality vendor proposal. If you are conducting market research to acquire this kind of developer team from a company, then you can use those same questions during market research. These questions are useful in the context of evaluating a competitive bid, but they can also be used to help you do your agency's information gathering and market research.

By the time an agency enters into a legally binding competitive bidding process, they'll be prepared and equipped with a much clearer understanding of what matters.

Sources of market information

Market research isn't really different from any other kind of research, so some basic concepts to consider about gathering information are "sources" and "methods."

Sources of information is a fairly straightforward concept. For buyers, it could mean going directly to a seller with pointed questions about some product. However, a government employee needs to account for bias and the reliability of the information they're getting from that seller. A single source rarely provides the most comprehensive understanding of anything.

Here's a layout for someone acting as a buyer for the American public.

As a best practice, government buyers should rely on multiple primary sources as well as secondary sources.

Primary

• Vendors

  • Manufacturers

  • Distributors

  • Resellers

• Other buyers

  • Private sector

  • Other federal agencies

    • Colleagues

    • State governments

  • Non-profit organizations

• Independents

  • Experts

  • Specialized consultants

  • Research companies

Secondary

• White papers or similar position statements

• Trade journals

• News reports

• Academia

• Subject-matter literature

• Databases

• Case studies

To put this into context, consider a major purchase in someone's personal life — buying a car.

Unfortunately, given the way that many government employees conduct market research and government contracting would be like someone walking into a single dealership they happened to be near, go to a single car model on the showroom floor, and pay the posted price for that car on the window immediately.

No one does that when they buy a car with their own money.

Cars are expensive, and reasonable people will do some form of market research before they buy. They will compare the available options of dealers, models, warranties, and all of the various factors to consider.

It should be the same when you are acting as the public's buyer.

Consider any typical exchange between a buyer and seller:

Buyer: Does your company offer X?

Seller: Yes of course we do. We sell the best X out there.

Buyer: Great, I am interested in purchasing it.

Seller: You should definitely buy mine over anyone else's out there.

Buyer: Why?

Seller: Mine is the best!

How do government employees know if the seller's assertions here are true and accurate? They can't know, not without other information to compare it against.

Methods for conducting market research

There's two ways agencies can engage with a source: directly or indirectly.

Direct contact is where an agency is engaged in a verbal or written conversation with a source. This is problematic when that source may be a future competitor for your requirement. Sometimes this will generate more useful or substantive information, but is far riskier and more easily misinterpreted.

Indirect contact is where an agency reviews material without directly engaging a potential future competitor, or talks with a disinterested party. The majority of market research should be done indirectly. Not only is it easier by far, but it is less prone to bias from direct conversations, especially with potential competitors.

Requests for Information (RFIs) are a very popular market research method but consider their limitations before using them:

• Often RFIs are the only market research that an agency may conduct before undertaking a contract.

• RFIs usually consist of a set number of questions rather than a dynamic, evidence-based inquiry developed over a period of time.

• RFI responses are a lot of work for most businesses, but especially small businesses and/or the types of "innovative" businesses new to government that are sought after.

• Because of the way that RFIs are set up, the majority of information provided in any response is mostly marketing material that is reused for numerous RFIs, regardless of the agency or topic.

• Because they are labor-intensive and can often come at a considerable cost to the company preparing them, relying on them often increases the likelihood of a protest.

RFIs have a place in market research, but most of their value has been eclipsed by the advent of information available through the internet. It's possible to find more reliable information through indirect research on the internet.

Considerations for sales and capture management

Whether agencies want to engage the market or not, chances are very high that if a government employee has anything to do with spending government money, the market will engage them.

Unlike many decades ago, there are now dedicated firms and resources designed to gather intelligence and package customer engagement solutions for companies to increase sales specifically for government agencies' budgets. Firms dedicate roles in departments for business development or "capture" in government with specific training regimens for effectiveness.

Salespeople can play an important role in helping to better educate potential customers about their company's offerings and helping potential customers reflect on their own needs.

But sales capture is also rife for potential abuse especially in the public sector where large amounts of government funds are being spent (the U.S. government is the largest buying entity in the world).

Government employees are working on behalf of someone else — the American public. This means that government employees have ethical and professional standards by which they must conduct themselves.

Numerous laws and regulations guide the communication or actions of government employees. If they don't adhere to these standards, such as FAR 9.5 for Organizational Conflict of Interest guidance, they may face civil or criminal penalties depending on the violation. This could mean fines, suspension, firing, and, sometimes, even felony prison time.

However, rather than scare government employees off from conducting market research, these regulations should reassure government employees. They give government employees protections from the kinds of pressure that the sales process can engender in the average person.

Considerations for any direct conversations with contractors:

• Any information that is shared in a conversation or a meeting could directly affect proposal preparation. By law, all potential offerors that could fulfill the agency's requirements must also have the exact same information, or they may suffer from an unfair competitive advantage. If information isn't shared properly, it could lead to a protest.

• All federal personnel have a responsibility to protect proprietary or confidential information, and not share such information with companies or potential competitors.

• Federal personnel must avoid appearance of commitment — only someone with a specific type of legal authority can obligate or commit the U.S. government through a contract (warranted Contracting Officer).

Sample approaches to use:

Start every conversation with some variation on the following phrases:

"Nothing discussed in this meeting authorizes you to work, start work, or otherwise obligates the government and is only for market research purposes. Any assumption on your part or on the part of your company is a mistake and has no effect on the government."

"We are talking as part of ongoing market research / for market research purposes only. This conversation in no way obligates the government, or should make you believe that we have entered into a contract of any kind."

Fairness: Treat all potential bidders, vendors, manufacturers, and everyone else fairly and impartially.

Sunshine test: Imagine that everything done has a public audience. Consider whether an disinterested, casual observer would believe the government employee acted responsibly and reasonably.

Institutional knowledge: Reach out to and rely on colleagues and other experienced procurement professionals to learn best practices for conducting interactions, documenting any exchanges, and developing requirements for competitive solicitations.

 
Government employees must ensure that they keep all transactions with sales people professional, transparent, and courteous. But they also have the same responsibility.

Here are some common sales tactics agencies face:

Cold contact: Where someone you probably have never met calls or emails you about their company or offerings and how it will help you.

Name-dropping: Mentioning someone in a higher position or a supervisor, or referencing a conversation that may never have happened to try and gain influence with you or sway you in the direction they want.

Networked intro: Best way is to develop a good reputation with one customer and then make that customer work for you by introducing or referring them to other potential customers within an organization.

Big pitch: Much broader or organized effort to provide a well-thought-out presentation to get a bigger group of customers excited or interested in buying a solution.

Talking to potential competitors can be extremely valuable, as the Office of Management and Budget's memo, Myth-Busting: Addressing Misconceptions to Improve Communication with Industry during the Acquisition Process, points out.

Don't avoid these conversations. But instead go into them adequately prepared. Be able to recognize these conversations' pitfalls and persuasive elements.

Why do any of this?

Besides being a good idea to do extensive research before buying, the FAR legally requires^[2] agencies to undertake some form of market research. If agencies have done an adequate amount of market research well, then they will be able to:

• Clearly communicate their needs or requirements to potential contractors

• Openly and transparently identify possible solutions or options

• Encourage innovation in designing and providing solutions

If agencies haven't done market research well, then they may end up:

• Being locked into inadequate solutions with poor quality contractors

• Inadequately describe requirements, that are then not met appropriately

• Hindering innovation

• Risking cost/schedule overruns

• Failing to meet mission goals

Depending on how extensive agencies want to search, and how much time agencies have to consider, research could take a considerable amount of time and effort. It's really at their discretion.

The size, scope, and complexity of the requirement will determine the amount of research and sources agencies gather information from. Should agencies take six months to conduct research for a product or service under the micro-purchase threshold? Probably not.

Should agencies spend a few months learning about custom software development projects, and find companies that seem to have a good track record before building a mission critical system? Absolutely.

Use the agile contract format to procure agile software development services

By Waldo Jaquith, Randy Hart, Mark Hopson, and Vicki McFadden

Challenge: Agencies are not structuring their contracts to support buying agile software development services.

Executive summary

• Legacy contract formats^[3] that detail hundreds to thousands of software requirements up front are not suitable for agile software development services.

• With the agile contract template, available as a template, agencies should procure developers' time, as prioritized by the government product owner through an agile cadence. Any contract must secure sufficient data rights for the agency in the work product or result of the development effort based on their mission needs.

Recommendation

For more than 20 years, using non-performance based formats, such as the Statement of Work, for professional services, has been strongly discouraged, as enshrined in FAR Part 37 - Service Contracting:

(a) Performance-based acquisition (see subpart 37.6) is the preferred method for acquiring services (Public Law 106-398, section 821). When acquiring services, including those acquired under supply contracts or orders, agencies must --

(1) Use performance-based acquisition methods to the maximum extent practicable

Performance-based service contracting (PBSC) emphasizes that all aspects of an acquisition must be structured around the purpose of the work to be performed with an objective assessment of contractor performance, instead of prescribing the manner in which the work is to be performed. It ensures that:

• contractors are given freedom to determine how to meet the government's performance objectives,

• appropriate performance quality levels are achieved, and

• payment is made only for services that meet these levels.

This proven methodology has yet to be fully implemented governmentwide for a variety of reasons, including inexperience in writing performance-based solicitations, cultural inertia, and concerns about more open and interactive communication with industry throughout the acquisition process.^[4]

The problems caused by not using performance-based methods is especially acute when it comes to the software development professional services. Government solicitations to procure non-performance based custom software are often long and complicated, include many pages of requirements, and can take months — even years — to write.

Structuring solicitations for agile projects

A quicker, more outcome-oriented method, such as 18F's agile contract format, would allow an agency to acquire an agile software contractor with a solicitation that's only a dozen pages and written in an afternoon. It can be executed under existing procurement regulations and within Contracting Officers' (CO) existing authority. It can save enormous amounts of time, and the contract structure supports — not impedes — agile development efforts.

The format is simple but contains a major shift in how agencies procure agile software development services. In our experience, these elements are needed to align the solicitation with agile software development best practices:

Use a Statement of Objectives — not a Statement of Work or Performance Work Statement^[5] — because, with agile, an agency doesn't know exactly what needs to be done, and can't possibly define it up front. The product owner, working with the development team, will determine on a sprint-by-sprint basis what work needs to be done. Anything else wouldn't be agile.

Contract for time and materials, not a firm-fixed-price. In most cases when an agency is purchasing for a software project, they're not buying a defined product, but instead buying a software development teams' time. Agencies can use a time-and-materials contract, as the CO may prefer, or a labor hour contract. Historically, the default contract types for IT projects were firm-fixed-price, based on the assumption that this reduces risk and aligns with the way software licenses have historically been offered.

However, if an agency is in a position to constantly measure ("inspect and accept" in agile) software quality, a time and materials contract — with a ceiling on total spending (or not-to-exceed ceiling) — allows for more flexibility for the software development team. A time and materials contract also allows for much easier escape clauses if the direction of the work changes or the contractor team does not produce quality software. If their work is inadequate, or their skills prove inappropriate, then no further work needs to be assigned to that contractor (effectively terminating the contract), and the contractor can be replaced. See our sample Determinations & Findings for more information on time and material contracts.

Have a short base period of performance, usually between 6–12 months. A couple of options to extend are fine, but keep them short, and never exceed a total contract length of three years. The agency is hiring a team to achieve a defined set of objectives and then leave. The government product owner should be heavily engaged throughout the project so contractor transitions should be relatively easy.

Maintain a nominal appendix of the backlog of user stories. User stories allow contractors to understand the specifics of the work that they're being asked to do, beyond whatever brief summary exists in the introduction. An agency must make it clear that the backlog has been included to illustrate the work that is presently believed will need to be done, but that it is only an illustration, and not a list of tasks that must be completed. It's a mistake to inventory user stories up front, rather than as part of performance. There's no way to know up front what work will need to be done, as is inherent to agile (design, build, test; inspect; repeat). The language provided in the agile contract format allows for this flexibility, since the backlog accounts for evolving outcomes rather than a predefined list of needs.

Write a Quality assurance and surveillance plan (QASP). This is the cornerstone of performance-based service contracting. Rather than an exhaustive list, we focus on key, objective criteria that are able to determine and ensure quality.^[6] For example, in our model, we require that at the end of each sprint, all code be complete, tested, accessible, deployed, documented, and secure.

Historically, the process of preparing software for use by its intended audience was complex, time-consuming, and risky. Standard industry practices have been automated making it simple, fast, and routine. Contractors must adhere to this practice ensuring agencies can deploy software themselves, without requiring highly technical staff to perform this trivial task. Allowing deployment to become complicated makes the agency dependent on that contractor, and makes it risky to replace them with a new contractor in the future.

We publish a sample QASP that can be incorporated as-is. We will continue to update this QASP as we discover additional, meaningful, and objective quality metrics.

Key personnel should include, at a minimum, the lead developer, and quite possibly the project manager. The agency is buying people's time, and wants to make sure that it's getting the people who were advertised. Be wary of specifying too many key personnel, though — this requires that the contractors put those people on the project if they get the contract, which can mean that they're functionally benched until the contract is awarded.

In a truly agile project, the project manager plays a very different role than in a waterfall project. Agile teams are self-organizing, which eliminates a major role traditionally played by a project manager. Instead, the project manager should serve as the contractor's interface to the client, be responsible for unblocking tasks and overseeing — but not controlling — the work performed by the vendor team. In many agencies, the project manager's role is usually required to prove that the agency is not engaged in personal services.

Allow for a distributed team, whenever possible. We advocate for remote development, rather than requiring the contractor team to be onsite at the agency's location. The best development resources are not in your city — they're spread out across your state and even the country. (There's a 150% difference in the salary of software developers between the most-expensive and least-expensive states in the United States.)

Using a modern technology suite, collaboration is easy. The product owner should always know what the team is working on, and the team should be readily available to answer questions or huddle on any issue that arises. They can do this with video calling for face-to-face collaboration (e.g. Google Meet, Zoom, etc.); instant messaging (e.g. Slack, Google Chat, etc.); task management (e.g. GitHub, Trello, etc.); and collaborative document editing (e.g. Google Docs, Office 365, etc.).

All work products must be published under an open source license — or, if using traditional proprietary code, the contract must secure sufficient data rights for the government in the work product to allow for use as intended and future development.

All work will be committed to a public code repository at least daily.

Keep technical proposals within page limitations, to minimize both contractor work and the time required for evaluation. Requests for more space may indicate that the contractor isn't taking an agile approach, or that they don't adequately understand what they're proposing. In 18F acquisitions, we typically ask for responses to be 2-3 pages.

The contractor must submit links to 2-3 source code repositories where their illustrative past work can be seen. Allow this to include work done by key personnel (e.g., the technical lead) outside of their employment with the contractor, since many contractors will not have any public repositories to point to for lack of clients willing to work in the open. This is a far better indicator for how they are likely to perform under real-world conditions rather than the attempted simulation of coding challenges or hackathons.

Evaluation criteria should emphasize the contractor's proposed technical approach and their similar experience / past performance. An agency should hire a team based on their experience and their general approach to the work at hand. There's little else to go on other than these two criteria. This focus also keeps the contractor's performance work statement brief and to the point.

Contractors' key personnel must participate in a verbal interview process. This provides an opportunity to hear directly from the people who will be doing the work, rather than contractors' capture managers, and usually proves a quick way of separating the wheat from the chaff. This is not the type of conversation that is defined in the FAR as "discussions," or "clarifications," and does not allow for any revisions to the already submitted, written proposal. Instead it's a critical quality control measure that confirms what was written down in the written proposal.

 
All of the above can be incorporated into a solicitation in a dozen or so pages, not including appendices like standard administrative clauses, a description of the technical environment that the contractor will be working within, the product backlog, etc.

You can put a solicitation like this together in a few hours. In a standard, three-day 18F procurement workshop, we use only the latter half of the third day to lead our client through an exercise in which we put together such a solicitation. With the right coaching, it's not hard.

Include the Contracting Officer (CO) in this process at every step of the way, starting as early as possible. When we host procurement workshops, we insist that the CO join the entire process, instead of just coming for the final day to hear about modular procurement and put together the solicitation. The CO needs to have the same base level of knowledge about the agile development process as everybody else in the room. Without knowing about agile software development, user-centered design, DevOps, etc., this prescribed contract format seems impossible.

Don't use hand-me-down government solicitations to procure agile software development services. By focusing on procuring competent development team(s) to achieve clear objectives, your agency can produce solicitations in hours, not months.

Use time and material contract types for custom agile software development services

By Mark Hopson, Vicki McFadden, Alan Atlas, and Waldo Jaquith

Challenge: Firm-fixed-price contracts are not appropriate for custom agile development.

Executive summary

• Use time and material (T&M) contracts with a not-to-exceed ceiling. If there is an engaged government product owner on the development team, they will know what the team is working on every day, so the historical risk of T&M contracts — costs spinning out of control — is reduced.

• Your team won't know the exact requirements for a software product before development, so don't use firm-fixed-price (FFP) for agile software development.

• Don't measure agile quality or outcomes by the completion of a number of sprints or user stories.

Recommendation

We recommend a time and material (T&M) contract for agile software development with a not-to-exceed ceiling.^[7]

Flexibility is an absolute necessity for agile software development. In an agile project, there are no predetermined requirements listed as "shall" statements.

Instead, there is a product backlog comprised of user stories, which the product owner regularly adds to and re-orders to reflect each story's priority. Priorities can change at any time based on user research, changing agency needs, law or policy changes, etc. There is no way to know in advance what work will need to be done.

In agile software development, an agency does not buy a defined product. Instead, it buys development teams' (developers, designers, content strategists, etc.) time. Therefore, the most appropriate contract type is T&M. The government, through an engaged product owner, can constantly prioritize the work to be done and measure software quality. The product owner should know what the development team is working on every day.

The traditional government fear of T&M contracts is that costs spin out of control. The empowered product owner^[8] reduces this risk though daily communication, and frequent — every sprint — inspection of performance against a quality assurance surveillance plan (QASP). We also encourage using a not-to-exceed ceiling to put an additional cap on expenses to protect the government's interest.

In addition to flexibility, another T&M benefit is that it allows for development team flexibility to scale up and down as needed. Stable teams are a core tenet of agile development. agile frowns on changing team structures more often than needed, but T&M allows the product owner to do so if necessary.

A real-life example: a new government product owner rightly determined that they could only manage one scrum team since the role was new. Six months into the project, the government product owner had mastered the role and was confident in their abilities. They decided to add an additional scrum team to speed development work. Once the key functionality has been built, the government product owner plans to return to one scrum team, likely smaller than the original. These changes are made possible by using T&M. No modifications are necessary because the agency is paying for the time humans spend building out the product — the number of humans and which roles are filled can change.

T&M allows agencies to save money, too. The contractor can only bill for time spent building a product. If there are delays hiring a new team member or a person goes on extended leave, the government is not paying for that time. T&M provides the opportunity for costs to come in under budget, which never happens with FFP.

A final T&M benefit is that if the contractor is not producing quality software — if their work is inadequate or their skills prove to not fit the work needed — no more work needs be assigned to the contractor, which effectively terminates the contract. There is no work in the product backlog, and the contractor cannot bill their time to the government. There is no need to terminate for convenience or cause.

To be clear, T&M does require more active management than the FFP "set it and forget it" contracts. The government must know what is going on, and review and approve invoices appropriately. Luckily, with agile development, this should already be happening. Read more about this shift in our recommendation on post-award management.

Find more information in our sample T&M determination and findings, which agencies can modify for their needs.

Labor hour contract

Why not a labor hour contract? Our experience reveals it's often necessary to buy additional materials for a software development project, such as cloud instances, software licenses, or SaaS fees. In those situations, it's better to pay the negligible costs for the contractor's materials; it will save a lot of headaches.

Firm-fixed-price contract

Don't procure agile under a firm-fixed-price contract. We understand that most agencies discourage the use of T&M contracts in favor of FFP. However, Contracting Officers can use time and materials contracts when it is not possible to estimate accurately the extent or duration of the work, or to anticipate costs with any reasonable degree of confidence. This is the case with agile software development.

Agencies can FFP agile. We just found it leads to a lot of headaches, fighting, and adverse outcomes. It is likely to go very poorly.

Firm-fixed-price contract types are perhaps the most commonly used methods to manage contractor performance. Government prefers the FFP contract type for a number of perceived benefits, and legislators and overseers encourage its use.

In this contract type, the contractor is rewarded when it delivers the exact list of requested features. An exact list of requested features becomes necessary up front, because using FFP contract types assumes that "performance uncertainties can be identified and reasonable estimates of their cost impact can be made, and the contractor is willing to accept a firm-fixed-price representing assumption of the risks involved."

When developing software using agile development, the best requirements for a product cannot be known accurately before development, so FFP is not appropriate.

That being said, we've seen several programs try to FFP agile, sometimes in elaborate and complex ways, and none of them have worked well. The most popular method is to FFP a sprint. Another commonly used method is FFP by story points.

Both scenarios result in unnecessary gamesmanship and fighting between the contractor and government. The contractor is incentivized to overbill and underdeliver. There are no canonical, objective ways to measure the number of user stories included in a given sprint or the size of a story point. So, when the government says "we think this story is size X" and the contractor says "we think this story is size Y" the resulting debate is difficult to resolve. These methods don't help to gauge contractor performance in any way. Agile quality or outcomes cannot be measured by the number of sprints or user stories completed.

An astounding number of projects are beleaguered by a desire to avoid using the most logical contract type (T&M), due to its reputation for cost overruns in projects that were not adequately monitored.

With FFP contracts, agencies don't have the flexibility to change the contractor software development team size over time based on product needs. If the contractor is not performing, agencies can't simply stop assigning them work and move onto another contractor — they must terminate for convenience or cause.

The only repeatable, standard, objective, scalable, universal way to measure work during a sprint is total effort-hours, which is not appropriate for firm-fixed-pricing. This is the purpose of time-and-materials or labor hour contracts.

Evaluate contractor proposals based on industry best practices

By Waldo Jaquith, Randy Hart, Mark Hopson, Vicki McFadden, Kelsey Foley, Miatta Myers, and Stephanie Rivera

Challenge: Traditional evaluation methods of custom technology practices are not based on industry standards.

Executive summary

• Review the strengths, weaknesses, and risks of contractors' proposals and then invite the most highly rated for a verbal interview.

Recommendation

Scenario: An agency has received contractors' proposals and their evaluation team is about to embark on the important mission of determining which contractor is best suited to meet the agency's needs.

This overview supports the evaluation team and helps them identify both indicators that show a strong proposal and red flags that indicate a weak proposal. This list is not exhaustive — every procurement is different.

Evaluation process

At 18F, we typically conduct targeted market research to identify a strong candidate pool of companies. As a best practice, we use the procedures allowed under FAR Part 8.405-2(c)(3)(iii)(B) to identify around ten potential companies interested in bidding before we issue any solicitation on GSA Schedule 70; this ensures that we solicit qualified and interested bidders. This method balances good competition and a reasonable number of proposals for the evaluation panel to assess.

Note: Your agency can only use this process if your agency intends to use GSA Schedules as their acquisition strategy. If your agency doesn't intend to use Schedules, your agency needs to follow the procedures required under the FAR that pertain to their acquisition strategy.

Don't use a point system to score proposals. Instead, each evaluation team member should review each proposal and list its advantages and disadvantages. Then, the whole evaluation team can discuss each proposal's advantages and disadvantages, and determine the strongest proposals.

A narrative review provides a detailed, defensible justification for the government's contractor selection in a way that numeric or color scoring schemes don't capture. It also allows the government to give feedback to the contractors that did not receive the award: the agency simply discusses the proposal's documented advantages and disadvantages. We encourage evaluation teams to use whatever materials they have available to make their decisions — websites, news articles, samples of prior work — and not only the proposals.

Invite the companies with the most highly rated proposals for an interview. This is when the agency can verify that the contractor can perform to the level proposed in their bid.

We ask that all named key personnel participate in these interviews. Each interview includes a timed, unstructured question-and-answer session, where the contractors answer questions about their proposal's technical aspects. Although we tailor each interview to the proposal, we often draw from an interview question bank that helps us plan interviews. This process allows the agency to better understand each contractor's proposed technical approach and to observe key personnel's interactions and working style.

Contractors will not be allowed to make presentations, ask questions, or change their submission in any way.^[9]

Determining the most highly rated proposals

When contracting for agile development services, we typically recommend that agencies use four evaluation factors: technical approach, staffing approach, similar experience, and price.

The three technical evaluation factors, when combined, are significantly more important than the price. Evaluate the interviews as part of the technical approach, rather than as their own evaluation factor. For similar experience, ask the contractor to submit code repositories that are similar in size, scope, and complexity to the work that the agency is undertaking.

Below is what we suggest agencies consider when they're reviewing proposals using these evaluation criteria.

Technical approach

What to look for:

• Competency. They should propose using the right tools in the right way and be able to defend their choices.

• A lack of novelty. The best approaches will recommend time-tested software and infrastructure, employing design patterns that are known to work.

• A lack of certainty. Maybe the contractor's idea is a good one, maybe it's a bad one — they can't really know yet, and they need to be aware that they can't know. A contractor will show they understand this by highlighting weak points or areas of uncertainty in their technical approach.

• A vision. The contractor needs to see the intended outcomes in a way that can act as a catalyst to the agency's vision.

• Understands program goals. The vendor should have a clear grasp of what the agency is doing. There should be no serious misunderstandings of information that was described clearly in the RFQ.

• Experience developing open source software.

• Collaboration and communication. The contractor expects the agency's product owner to be a valuable, active team member. They also expect to communicate proactively about risks and roadblocks, so they can work as effectively as possible.

• Regular and ongoing user research to understand users' goals and needs, and what to build that supports them. They will combine user research with usability testing to ensure that users can achieve their broader goals in using the software, and that it addresses their needs along the way. They plan to conduct user research, and test everything from rough prototypes to more polished software with actual users, throughout the entire design and development process.

• They follow a user-centered design process. They can explain how they make design decisions in relation to broader user goals and specific needs learned through their research.

• The contractor focuses on automation, reliability, testability, infrastructure as code, and other core DevOps principles. The proposal refers to modern automation and deployment tooling like Jenkins, Puppet, Chef, Travis CI, CircleCI, Kubernetes, Terraform, AWS, and Heroku.

Red flags:

• Don't seem to understand program goals. They seriously misunderstand information that was described clearly in the solicitation.

• Misidentifying the name of technologies in such a way that shows a lack of experience communicating about them (e.g. "we'll index records with an Elasticsearch," instead of "with Elasticsearch," or "we recommend using JAVA," instead of "Java").

• Excessive complexity.

• They shirk page-limit rules (tiny fonts, reduced leading, etc.) because they believe their technical approach to be so brilliant that it can't possibly fit within the prescribed limit.

• Basing their solution on a fundamental misunderstanding of the agency's needs that they should have understood.

• Proposing the use of arcane platforms and technologies, especially when those arcane platforms and technologies are contractor specialties.

• They never once mention the software's accessibility, or do not identify how they will evaluate whether their software meets accessibility standards.

• They don't consider, explicitly or implicitly, that user research will ultimately determine the approach, which in turn will dictate the technical approach.

• Uses terminology like "requirements will be collected from the business owner"; user needs should be uncovered through research and listed as user stories in the product backlog.

• They're proposing to outsource what should be core competencies, e.g., DevOps or Javascript.

• They propose a process that includes working for long stretches of time without interacting with the agency and/or users.

• They describe the goal of research as being to "test the app with users," "find problems," or ask users what they "like," "want," or "might do" (shows that they draw conclusions based on what users say instead of observing and learning from users what they do).

• They use the term "user testing" instead of "usability testing" (not testing the user, testing the system's functionality).

• They propose relying on focus groups, instead of structured, one-on-one research interviews or usability testing sessions.

• They prioritize aesthetics over usability and usefulness, and cannot explain why they made design decisions.

• Don't mention anything about secure code practices.

• Don't demonstrate that testing is important.

• They propose long-term staff augmentation.

Staffing approach

What to look for:

• A small number of team members, each providing a clear value. Everyone proposed has a purpose.

• Familiar with and demonstrate use of modern software languages (e.g. Python, Ruby, PHP, C#, Javascript).

• Familiar with and demonstrate use of web-based application programming interfaces (APIs), especially REST and GraphQL.

• Experience using Git for software version control.

• The lead developer's skill mix and experience cover much of the work that the agency's project requires.

• If the developers have presences on social coding platforms (e.g., GitHub, GitLab, Bitbucket), how does their work look? What expertise is evident there? Do they have expertise that doesn't appear in their qualifications, but their work reveals?

• Staff qualifications support their claimed expertise. (For example, does the content strategist have any actual content strategy experience, or are they a project manager in sheep's clothing?)

• The lead user researcher's background indicates an understanding of how research can inform and shape strategy, design, and development; familiarity with a variety of research and testing methods; and experience deciding which method or methods to use based on the learning goals of the phase or needs of the project, and with recruiting users based on those goals and needs.

• The lead UX designer's background demonstrates strong craft skills and experience in generating concepts based in overall strategy, user research, and user-centered design best practices; and experience communicating those concepts visually via a variety of methods including but not limited to sketching, wireframing, prototypes, and more polished mockups to use in research and to guide development.

• Generally, the team is assigned to the project full-time and will not be splitting their time across other unrelated projects. There may be acceptable exceptions, such as for a scrum master or agile coach, but in general everyone should be fully staffed to the project. This is critical for developers, user researchers, designers, and all key personnel.

Red flags:

• Overstaffing the bid. A team that consists of people with far more experience than necessary or more people than necessary means that the contractor either doesn't understand this way of working or is trying to over-staff the engagement.

• Proposing positions that do not belong in iterative development — business analysts, enterprise architects, delivery managers, etc.

• Poorly designed websites for the company, proposed subcontractor, or proposed staff qualifications.

• Proposing antiquated software technologies that don't have an active developer community (e.g. Cold Fusion, ASP, FoxPro).

• Lack of experience with test automation, aka DevOps, aka test-driven development (TDD).

• Insufficiently qualified lead developer.

• No apparent experience with usability research.

• No apparent experience with visual design.

• The flashiest team member is proposed to spend a tiny amount of time on the project.

• Key skills don't appear in any qualifications, such as:

  • Platform migration

  • Agile development practices

  • Automated (unit/integration/end-to-end) testing

  • Continuous Integration and Continuous Deployment

  • DevOps

  • Refactoring to minimize technical debt

  • Application Protocol Interface (API) development and documentation

  • Open-source software development

  • Cloud deployment

  • Product management and strategy

  • Usability research, such as (but not limited to) contextual inquiry, stakeholder interviews, and usability testing

  • User experience design

  • Sketching, wireframing, and/or prototyping, and user-task flow development

  • Visual design

  • Content design and copywriting

  • Building and testing public-facing sites and tools

• No actual technical staff, but "access to a database of resumes."

• Proposed staff don't currently work for the contractor, and there is no letter of intent from the proposed staff.

• Proposed staff qualifications are copied from the Internet, in large part or, more rarely, in whole.

• Key staff are not proposed to be 100% full time to the project, or the project is staffed with a number of partial FTE personnel.

Similar experience

18F often asks contractors to submit code repositories that demonstrate producing quality code that is in similar size, scope, and complexity to what the agency needs. If you do not have someone on your evaluation team that is familiar with code repositories, you should find one to serve as a technical advisor. 18F can support agencies with this need with a signed interagency agreement.

Technical evaluations should look for:

• Proper use of Git, commit changes with personal accounts (not organizational), use of a branching / merging strategy, informative comments, evidence of code reviews, and use of a CI/CD pipeline.

• Code that conforms well to the solicitation's QASP.

• Git collaboration. Work was performed in a reasonable number of GitHub comments.

• Substantial projects. The projects were not created just to have something to point to for this RFQ.

• How they incorporate user feedback into their development process.

• Their tests are written well, and cover the supermajority of the code.

• Consistent, enforced code style.

Programmatic evaluations should look for:

• Work that is conceptually similar to the agency's needs.

• Work that was centered on user needs, as opposed to leading with solutionism.

• Work that was completed by a team of a size that's similar to the size of the team that they're proposing.

• Design artifacts that show continuous and ongoing usability testing that indicate a user-centered approach to iterative design and development.

Red flags:

• The cited projects bear little evidence that the contractor created them.

• The projects are trivial.

• There's a finished product, but no code, or vice versa.

• The projects do not include good design artifacts and research plans.

This list is not meant to be exhaustive, but it gives the evaluation team some tips to help empower them to decide which proposals to evaluate as most highly rated and to bring forward to an interview.

Next: Doing the work

Footnotes