Author: ln

How much weight should be given to past performance indicators when selecting contractors? Does a large weight assigned to suppliers’ previous performance deter entry by new, innovative suppliers that have no track records for the very reason that they are new? If yes, how should we take this into account when designing procurements for firms or governments? This note describes recent research that sheds light on these questions crucial for every government and organization.

Introduction

How should past performance be accounted for when selecting (public or private) contractors? On one hand, giving large weight to suppliers’ previous performance in assigning contracts may improve incentives in procurement; on the other hand, it may deter entry by new, innovative suppliers for the very reason that they are new. Are there ways to structure procurement rules and procedures to minimize or eliminate these costs? How can well performing suppliers be rewarded, but not at the expense of losing the most innovative start-ups, that could pose important positive externalities on the buyer and on society overall? These questions are important ones and every procurement manager, in the private and public sector, should know how to answer, or at least how to think about, them. Unfortunately, if one looks at the leading management and operations textbooks, or at public procurement textbooks, it is hard to find a line that could help in making these crucial decisions. The only procurement book that at least mentions these crucial questions, to our knowledge, is the Handbook of Procurement (2006; see Ch. 18, by Dellarocas et al.). However, even that handbook falls short in providing evidence-based or – more generally – research-based guidance for these questions. This is the case because there is practically no research dealing with these everyday problems. One recent exception is an experimental study recently undertaken by Butler, Carbone, Conzo and Spagnolo (2013) that will be discussed in depth in the reminder of this brief.

Public Policy Relevance  

Before getting into the results of this recent study, let me provide some background information that will give an idea of the relevance of these questions for public policy.

Public procurement currently accounts for between 15% and 20% of the GDP in developed countries (see http://cordis.europa.eu/fp7/ict/pcp/key_en.html).  In 2011, the total public procurement market in the EU – i.e. the purchases of goods, services and public works by governments and public utilities – reached a size of approximately €2,500 billion, corresponding to 19 percent of GDP (see e.g. http://ec.europa.eu/internal_market/publicprocurement/docs/modernising_rules/public-procurement-indicators-2011_en.pdf). As Table 1 below shows, despite year-to-year fluctuations, there has been an overall increase in procurement expenditures relative to 2007 levels, both absolutely and proportionately. The increasing shift in focus in EU innovation policies, from “push-based” mechanisms like R&D subsidies/tax breaks towards demand-led “pull” mechanisms, like Pre-Commercial Procurement, is likely to further increase the volume of this market.

The enormous size of this market notwithstanding, we know relatively little about whether, when, and how buyers should use reputational indicators based on past performance in selecting among sellers, and whether the use of such indicators necessarily reduces the ability of new sellers—i.e., sellers with no history of past performance—to enter the market.

It is well known that reputational mechanisms that reward past performance are important governance tools that complement (and sometimes substitute for) contracts in private transactions (Calzolari and Spagnolo 2009). Private buyers, however, are typically only concerned about the price and quality of the good they buy. Regulators in charge of public procurement, instead, are usually also concerned that the public procurement process is transparent and open for obvious accountability reasons. The need to prevent favoritism and corruption has led lawmakers around the world to ensure that open and transparent auctions where bidders are treated equally—even when in some crucial dimensions they have very different track records—are used whenever possible.

The trend in the US

The costs of limiting discretion to ensure public buyers’ accountability – such as the possibly large cost of not allowing reputational forces to complement incomplete procurement contracts – were stressed by Kelman (1990), who pushed for a deep reform of the US procurement system when he was the head of public procurement during the Clinton administration. The reform was targeted at reducing the rigidity of procurement rules in the Federal Acquisition Regulations and allowing public buyers to adopt more flexible purchasing practices common in the private sector, including giving more weight to suppliers’ past performance. Since the Federal Acquisitions Streamlining Act of 1994, US federal departments and agencies are expected to record past contractors’ performance evaluations and share them through common platforms for use in future contractor selection.

However, the US Senate recently expressed the apparently widespread concern that past performance-based selection criteria could hinder new and small businesses’ ability to enter and compete effectively, leading to an intriguing, but inconclusive report by the General Accountability Office.

This is not to say that US regulators were not concerned with the ability of small and medium enterprises (SME) – sometimes the most innovative part of the economy – to enter public procurement markets. In the US, this long-held concern led to large programs like the Small Business Act, with its rules limiting the bundling of public demand in very large procurements and establishing the Small Business Agency, and the ‘set aside’ (procurement only open to SMEs) common in many types of procurement auctions. However, the worry that past-performance based selection may contribute to the exclusion of novel and smaller firms only arose in the last couple of years.

The (opposite) trend in the EU

The European Union has instead been moving precisely in the opposite direction. An important concern driving procurement regulation in Europe since the Treaty of Rome has been helping the process of common market integration by increasing cross-border procurement, i.e., the amount of goods and services each EU Member State buys from contractors based in other states. The EU Procurement Directives that coordinate public procurement regulation in the various European states have been limiting the use of past-performance information in the process of selecting among offers—a feature that came under broad attack during the 2011 consultation for the revision of the EU Directives (see Replies to the Consultation on the 2011 EU Green Book on Public Procurement regulation). The EU regulators appear to have been always convinced that using reputational indicators as a criteria for selecting contractors leads to manipulations in favour of local incumbents, at the expense of cross-border procurement and market integration.  (see e.g., http://ec.europa.eu/internal_market/publicprocurement/modernising_rules/consultations/index_en.htm).

Only very recently – now that the US is moving towards reconsidering the effects, and possibly limiting the use of past-performance indicators – has the EU started to move (again in the opposite direction to the US) towards leaving more space to these indicators, perhaps as a reaction to the comments they received in recent consultations.

Finally, to have an idea of the lack of research-based knowledge guiding policy in this field, note that in some cases EU regulation already acknowledges the crucial importance of past-performance based reputation for some types of procurement. For example, the European Research Council (ERC) provides funding to top researchers in Europe, who are selected through peer review, and the track record of the researchers is usually the main awarding criterion. ERC funding is distributed almost exclusively based on reputation criteria in order to support the best and the brightest. Other European instruments for the procurement of research, such as the FET-OPEN program, are based on a strictly enforced, completely anonymous evaluation instead, without obvious reasons justifying this opposite approach. On the dedicated homepage of these programs it states: “The anonymity policy applied to short proposals has changed and is strictly applied. The part B of a short STREP proposal may not include the name of any organization involved in the consortium nor any other information that could identify an applicant. Furthermore, strictly no bibliographic references are permitted.”

Reputation and Entry in Procurement

In a recent research paper (Butler, Carbone, Conzo and Spagnolo, 2013) we have been trying to fill at least part of this knowledge gap and offer some initial evidence-based guidelines for future policy.

The study

We build a simple model of repeated procurement with limited enforcement and potential entry and implement it in the laboratory. We focus on reputation as an incentive system to limit moral hazard in the quality dimension as well as on the effect of reputation on selection through entry. We assume that some costly-to-produce quality dimension of supply, although observable to the parties, is too costly to verify for a court to be governed through explicit contracting and is therefore left to reputational governance. We make the additional assumption that there is a potential entrant firm that is more efficient than all incumbent firms. In this context, we study how quality, price, entry and welfare change with the introduction of a simple and transparent reputational mechanism. This mechanism rewards an incumbent firm that chooses to provide (costly) high-quality production with a bid subsidy in the subsequent procurement auction, and may also award a bid subsidy (of varying size) to an entrant with no history of production.

Note that in the case of public procurement and of firms’ vendor rating systems, we are talking about reputational mechanisms based on public rules, known and accepted by suppliers. Formal mechanisms and rules give commitment power to the buyer and can be designed in many different ways. A common mistake is to assume that reputational mechanisms must be designed along the line of the eBay feedback system, in which new sellers start with “zero reputation”. However, a buyer with some commitment power concerning the rules for information aggregation and diffusion and for selecting suppliers may well award a positive rating to new entrants—e.g. the maximum possible rating, or the average rating in the market, putting entrants at less of a disadvantage—and ensure that this is taken into account by the scoring rule that selects the contractor, even if the contractor has never before interacted with the buyer. Indeed, private corporations often have vendor rating systems in which all suppliers start off with the same maximal reputational capital—a given number of points—and then lose points when performing poorly and are suspended for some time if their reputational capital falls below a certain low threshold. This type of vendor rating system creates an advantage for new suppliers, most likely stimulating rather than hindering entry, suggesting that it is possible to design a reputational mechanism in public procurement that simultaneously sustains quality and entry.

The results

First of all, the study shows that concerns about reputation-based selection hindering entry are justified: naively introducing a “standard” reputational mechanism in which only good past performance is rewarded with a bid subsidy in the following procurement auction increases quality provision, but it also significantly reduces entry.

In contrast to this first result, the study goes on to show that properly designed reputational mechanisms in which new entrants, with no history of past performance, are awarded a moderate or high reputation score—as is often done in the private sector—actually foster rather than hinder entry while, at the same time, delivering a substantial increase in high quality goods provision.

The third important result of this study is that the total cost to buyers (buyer’s transfer) does not increase when a reputational mechanism is introduced, even though (costly) quality provision increases. The introduction of bid subsidies for good past performance appears to benefit the buyer/taxpayer by increasing competition for incumbency, driving winning bids down sufficiently to offset the potential increase in procurement costs due to bid subsidies.

Conclusions

Considered together, the findings in Butler et al. (2013) suggest that there need not be a trade-off between reputation and entry in procurement, and that the debates both in the EU and the US are rather misplaced. The results suggest that the dual goals of providing incentives for quality provision and increasing entry and cross-border procurement – in the EU or elsewhere – are achievable through an appropriately designed reputational mechanism. Policy makers should therefore probably stop quarrelling about whether a generic past-performance based reputational mechanism should be introduced, and instead focus on how such a mechanism should be designed.

References

• Butler Jeff, Carbone Enrica, Conzo Pierluigi and Giancarlo Spagnolo. 2013. “Reputation and Enty in Procurement.” CEPR Discussion Paper No. 9651.
• Calzolari, Giacomo and Giancarlo Spagnolo. 2009. “Relational Contracts and Competitive Screening.”  CEPR Discussion paper No. 7434.
• European Commission. 2011. Green Paper on the modernisation of EU public procurement policy Towards a more efficient European Procurement Market. Available at http://ec.europa.eu/internal_market/consultations/index_en.htm
• European Commission. 2011. Green Paper on the modernisation of EU public procurement policy Towards a more efficient European Procurement Market. Synthesis of Replies.  Available at http://ec.europa.eu/internal_market/consultations/docs/2011/public_procurement/synthesis_document_en.pdf
• Government Accountability Office. 2011. (GAO-12-102R, Oct. 18, 2011). Prior Experience and Past Performance as Evaluation Criteria in the Award of Federal Construction Contracts. Available at  http://www.gao.gov/products/GAO-12-102R
• Kelman, Steven. 1990.  Procurement and Public Management: The Fear of Discretion and the Quality of Government Performance: American Enterprise Institute Press.
• Yukins, Christoher. 2008. “Are IDIQS Inefficient? Sharing Lessons with European Framework Contracting.” Public Contract Law Journal,  37(3): 545-568.