Location: United States
Reputation for Quality and Entry in Procurement: Is there a Trade-Off?
How much weight should be given to past performance indicators when selecting contractors? Does a large weight assigned to suppliers’ previous performance deter entry by new, innovative suppliers that have no track records for the very reason that they are new? If yes, how should we take this into account when designing procurements for firms or governments? This note describes recent research that sheds light on these questions crucial for every government and organization.
Introduction
How should past performance be accounted for when selecting (public or private) contractors? On one hand, giving large weight to suppliers’ previous performance in assigning contracts may improve incentives in procurement; on the other hand, it may deter entry by new, innovative suppliers for the very reason that they are new. Are there ways to structure procurement rules and procedures to minimize or eliminate these costs? How can well performing suppliers be rewarded, but not at the expense of losing the most innovative start-ups, that could pose important positive externalities on the buyer and on society overall? These questions are important ones and every procurement manager, in the private and public sector, should know how to answer, or at least how to think about, them. Unfortunately, if one looks at the leading management and operations textbooks, or at public procurement textbooks, it is hard to find a line that could help in making these crucial decisions. The only procurement book that at least mentions these crucial questions, to our knowledge, is the Handbook of Procurement (2006; see Ch. 18, by Dellarocas et al.). However, even that handbook falls short in providing evidence-based or – more generally – research-based guidance for these questions. This is the case because there is practically no research dealing with these everyday problems. One recent exception is an experimental study recently undertaken by Butler, Carbone, Conzo and Spagnolo (2013) that will be discussed in depth in the reminder of this brief.
Public Policy Relevance
Before getting into the results of this recent study, let me provide some background information that will give an idea of the relevance of these questions for public policy.
Public procurement currently accounts for between 15% and 20% of the GDP in developed countries (see http://cordis.europa.eu/fp7/ict/pcp/key_en.html). In 2011, the total public procurement market in the EU – i.e. the purchases of goods, services and public works by governments and public utilities – reached a size of approximately €2,500 billion, corresponding to 19 percent of GDP (see e.g. http://ec.europa.eu/internal_market/publicprocurement/docs/modernising_rules/public-procurement-indicators-2011_en.pdf). As Table 1 below shows, despite year-to-year fluctuations, there has been an overall increase in procurement expenditures relative to 2007 levels, both absolutely and proportionately. The increasing shift in focus in EU innovation policies, from “push-based” mechanisms like R&D subsidies/tax breaks towards demand-led “pull” mechanisms, like Pre-Commercial Procurement, is likely to further increase the volume of this market.
Table 1: Total EU procurement expenditure on works, goods and services In EUR billion|
2007 |
2008 |
2009 |
2010 |
2011 |
|
2,178.55 |
2,263.62 |
2,346.00 |
2,416.52 |
2,405.89 |
|
2007 |
2008 |
2009 |
2010 |
2011 |
|
17.6 |
18.1 |
20.0 |
19.7 |
19.0 |
The enormous size of this market notwithstanding, we know relatively little about whether, when, and how buyers should use reputational indicators based on past performance in selecting among sellers, and whether the use of such indicators necessarily reduces the ability of new sellers—i.e., sellers with no history of past performance—to enter the market.
It is well known that reputational mechanisms that reward past performance are important governance tools that complement (and sometimes substitute for) contracts in private transactions (Calzolari and Spagnolo 2009). Private buyers, however, are typically only concerned about the price and quality of the good they buy. Regulators in charge of public procurement, instead, are usually also concerned that the public procurement process is transparent and open for obvious accountability reasons. The need to prevent favoritism and corruption has led lawmakers around the world to ensure that open and transparent auctions where bidders are treated equally—even when in some crucial dimensions they have very different track records—are used whenever possible.
The trend in the US
The costs of limiting discretion to ensure public buyers’ accountability – such as the possibly large cost of not allowing reputational forces to complement incomplete procurement contracts – were stressed by Kelman (1990), who pushed for a deep reform of the US procurement system when he was the head of public procurement during the Clinton administration. The reform was targeted at reducing the rigidity of procurement rules in the Federal Acquisition Regulations and allowing public buyers to adopt more flexible purchasing practices common in the private sector, including giving more weight to suppliers’ past performance. Since the Federal Acquisitions Streamlining Act of 1994, US federal departments and agencies are expected to record past contractors’ performance evaluations and share them through common platforms for use in future contractor selection.
However, the US Senate recently expressed the apparently widespread concern that past performance-based selection criteria could hinder new and small businesses’ ability to enter and compete effectively, leading to an intriguing, but inconclusive report by the General Accountability Office.
This is not to say that US regulators were not concerned with the ability of small and medium enterprises (SME) – sometimes the most innovative part of the economy – to enter public procurement markets. In the US, this long-held concern led to large programs like the Small Business Act, with its rules limiting the bundling of public demand in very large procurements and establishing the Small Business Agency, and the ‘set aside’ (procurement only open to SMEs) common in many types of procurement auctions. However, the worry that past-performance based selection may contribute to the exclusion of novel and smaller firms only arose in the last couple of years.
The (opposite) trend in the EU
The European Union has instead been moving precisely in the opposite direction. An important concern driving procurement regulation in Europe since the Treaty of Rome has been helping the process of common market integration by increasing cross-border procurement, i.e., the amount of goods and services each EU Member State buys from contractors based in other states. The EU Procurement Directives that coordinate public procurement regulation in the various European states have been limiting the use of past-performance information in the process of selecting among offers—a feature that came under broad attack during the 2011 consultation for the revision of the EU Directives (see Replies to the Consultation on the 2011 EU Green Book on Public Procurement regulation). The EU regulators appear to have been always convinced that using reputational indicators as a criteria for selecting contractors leads to manipulations in favour of local incumbents, at the expense of cross-border procurement and market integration. (see e.g., http://ec.europa.eu/internal_market/publicprocurement/modernising_rules/consultations/index_en.htm).
Only very recently – now that the US is moving towards reconsidering the effects, and possibly limiting the use of past-performance indicators – has the EU started to move (again in the opposite direction to the US) towards leaving more space to these indicators, perhaps as a reaction to the comments they received in recent consultations.
Finally, to have an idea of the lack of research-based knowledge guiding policy in this field, note that in some cases EU regulation already acknowledges the crucial importance of past-performance based reputation for some types of procurement. For example, the European Research Council (ERC) provides funding to top researchers in Europe, who are selected through peer review, and the track record of the researchers is usually the main awarding criterion. ERC funding is distributed almost exclusively based on reputation criteria in order to support the best and the brightest. Other European instruments for the procurement of research, such as the FET-OPEN program, are based on a strictly enforced, completely anonymous evaluation instead, without obvious reasons justifying this opposite approach. On the dedicated homepage of these programs it states: “The anonymity policy applied to short proposals has changed and is strictly applied. The part B of a short STREP proposal may not include the name of any organization involved in the consortium nor any other information that could identify an applicant. Furthermore, strictly no bibliographic references are permitted.”
Reputation and Entry in Procurement
In a recent research paper (Butler, Carbone, Conzo and Spagnolo, 2013) we have been trying to fill at least part of this knowledge gap and offer some initial evidence-based guidelines for future policy.
The study
We build a simple model of repeated procurement with limited enforcement and potential entry and implement it in the laboratory. We focus on reputation as an incentive system to limit moral hazard in the quality dimension as well as on the effect of reputation on selection through entry. We assume that some costly-to-produce quality dimension of supply, although observable to the parties, is too costly to verify for a court to be governed through explicit contracting and is therefore left to reputational governance. We make the additional assumption that there is a potential entrant firm that is more efficient than all incumbent firms. In this context, we study how quality, price, entry and welfare change with the introduction of a simple and transparent reputational mechanism. This mechanism rewards an incumbent firm that chooses to provide (costly) high-quality production with a bid subsidy in the subsequent procurement auction, and may also award a bid subsidy (of varying size) to an entrant with no history of production.
Note that in the case of public procurement and of firms’ vendor rating systems, we are talking about reputational mechanisms based on public rules, known and accepted by suppliers. Formal mechanisms and rules give commitment power to the buyer and can be designed in many different ways. A common mistake is to assume that reputational mechanisms must be designed along the line of the eBay feedback system, in which new sellers start with “zero reputation”. However, a buyer with some commitment power concerning the rules for information aggregation and diffusion and for selecting suppliers may well award a positive rating to new entrants—e.g. the maximum possible rating, or the average rating in the market, putting entrants at less of a disadvantage—and ensure that this is taken into account by the scoring rule that selects the contractor, even if the contractor has never before interacted with the buyer. Indeed, private corporations often have vendor rating systems in which all suppliers start off with the same maximal reputational capital—a given number of points—and then lose points when performing poorly and are suspended for some time if their reputational capital falls below a certain low threshold. This type of vendor rating system creates an advantage for new suppliers, most likely stimulating rather than hindering entry, suggesting that it is possible to design a reputational mechanism in public procurement that simultaneously sustains quality and entry.
The results
First of all, the study shows that concerns about reputation-based selection hindering entry are justified: naively introducing a “standard” reputational mechanism in which only good past performance is rewarded with a bid subsidy in the following procurement auction increases quality provision, but it also significantly reduces entry.
In contrast to this first result, the study goes on to show that properly designed reputational mechanisms in which new entrants, with no history of past performance, are awarded a moderate or high reputation score—as is often done in the private sector—actually foster rather than hinder entry while, at the same time, delivering a substantial increase in high quality goods provision.
The third important result of this study is that the total cost to buyers (buyer’s transfer) does not increase when a reputational mechanism is introduced, even though (costly) quality provision increases. The introduction of bid subsidies for good past performance appears to benefit the buyer/taxpayer by increasing competition for incumbency, driving winning bids down sufficiently to offset the potential increase in procurement costs due to bid subsidies.
Conclusions
Considered together, the findings in Butler et al. (2013) suggest that there need not be a trade-off between reputation and entry in procurement, and that the debates both in the EU and the US are rather misplaced. The results suggest that the dual goals of providing incentives for quality provision and increasing entry and cross-border procurement – in the EU or elsewhere – are achievable through an appropriately designed reputational mechanism. Policy makers should therefore probably stop quarrelling about whether a generic past-performance based reputational mechanism should be introduced, and instead focus on how such a mechanism should be designed.
References
- Butler Jeff, Carbone Enrica, Conzo Pierluigi and Giancarlo Spagnolo. 2013. “Reputation and Enty in Procurement.” CEPR Discussion Paper No. 9651.
- Calzolari, Giacomo and Giancarlo Spagnolo. 2009. “Relational Contracts and Competitive Screening.” CEPR Discussion paper No. 7434.
- European Commission. 2011. Green Paper on the modernisation of EU public procurement policy Towards a more efficient European Procurement Market. Available at http://ec.europa.eu/internal_market/consultations/index_en.htm
- European Commission. 2011. Green Paper on the modernisation of EU public procurement policy Towards a more efficient European Procurement Market. Synthesis of Replies. Available at http://ec.europa.eu/internal_market/consultations/docs/2011/public_procurement/synthesis_document_en.pdf
- Government Accountability Office. 2011. (GAO-12-102R, Oct. 18, 2011). Prior Experience and Past Performance as Evaluation Criteria in the Award of Federal Construction Contracts. Available at http://www.gao.gov/products/GAO-12-102R
- Kelman, Steven. 1990. Procurement and Public Management: The Fear of Discretion and the Quality of Government Performance: American Enterprise Institute Press.
- Yukins, Christoher. 2008. “Are IDIQS Inefficient? Sharing Lessons with European Framework Contracting.” Public Contract Law Journal, 37(3): 545-568.
Putting the “I” Back in Team: The Rise of International Teams in Science
In this policy brief, I discuss the increasing prevalence of international teams in the production of scientific knowledge. I outline several potential factors that may explain these trends and discuss recent evidence from an original survey of coauthors on scientific papers regarding their collaboration behavior. Finally, as a notable example of increased international collaboration, I discuss the increase in scientific collaboration between Russia and the US after the end of the Cold War.
The Increase in Collaboration and Internationalization of Teams
Teams are becoming more prevalent in science. Both the share of papers produced by teams and the number of scientists working on scientific papers has increased in recent decades (Wuchty, Jones and Uzzi, 2007). Economic theory suggests that scientific research is becoming increasingly collaborative since the frontier of scientific knowledge has become more complex and specialized so that more researchers are needed to combine their expertise to make advances (Jones, 2009). Team members are also becoming more geographically dispersed: the share of papers resulting from international collaborations has increased, and within the US, scientists today are more likely to have coauthors located in a different city than before (Freeman, Ganguli and Murciano-Goroff, 2014).
These trends can be seen clearly in the graph below from the National Science Board’s Science and Engineering Indicators 2012. It shows the share of both world papers and US papers from 1990-2010 that are coauthored, coauthored with domestic coauthors only, and coauthored with at least one international coauthor. Collaboration in general and international collaboration have been increasing steadily since 1990 both in the world and in the US. However, for the US, the share of domestic-only collaborations has plateaued, while it is increasing in the rest of the world. In a recent Nature article, Adams (2013) shows that this trend similarly holds for other Western countries (United Kingdom, Germany, France, the Netherlands, Switzerland), while for emerging economies (China, India, South Korea, Brazil, Poland), domestic collaborations are also increasing.
Figure 1. World and US Trends in Scientific Collaboration, 1990-2010
Source: From National Science Board (2012)
Why has Science Become More International?
There are many potential reasons for the recent increases in international collaboration. An important factor has likely been the spread of the scientific workforce and R&D activities throughout the world (Freeman, 2010). The growing number of science and engineering PhDs in developing countries, some of whom are international students and post-docs returning to their home countries has expanded the supply of potential collaborators around the world (Scellato, Franzoni, and Stephan, 2012). Another factor is funding that has shifted scientific production towards international teams, as increased government and industry R&D spending in developing countries and grant policies by the European Union and other countries have supported international cooperation.
The lower cost of travel and communication in recent decades has also reduced the cost of collaborating with people in different locations. For example, Agrawal and Goldfarb (2008) show how the expansion of Bitnet, the precursor to the Internet, led to increased collaboration between institutions within the US. Finally, the location of scientific equipment and materials, such as the CERN Large Hadron Collider, telescopes, or climatological data available only in certain parts of the world, have increased international collaboration, and in some fields, has made international collaboration a necessity.
Survey Evidence on Scientific Collaborations
In a recent paper, my coauthors and I present the results of an original survey we conducted of scientists regarding collaboration (Freeman, Ganguli and Murciano-Goroff, 2014). In August 2012 we conducted a web-based survey of the corresponding authors of scientific papers with at least one US coauthor published in 2004, 2007, and 2010 in the fields of Nanotechnology, Biotechnology, and Particle Physics.
We customized each survey to ask the corresponding author about the collaboration and individual team members. The survey questions asked about how the team formed, how it communicated and interacted during the collaboration, the contribution of each coauthor, types of research funding, and the advantages and disadvantages of working with the team. We received 3,925 responses, so that our response rate was approximately 20%.
The survey also asked the respondent which country each coauthor was “primarily based in during the research and writing” of the article. This gives us a more accurate measure of whether teams are international than can be typically gleaned from publication data, which are based on author affiliations at the time of publication. Defining international teams from author affiliations alone can produce errors if affiliations change between the time the research was undertaken and the time of publication, or because some people have affiliations from more than one country.
Our analysis of the survey data uses the respondents’ information to define US collocated, US non-collocated and international teams. One of our key results is that face-to-face meetings continue to play an indispensible role in collaborations: most collaborators first met while working in the same institution. Teams also reported that while carrying out the research, they communicated often through face-to-face meetings, even with coauthors from distant locations.
Figure 2 below displays how the corresponding author responded about how they first met their team members. It shows that former colleagues play a very important role in the formation of international teams, followed by former students, conferences and institution visits, which equally contribute. The graph also shows the similarity between international teams and US non-collocated teams in how coauthors met. For other survey questions, our analysis also shows similarities between international teams and US non-collocated teams, suggesting that the salient issues are more about geography in general rather than necessarily about national borders.
Figure 2. How Coauthors First Met
Source: From Freeman, Ganguli and Murciano-Goroff (2014)
Another key finding from our survey is that the main reason for most collaborations, whether domestic or international, is to combine the specialized knowledge and skills of coauthors. We also asked the corresponding authors their views of the advantages and challenges of their collaboration. The most often cited advantage for all types of collaborations was “Complementing our knowledge, expertise and capabilities” and “learning from each other”. For the challenges, US non-collocated and international teams tended to agree more that there was “Insufficient time for communication”, “Problems coordinating with team members’ schedules”, and “Insufficient time to use a critical instrument, facility or infrastructure”, but international teams did not report these problems more often than US non-collocated teams. Where international teams differed is that these teams were the most likely to agree that their “research reached a wider audience”.
International Collaboration After the End of the USSR
A small but significant part of the increase in international collaboration since the 1990s can be attributed to the end of the Cold War. In “Russian-American Scientific Collaboration” (Ganguli, 2012), I examine trends in international collaboration by Russian and US scientists since the end of the USSR. Given the nature of the Cold War and restrictions on travel and communication with the West, I show that there was a dramatic increase in the number of publications with at least one Russian and a US coauthor from 1985 to 2005.
In addition to the lifting of travel and communication restrictions, there are several factors that contributed to the surge in collaborations between American and Russian scientists after the end of the USSR. First, at the level of the Russian government, there was a switch to a more open and collaborative approach to science. Part of this effort included establishing international centers for research in Russia aimed at integrating Russia into the global science community. Another important factor facilitating collaborations with Western researchers were foreign grant programs. The large increase in the emigration of Russian scientists in the 1990s to the West also contributed to international collaboration. After emigrating, many Russian scientists maintained close links to their colleagues in Russia, and coauthored papers with their former colleagues, which are counted as internationally coauthored publications.
While many of these factors have aided international cooperation after the end of the USSR, there have also been significant challenges that made cooperation difficult. Some of these challenges in the early 1990s included the political instability, organizational turnover making long-term funding agreements difficult to implement, difficulty transferring funds due to the underdeveloped banking system, high taxation and customs duties, lack of effective intellectual property rights, poor infrastructure, lack of a shared language (both linguistic and cultural), and external regulations (see further discussion in OECD, 1994). However, many of these challenges have now been overcome, leading to the continued increase in international collaboration between Russian and US scientists.
My analysis in Ganguli (2012) shows that the increase in Russian-American collaboration was more pronounced in some fields of science versus others, particularly in Physics. Figure 3 shows that the bulk of the articles published with Russian and American coauthors were Physics articles, with a sharp increase occurring immediately after 1991.
Figure 3. Russia-United States Publications By Field, 1985-2005
Source: From Ganguli (2012)
While some of the differences across the fields can be attributed to the number of scientists active in these fields, there are also other potential contributing factors. For example, it may be that there was greater emigration of scientists from certain fields abroad, and links between emigrants and those who remained in Russia persisted. Graham and Dezhina (2008: 24) suggest that over 50 percent of emigrants were physicists and mathematicians. Another reason may be that international collaboration was more important in some fields due to the knowledge or resources needed to conduct research during the economic crisis of the 1990s. As Wagner Brahmakulam, Peterson, Staheli, and Wong (2002) point out, physics research received significant amounts of US government funding for international collaboration, partly because expensive equipment that is needed and through collaboration, countries could share costs. Also, physicists from many countries often meet and work together at international research centers like CERN. Moreover, in some fields, the US and Russian governments shared priorities in funding international cooperation, like biomedical and health sciences, energy, physics, while there were gaps in some areas where Russia devoted resources and the US did not, like chemistry (Wagner et al. 2002: 24). Graham and Dezhina (2008: 141) also discuss how Western colleagues benefited from working with Russians especially in fields like zoology, botany and the earth sciences, since the Russian colleagues provided access to data from unique regions not available previously.
Support for International Teams?
This policy brief has discussed some reasons for the increase in international scientific collaboration and related empirical evidence, including insights from collaboration after the end of the USSR. The growth in collaboration and the geographic dispersion of teams is likely to continue; the frontier of scientific knowledge will become more complex and specialized, so that an even greater numbers of researchers will be needed to combine their expertise, and they are likely to be spread across increasingly distant locations.
These trends raise many complex issues for policymakers. For some countries, international collaboration may be the only way to sustain the science sector as the frontier of knowledge becomes more complex and resource-intensive. For some, international collaborations may increase the emigration of home-grown talent to wealthier countries. To what extent international collaboration should be supported, and how, will be important policy questions going forward. Typically, funding for international projects has been the main policy lever, and the Russian experience suggests that grant programs did play a critical role in that case. As our survey evidence in Freeman, Ganguli and Murciano-Goroff (2014) suggests, face-to-face meetings are especially important in forming and sustaining international collaborations. Thus, funding mechanisms that include provisions for research stays and face-to-face meetings may be the most effective means for fostering international collaborations.
▪
References
- Adams, J. (2013). “Collaborations: The Fourth Age of Research.” Nature, 497(7451), 557-560.
- Agrawal A, Goldfarb A (2008). “Restructuring Research: Communication Costs and the
- Democratization of University Innovation,” American Economic Review, 98(4):1578-1590.
- Freeman, Richard B. (2010). “Globalization of Scientific And Engineering Talent: International Mobility of Students, Workers, and Ideas and The World Economy.” Economics Of Innovation And New Technology, Volume 19, issue 5, 201 pp. 393-406.
- Freeman, Richard B., Ina Ganguli and Raviv Murciano-Goroff (2014). “Why and Wherefore of Increased Scientific Collaboration,” NBER Working Paper No. 19819, Issued in January 2014.
- Ganguli, Ina (2012). “Russian-American Scientific Collaboration” in Y.P. Tretyakov (ed), Russian-Аmerican Links: Leaps Forward and Backward in Academic Cooperation. St. Petersburg, Russia: Nestor-Historia, pp. 120-135.
- Graham, Loren and Irina Dezhina (2008). Science in the New Russia: Crisis, Aid, Reform. Bloomington and Indianapolis: Indiana University Press, 2008.
- Jones, Ben (2009). “The Burden of Knowledge and the ‘Death of the Renaissance Man’: Is
- Innovation Getting Harder?” Review of Economic Studies, 76:283-317.
- National Science Board (2012). Science and Engineering Indicators 2012. Arlington VA: National Science Foundation (NSB 12-01).
- National Science Board (2006). Science and Engineering Indicators 2006. Arlington VA: National Science Foundation (NSB 06-01).
- OECD (1994). Science, Technology, and Innovation Policies. Federation of Russia. Paris: Organisation for Economic Co-operation and Development, 1994.
- Scellato, G., Franzoni, C., & Stephan, P. (2012). “Mobile Scientists and International Networks,” NBER Working Paper 18613.
- Wagner, Caroline, Irene Brahmakulam, D.J. Peterson, Linda Staheli, and Anny
- Wong (2002). U.S. Government Funding for Science and Technology Cooperation with Russia.
- Santa Monica, CA: RAND Corporation, 2002.
- Wuchty, S., Jones, B. F., & Uzzi, B. (2007). “The Increasing Dominance Of Teams In Production Of Knowledge.” Science, 316(5827), 1036-1039.