Tag: survey experiment

How can societies accurately gauge the share of the workforce engaged in the shadow economy when direct questions inspire selective silence or evasion? This policy brief presents findings from a new cross-country survey experiment combining direct questions and an indirect “list experiment” method, conducted in Latvia, Italy, and Denmark. Results show that, contrary to expectations, indirect methods did not yield higher estimates of undeclared work compared to direct questions. The research reveals that in environments with high tax morale and a substantial shadow economy, both direct and indirect measurements can be biased. Sharing information about prevailing tax norms with respondents can improve survey consistency, informing future tax evasion measurement and anti-evasion policymaking.

Social Desirability Bias in Tax Evasion Surveys

Surveys on tax evasion often provide respondents with multiple response categories beyond simple “yes” or “no.” For example, the survey for Latvia (Kantar, 2024) found that 3% openly acknowledged undeclared income, but refusal (2%) and “hard to say” responses (4%) illustrate additional uncertainty and possible underreporting due to social desirability bias when respondents consciously avoid disclosing disapproved or illegal acts to maintain a positive self-image or avoid perceived censure. This bias is potentially serious in tax compliance research, where both tax morale and fear of consequences can shape reporting behavior.

Indirect questioning techniques, such as list experiments, aim to reduce social desirability bias by allowing individuals to conceal answers within a broader set of innocuous items (Blair et al., 2020). In a typical list experiment, respondents are randomly assigned to receive either a list of non-sensitive items or the same list with an additional sensitive item; by comparing the mean number of items endorsed across groups, researchers estimate the prevalence of the sensitive behavior without requiring explicit disclosure (Blair & Imai, 2012; Glynn, 2013).

Recent empirical work employing list experimental designs has significantly advanced the understanding of tax evasion dynamics across diverse fiscal and cultural contexts. Fergusson, Molina, and Riaño (2019) analyzed VAT evasion among Colombian consumers and found minimal social desirability bias, with list experiments and direct self-reports yielding similar evasion rates (~20%). They attributed this to the normalization of evasion in high-informality regions, where descriptive norms (perceived prevalence of evasion) outweighed injunctive norms, reducing stigma.

This contrasts with Genest-Grégoire et al. (2022), who detected significant bias in Canadian income tax self-reports: list experiments revealed 13.5% income tax evasion (compared to 5.6% in direct questions) and 28.5% consumption tax evasion (compared to 26.2% in direct questions). The study identified stronger stigma around income tax evasion, particularly due to institutional withholding mechanisms that make income tax evasion more difficult compared to consumption taxes. Authors posit that divergent motivational mechanisms underlie these evasion types: income tax noncompliance triggers stronger moral condemnation due to its association with deliberate fraud, while consumption tax evasion is often rationalized as a “victimless” violation of complex regulations.

Hence, high tax morale, while generally associated with greater compliance, also leads individuals to conceal or misrepresent socially undesirable actions more rigorously, which amplifies social desirability bias in survey responses. This effect is particularly pronounced in environments where tax evasion is strongly stigmatized, as respondents may feel increased pressure to align their self-reports with prevailing moral standards, even if those reports do not reflect their true behavior. Conversely, in contexts where evasion is normalized or perceived as widespread, the stigma associated with noncompliance decreases, potentially making individuals less reluctant to report such behavior.  Nevertheless, both direct and indirect measurement techniques may still fail to accurately capture the true prevalence. This is because reduced stigma alone does not eliminate other sources of bias, including cognitive complexity, survey design imperfections, and strategic respondent behavior, such as misinterpreting instructions or using responses to send political or social signals beyond truthful self-disclosure.

Recognizing these persistent methodological challenges, this policy brief presents evidence from a study employing both direct and indirect questions on tax evasion across three European countries with varying levels of tax morale and shadow economy prevalence. By analyzing how social contexts influence reporting behaviors, the brief provides insights into the effectiveness and limitations of these survey approaches in different normative environments.

Approach

The research used a nationally representative sample of 6,915 respondents from Latvia, Italy, and Denmark, utilizing Norstat online panels in the respective countries. It was administered as an online Computer Assisted Web Interview (CAWI) in May 2024. Respondents in the study were randomly assigned to one of two list experiment conditions: half received a 5-item list including the sensitive tax evasion item, while the other half received a 4-item list without the sensitive item (see Figure 1). Importantly, all respondents—regardless of their list group assignment—were asked a direct question about undeclared income at the end of the survey. This design allows comparison between indirect and direct measures within the same individuals, clarifying reporting patterns and social desirability effects.

Figure 1. Indirect question for the control group of the list experimental study

Notes: The 5-item list for the treatment group included additional activity “Received all or part of the income without paying taxes (received money “off the books”)” and asked to indicate max 5 items. The activities were listed in random order for each respondent.

All participants also completed a placebo list experiment, in which both lists – i.e., containing 4 or 5 items – consisted entirely of non-sensitive behaviors (see Figure 2). Correspondingly, everyone was also asked a direct question about the non-sensitive behavior (“Bought a house or apartment (including on credit)”), thereby mimicking the structure of the tax evasion list experiment. This design allowed controlling for possible cognitive errors in filling out a complicated survey task, such as a list survey question, that are unrelated to social desirability bias.

In addition, half of all respondents were primed to information with actual data on how many citizens in their country consider tax evasion unacceptable, sourced from a recent representative survey that was carried out in January 2024. In this pre-survey, just 39% (i.e., minority) found tax evasion wholly unacceptable in Latvia; 59% in Italy, and 53% in Denmark (i.e., majority). The goal of this priming was to test whether informing respondents about local norms affected reporting patterns.

Figure 2. Placebo list of the study

Notes: The 5-item list for the treatment group included additional activity “Bought a house or apartment (including on credit)” and asked to indicate max 5 items. The activities were listed in random order for each respondent.

Key Findings

Results show that indirect list experiment estimates of undeclared work (4.1% overall) did not significantly differ from direct question estimates (7.2%). Hence, respondents did not find the topic sensitive enough to avoid honest answers in either format.

Priming respondents with information about the unacceptability of tax evasion in their country had no statistically significant effect on the direct measure of admitted undeclared income, nor on aggregate estimates from the indirect list experiment, indicating that willingness to disclose undeclared work remained unchanged regardless of norm priming.

Figure 3. Estimates of tax evasion from the list experiment and direct question

Source: Author’s estimate from the survey results.

However, country-level analysis revealed an anomaly in Italy: the list experiment produced an implausible negative estimate, driven by some respondents who marked zero items in the treatment list but later admitted to undeclared work in direct questioning. While this inconsistent response pattern was most prominent in Italy, the country with the highest tax morale (as based on pre-survey), and the largest shadow economy across the three countries (Medina and Schneider, 2018), it has also been recorded in the other two countries. Specifically, the pattern was observed among 11% of respondents who admitted to tax evasion in the direct question in Italy, compared to 5–7% in Latvia and Denmark.

Considering the complexity and unusual formulation of the question for the list experiment, one might attribute this pattern to a respondent’s confusion or cognitive error. However, this explanation is unlikely because of the responses to the placebo list experiment, where all list items and a direct question are non-sensitive. There, the specific response pattern – respondents reporting zero items on the list question while simultaneously admitting to the direct question – is observed substantially less frequently, indicating a low baseline error rate for misunderstanding or inconsistent reporting on non-sensitive items.

The comparison between the sensitive and placebo list experiment results indicates that the anomalous pattern observed in the tax evasion list experiment is unlikely to be due to confusion with the survey format, but rather represents a deliberate, context-specific form of strategic misreporting. One possible reason for this pattern might be that some Italians who admit to tax evasion in the direct question may believe that inflating shadow economy estimates will spur stronger policy reactions or public debate. In this way, their answers to the survey may represent strategic “signal sending.”

Priming respondents with accurate information about societal norms regarding the unacceptability of tax evasion – an approach referred to as vignette priming – consistently reduced the occurrence of this contradictory response pattern. Fewer respondents reported zero items in the list experiment while admitting to evasion in direct questioning, a change observed universally across the three countries.

Two main interpretations of the effects of such vignette priming can be suggested. The first interpretation, related to the strategic motive discussed above, suggests that vignette priming helps align respondents’ understanding of prevailing social norms on tax evasion. This improved awareness discourages deliberate misreporting, thus improving the overall validity and reliability of the survey’s methodology, even if it does not increase overall admissions of tax evasion itself. An alternative explanation is that vignette priming helps respondents better recognize and correctly count items in the list experiment, thereby improving response accuracy and alignment across question formats.

In other words, norm priming fosters more consistent survey responses, whether by reducing the temptation to manipulate results or by increasing recognition and attention among respondents.

Conclusion

Efforts to estimate tax evasion through surveys must strike a balance between the limitations of direct self-reports and the incomplete protection against bias afforded by indirect methods. This study finds that, in the surveyed countries, list experiments do not yield higher or more accurate prevalence estimates than direct questioning. However, particularly in high-morale environments with substantial shadow economies, some respondents may strategically manipulate survey results in hopes of prompting political action.

Norm priming through vignettes enhances experimental integrity and reduces strategic errors, underscoring the importance of accounting for social context in survey designs. For tax policy makers, measurement should always be validated with error diagnostics and social context cues, and survey formats should be adapted for cross-country comparability and public trust.

Acknowledgements

The study is financed by the European Commission’s Marie Sklodowska-Curie Individual Fellowship Action (Grant agreement ID: 101109679).

References

• Blair, G., Coppock, A., & Moor, M. (2020). When to Worry about Sensitivity Bias: A Social Reference Theory and Evidence from 30 Years of List Experiments. American Political Science Review, 114(4), 1297–1315.
• Blair, G., & Imai, K. (2012). Statistical Analysis of List Experiments. Political Analysis, 20(1), 47–77.
• Fergusson, L., Molina, C., & Riaño, J. F. (2019). Consumers as VAT ‘Evaders’: Incidence, Social Bias, and Correlates in Colombia. Economía, 19(2), 21–67.
• Genest-Grégoire, A., Godbout, L., & Guay, J.-H. (2022). Lists: A Novel Experimental Method to Measure Tax Evasion. National Tax Journal, 75(3), 517–537.
• Glynn, A. N. (2013). What Can We Learn with Statistical Truth Serum? Public Opinion Quarterly, 77(S1), 159–172.
• Kantar. (2024). Kvantitatīva Latvijas iedzīvotāju aptauja par nodokļu morāli. Valsts Kanceleja.
• Medina, L., & Schneider, F.  (2018). “Shadow Economies Around the World: What Did We Learn Over the Last 20 Years?“, IMF Working Papers 2018, 017 (2018),

Disclaimer: Opinions expressed in policy briefs and other publications are those of the authors; they do not necessarily reflect those of the FREE Network and its research institutes.