Read the analysis paper n°109 : Who uses academic assessments of public policies?
Read the analysis paper n°110 : What assessments are conducted before and after a law is passed?
Part 1 – Who uses academic assessments of public policies?
In terms of academic assessments of the impact of public policies, France is catching up with more advanced countries [1]. But are these assessments used more in public policy debates as a result? To explore this question, here we consider exclusively quantitative and causal assessments published in peer-reviewed academic journals and addressing the evaluative criterion of impact on an ex-post basis. Based on a sample of 227 assessments, this analysis identifies nearly 8,300 citations over the 2008-2020 period [2].
The first observation is that the overwhelming majority of citations are made by researchers (94%). Public policy makers, administrations and the media make little use of public policy assessments: 60 assessments in our sample were never even mentioned by these stakeholders over the period under review. Secondly, on an aggregate basis, citations increase rapidly between 2012 and 2020, from 230 citations in 2012 to almost 1,700 in 2020. The increase in fact occurs faster than the increase in the number of assessments available: each available assessment is cited 3.9 times in 2012, compared to 7.4 times in 2020. For public policy makers, public administrations and the media combined, the number of citations increased by a factor of 3.6 between 2012 and 2020. The number of citations remains stable, however, when corrected for the volume effect (0.4 citations per available assessment).
The “economy” as a topic is assessed frequently (30 times) and its assessments are among the most cited, with more than 50 citations on average per assessment. In contrast, topics such as “security and police” and “culture” are seldom assessed and are among the least cited (between 0 and 6 citations per assessment). Furthermore, the few assessments available on the topics of “family” and “environment” (6 assessments each) are more frequently cited than the multiple assessments on the topic of “labour”, with an average of 53, 42 and 35 citations observed, respectively, per assessment For all topics combined, assessments are more often cited by these stakeholders if they are written in French, published by researchers coming from a public administration, or if they provide extensive (and not necessarily conclusive) results coupled with sufficiently broad recommendations. Confirmed by interviews with researchers and French National Assembly committees, these stylised facts have the potential to inspire best practices in evaluators keen on contributing to public debate.
Part 2 – What assessments are conducted before and after a law is passed?
Public policy assessments are gaining ground in France Their actual utility still needs to be measured, however. This can be done by looking at the response to academic assessments in the public debate [1]. From a more unprecedented standpoint, we can also look at the number of broadly defined evaluative research publications cited during the legislative process and the number of impact assessments produced after a law is passed [2].
In a sample of 262 laws passed between 2008 and 2020, an average of 18 assessment studies were cited before the vote. This number is higher for bills put forward by the executive branch (projets de loi) than for bills put forward by the legislative branch (propositions de loi) or for Finance Act measures. Only 14 laws did not give rise to assessment citations. Conversely, some laws generate more than a hundred citations, with the record going to the 2019 Pacte Act. The numbers have been on the rise, with 25 citations for each law in 2020 compared to only 8 in 2008. For executive-branch bills, 25% of citations appear in the government’s preliminary impact studies, and the rest in parliamentary proceedings. As expected, there are more citations for laws containing an evaluative or experimental clause, and for those with strong media impact.
After being passed, 40% of the 262 laws were subject to at least one ex-post assessment, averaging 2.7 assessments per law (1.1 for the whole sample). Logically, given the time needed to do an ex-post assessment, the percentage of laws evaluated is higher for the 2008-2017 period (46%). From 2008 to 2020, this percentage is higher for Finance Act measures (76%) and lower for legislative-branch bills (25%). It sits at 62% for laws subject to an evaluative or experimental clause. Laws on labour, the economy, housing or energy were evaluated much more often than laws on tourism, security and police, or civil service. Nearly 42% of assessments were carried out by public administrations, 23% by researchers, 18% by joint committees and 13% in the course of parliamentary proceedings.
Roughly half of assessments, covering 74 laws, contain recommendations: 45 of these laws followed - at least partially - one of these recommendations, i.e. 61%. This percentage is higher for laws with an assessment clause (66%) or an experimental clause (72%). This statistical analysis may be extended by case studies in order to clarify the dynamics at work and formulate appropriate recommendations.
Téléchargez le rapport Quelles évaluations des politiques publiques
pour quelles utilisations ?
Téléchargez la note d'analyse 109 Qui utilise les évaluations académiques
des politiques publiques ?
Auteurs
