Data-Driven Policy Making — A Cautionary Tale
Data — The Sorcerer’s Stone
The meteoric growth in digital technologies has undoubtedly been the defining element of the late twentieth and early twenty-first centuries. The digital realm has redefined sundry aspects of our everyday lives from socializing, dating, banking, shopping, to even how and where we work. The concept of ‘Work from Home’ would have been inconceivable if not for the digital technologies (such as video conferencing and cloud storage) that facilitate it.
This rise in digital technologies and computing power have enabled us to collect, organize, transform, and analyze data like never before. The digitization of several business and governmental processes enable us to capture a cornucopia of information. Further, this rise in the availability of data has been complemented by a stellar growth in computing power. Today, we can implement the most advanced of statistical techniques using vast datasets in a matter of seconds, with nothing more than a run-of-the-mill computer.
This ubiquitous presence of data and computing power have made a tremendous impact on how we make decisions and have spurred the recent buzz around ‘data-driven’ decision making. Some now believe that data is the sorcerer’s stone, the magic bullet, that has the potential to answer any policy question, sans subjectivity. Some go on to claim, everything ought to be data-driven and that there is no need for any theory or philosophy. Is this really the case?
The Complicated Relationship between Data and Public Policy
Off to our hypothetical country of PeopleLand. The civil society of PeopleLand are debating if the government must continue running its two job training centers that each offer a unique training program (named WeGrow and WeThrive respectively), or if it must axe one of the centers to reduce costs and improve efficiency. This is in fact one of the more vapid questions a government faces. How can data enable decision making in this case?
The Data-Driven Solution
In response to this debate, the government commissions a statistical study called a ‘randomized controlled trial’. For the trial the government randomly selects a sample of eligible participants. It then randomly splits the sample into three groups.
How does the splitting work? Let’s say each participant in the sample draws from a giant bowl with balls of three colors — red, green and blue. All those that drew a red colored ball are assigned to participate in WeGrow, while those that drew a green ball are assigned WeThrive. Those that drew a blue ball are not permitted to participate in either programs.
The government collects data on the income of all participants at two different points in time — a couple of months prior to and a couple of months post the job training.
Given that the workers of PeopleLand are often reluctant to participate in such studies, the government had to provide a $500 cash incentive to encourage participation. Now, let’s look at the results of the study.
Clearly, the group that underwent no job training shows the least increase in monthly income during the period of study, while WeThrive shows the most increase (an increment of around $7,000 as opposed to $5,000 of WeGrow).
The results are all fine but the obvious question that pops up is about the randomization. Why did the government have to go through the trouble of asking participants in the sample to draw from the bowl and assign them into three groups?
Let us suppose the government did not randomize the assignment of job training programs and let the participants in the sample choose for themselves.
Surprise! The results tend to indicate that WeGrow is more effective (an increment of around $8,000 as opposed to $7,000 of We Thrive). This is quite to the contrary of the previous results! What’s happening here?
Notice how in the case of randomization, the average monthly income, prior to training, are very similar across the three groups. However, in the second case, the average pre-training income across the three groups are very different. This is the crux of the problem. When we do not randomize and leave it to the participants’ choice, the participants in the three groups often end up being very different and are hence incomparable. Why are they incomparable?
It could be the case that the center that conducts WeGrow is in an industrial suburb, primarily attracting industrial workers who could greatly benefit from any job training. On the other hand, WeThrive could be conducted in an urban center, primarily attracting the already well-to-do urban workers, who have a lesser scope to further increase their income with job training. Hence, apart from the quality of the job training program itself, who participates in the program also makes a big impact on the extent of improvement that can be achieved.
In a randomized trial, the government ensured that there is no clustering of urban workers in WeThrive, or industrial workers in WeGrow, by using a lottery to assign the training that they attended. Randomization ensures that the odds of those in urban centers attending WeGrow (i.e. drawing the red ball) is the same as that for industrial workers, making the groups more comparable. Hence, the results from the randomized controlled study are not affected by who attends the program and purely capture the impact of the program itself. This gives us a more accurate estimation of the true impact of the job training programs.
Great! The government has now understood from the data that WeThrive is better (in terms of boosting income). Now, how should the government act on these findings? Shut down WeGrow? If one were to purely go by the data, one would recommend shutting down WeGrow.
However, the reality is a little more complicated than what is captured in the data.
The Decision Making Process and the Pitfalls
The data merely captures income of the workers prior to and post job training, but does it capture the rationale behind their decision making process in choosing which training to attend?
Workers choose their job training programs based on the proximity of the training center. Therefore, left to their own will, industrial workers would choose WeGrow, while urban workers would choose WeThrive. Further, the cost of making the commute to the urban job training center for industrial workers is very high, and hence they would decide not to attend any job training at all if the WeGrow center (i.e. industrial region center) is shut down.
The only reason why industrial workers in the randomized controlled study made the commute to the urban training center was because of the $500 incentive they were provided, which covered their travelling expense. While it was feasible to provide the incentive for the study, it would be infeasible for the government to scale up and provide the incentive as a part of its policy, owing to the enormous cost it would entail.
Hence, if the government were to shut down WeGrow based on the results of the study, it would have adversely affected the industrial workers who would have not been able to attend any job training at all. This example illustrates how a purely data driven policy choice, without the necessary decision making model could have devastating consequences.
When purely relying on data, we often fail to recognize the theoretical underpinnings of decision making and behavior. Being cognizant of the decision making theory would have helped develop the study better, ensuring that the study took into consideration key factors such as cost of commute and proximity. In short, there is tremendous value to theoretical explorations of the factors that affect people’s choices, and how they value the outcomes of such choices.
The Role of Philosophy
How about philosophy? Sure, there is tremendous scope for it as well. In our study, we used income to measure the outcome of job training. Should we have included leisure time as well? Time spent with family and friends is definitely valuable to many, so why should the study not capture that? The broader philosophical question being, what is well-being? Is it money, fame, power, prestige, leisure time, societal approval, some combination of the above or something else completely?
It is important to remember that ‘data collection and analysis’ is informed by a theory and philosophy. Lack of a strong theoretical or philosophical underpinning could lead to flawed studies that simply fail to adequately inform decision making.
The increase in computing power and the ubiquitous presence of data should not stop us from positing theories, debating, and most importantly critically thinking through our statistical results. Paraphrasing Jesse Shera, we should never find ourselves in a position where we have “data everywhere and not a thought to think”.
Author: Akshay Natteri Mangadu
The example in this article is broadly inspired by the Roy Model of Self Selection. In essence, the issue with the policy decision made by PeopleLand was that it did not pay due attention to the factors influencing self selection.
