In today's NY Times, Steven Lohr has published a great piece about a new National Academy of Sciences report calling for private sector "Big Data" to be used as one source of real time knowledge on the extent and in which sectors that robots and AI is taking over human work.
“But our surveys are not really designed to track technology or its impact,” said Professor Haltiwanger, who was a member of the expert panel. “The best shot at that is the private sector data.”
This is an interesting quote because it suggests that the private sector has "better data" than the public sector that administers surveys.
In a very different context, I had the same thoughts and ran into the same incentive problems. Let me explain.
4 years ago, I served on a panel for the Air Resources Board in California to investigate whether the state's AB32 regulation (the low carbon regulation) was harming the California economy.
I argued that each electric utility in California has monthly electricity bills for households, industry and commercial accounts. If these electric utilities could supply anonymous data on each account's monthly energy consumption and expenditure and a unique identifier (so Matthew Kahn's privacy could be protected by calling me #7896). A research team could investigate whether household energy expenditure (price * quantity) was rising quickly as carbon pricing raised the price of California energy. The Wall Street Journal argued that Californians would be unable and unwilling to substitute away from increasingly expensive resources (an inelastic demand). The big data revolution would allow a test of this.
If we knew each account's zip code (so Beverly Hill is 90210). A researcher could slice and dice the data to study expenditure dynamics in rich areas, poor areas, black areas, Hispanic areas, rural areas, urban areas, liberal areas etc. Using the industrial and commercial account data, one could engage in similar sample splitting such as comparing big firms to small firms or by industry code.
This would allow us to study the economic costs (or incidence) of the regulation. My ideas were not implemented. Because the regulatory authority was unable to think of a mechanism to induce the electric utilities in the state to assemble their private data to serve the "public good" (i.e knowledge creation). Now, do you see why this post has its title?
My experience will playout again. Google offers you free services such as gmail in order to study you in your natural environment. Such firms as Google and Facebook are not going to voluntarily share such data. This would be a "takings". The government must offer such firms an incentive to provide such data. Rival firms will hope to learn about their rivals from such data.
So, I agree with the NAS panel that private Big Data must be assembled to study the robot challenges and opportunities but how does the NAS propose to access such data? Volunteer agreements will create sample selection bias concerns. Data confiscation would set a dangerous precedent.