UnveilML: Combining Econometrics and Machine Learning to improve the estimation of factors we cannot see

For decades, economists have relied on state space models to pull an invisible signal out of noisy measurements. The idea is to treat what we observe (say, macroeconomic statistics, asset prices, temperature readings) as the visible surface of a hidden variable we actually care about (say, the business cycle market risk, global warming), and to work backwards to that hidden variable. Such state space models have real virtues: They are transparent and interpretable. They allow for and benefit from strong time-dependence. And they come with honest margins of error, so an estimate arrives with a built-in sense of how sure we can be.

Their weakness is scale and flexibility: State space models work beautifully with a handful of series and linear relationships, but they strain when you hand them thousands of variables at once, or ask them to capture the tangled, nonlinear ways the real world behaves. Market risk might be driven by thousands of assets moving together; the classical tools become computationally infeasible long before you get there. Back to the trading floor example: they can follow a quiet desk of a few traders, but not the roar of the whole floor.

Machine learning is almost the mirror image. It thrives on enormous, messy datasets and finds subtle patterns no human, and no traditional model, would ever spot. It can pay attention to the whole floor at once. But it has three shortcomings that limit its use in economics: It tends to be a black box, giving you an answer without telling you what that answer means. It has a short memory, tuned to recent patterns, when economic and climate data are so often driven by forces that persist for years or decades. And it rarely tells you how confident it is, so a precise-looking number can quietly conceal a great deal of uncertainty.

I will put these methods to work on three problems where the stakes are high and the hidden variable is of immediate interest.

The first is financial risk. Market-wide volatility is the force that drives asset prices and shapes how institutions manage their exposure, but it has to be inferred from the joint behavior of thousands of markets at once. This is precisely the high-dimensional setting where the classical tools give out. Better estimates here mean more robust stress tests and earlier warning of risk building across the system.

The second is the business cycle. Whether the economy is heating up or cooling down is one of the most closely watched hidden variables there is, and policymakers need it in something close to real time. By drawing on far larger datasets than traditional methods can handle, the goal is a sharper, more timely read for the institutions that set fiscal and monetary policy.

The third is global warming. Buried in the records of thousands of weather stations around the world is a common warming trend, distinct from seasonality, local weather and the quirks of individual instruments. UnveilML aims to extract that trend from the full, high-dimensional dataset, and, crucially, to attach honest uncertainty to it, so we know not just how much warming has occurred but how confident we can be in the figure.

If the project succeeds, we will be able to see the economy's and the climate's hidden drivers more clearly despite the noise. Just as importantly, we will be more honest about everything we still can't make out. In a world increasingly tempted to treat whatever a powerful algorithm produces as fact, that second achievement might matter most of all.