Rafael Frongillo
How should we measure the accuracy of predictions? If the weather forecast calls for a 10% chance of rain, and it rains, was that a bad forecast? How would you explain what a "good forecast" is to a computer?
These are the types of questions on the mind of Rafael Frongillo, an assistant professor in the CU Boulder Department of Computer Science. He studies how changing the way we measure prediction accuracy can ultimately influence the decisions made by humans and machines alike.
Frongillo’s research lies between theoretical machine learning and economics, using convex analysis, game theory, optimization and other techniques to understand and quantify the exchange of knowledge. This theoretical work has many applications – from designing self-driving cars and betting markets to forecasting the spread of a disease – all of which rely on making and evaluating predictions.
The common thread, Frongillo said, is loss functions. Loss functions are mathematical tools that measure prediction accuracy or inaccuracy. Well-designed loss functions can guide both computers and humans in making accurate predictions.
“The human side is where economics comes in. How should a firm hire someone to produce business-relevant forecasts? How do you structure a contract around that exchange, so better performance leads to higher pay? The answer to that question is actually the same as one as when you are designing a machine-learning algorithm – the contract becomes a loss function that tells the computer how to produce a good forecast from the data,” Frongillo said. “At the mathematical level, these problems are heavily intertwined – loss functions are in turn a basic building block in economic forecasting mechanisms.”
Machine learning is a branch of artificial intelligence that uses computer algorithms to solve problems, improving automatically through experience over time. In this case, computers make predictions by choosing the predictive model that best fits historical data, as judged by a loss function, Frongillo explained.
Yet a general framework to design and analyze loss functions is lacking for many common machine-learning tasks. Meanwhile in economics, an efficient information economy is crucial to facilitate the trade of information, but current loss function systems fall short in instances involving competition or collaboration.
Frongillo recently earned a National Science Foundation CAREER award to address this overlapping issue at the theoretical level. The award provides about $500,000 over five years to support the research and educational activities of early career faculty members who have the potential to become leaders in their field.
Through the new award, he will develop a general framework to design and analyze loss functions in machine learning, as well as design collaborative and competitive mechanisms to facilitate markets for predictions, data and beyond.
“These results will impact a number of key economic sectors as our society continues to progress toward an information economy,” Frongillo said. “Well-designed mechanisms could increase consumer access to information. They can also improve economic efficiency as micro-contracts replace formal employment, just like the sharing economy for transportation and lodging. Instead of driving people around in your spare time for extra cash, you could upload data to improve forecasts.”
Before coming to CU Boulder, Frongillo received his PhD in computer science at the University of California Berkeley in 2013, with support from the National Defense Science and Engineering Graduate Fellowship Program. He also served as a postdoctoral researcher at the Center for Research on Computation and Society at Harvard University and at the Microsoft Research Lab in New York.
Through this award, Frongillo will also work with the Colorado Data Science Team, an interdisciplinary club and course he founded in 2016 that combines education, research and outreach – all while broadening involvement of students historically under-represented in computing.
“The consistent student interest in machine learning competitions and forecasting provides a unique testbed for new elicitation mechanisms, and I really want to tap into and support that in a formal way through this project,” he said.
