The Hunt for Dark Data

1960s Satellite recordings recovered by a CU team may tell us something about climate change, hurricanes, rainforests and a lot of other things we care about now.

In 1964 NASA launched the first in a series of seven satellites that would provide a wealth of new information about Earth’s atmosphere, weather patterns, ecosystems and oceans. The satellites, part of the Nimbus program, photographed the planet twice daily, taking temperature readings, observing clouds, gathering oceanographic information and testing meteorological sensors — generating millions of images.

The data and images were eventually dispersed to archives around the country. Some landed in a warehouse in Asheville, N.C., and it was there that thousands of feet of deteriorating 35 mm film stored in dozens of poorly labeled boxes remained unnoticed until CU geoscientist Dave Gallaher came along.

Since then Gallaher and lead scientist Garrett Campbell have been immersed in a major recovery and restoration project that has revealed the old information’s value for 21st-century studies of polar sea ice, and potentially for hurricanes, Brazilian rainforests and other topics of study also.

“For scientists, data from 30 years ago is every bit as valuable as data that came today,” says Gallaher, technical services manager for the National Snow and Ice Data Center at CU’s Cooperative Institute for Research in Environmental Sciences (CIRES).

Projects like Gallaher’s are sometimes called “dark data rescue.” They aim to recover information captured by obsolete technologies or stored in hard-to-decipher repositories, such as old logs, that might still prove useful to science.

In a 2007 project at Columbia University’s Lamont-Doherty Earth Observatory, for example, scientists converted to a modern digital format decades’ worth of seismic recordings of underground nuclear explosions in Eurasia. They hoped the data would help them distinguish recordings of explosions from those of earthquakes. Another project involving CIRES recruited more than 17,000 volunteers to transcribe ship logbooks, in some cases centuries old. The aim was to expand a marine weather database that could aid climate change studies.

“People are rescuing data from the edge of oblivion,” says Gallaher, whose main interest is analyzing remote satellite sensings.

He first learned about the Nimbus data at an American Geophysical Union meeting in 2009. It thrilled him: Maybe it would reveal the extent of the sea ice in Earth’s polar regions during the ’60s, long before scientists began officially tracking it, in 1979.

“The taxpayers had spent, in some cases, billions to collect this data,” he says. “To spend a few thousand dollars to recover it all is a pretty good investment.”

Dark data rescue can be urgent and often involves more than merely finding information. Old data is often unusable without converting it from its original format to one readable by current technologies. In order to see the photos stored on the Nimbus film, Gallaher had to track down a discontinued Kodak HR-500 film scanner.

He hired a team of CU undergraduates to scan each of 250,000 images taken by the satellite. Painfully slow, the giant Kodak jammed constantly. Once an image was scanned onto a computer, distinguishing ice edges from clouds also proved time-consuming.

“We had to inspect every individual image — all 250,000 of them,” says Campbell, also of CIRES, who has devoted his entire career to working with satellite observations, specifically those looking at real-time cloud formations.

“We compiled all of these images and created our own Google Earth,” says Gallaher’s son, Carl Gallaher (A&S ex’14), who worked on the project for two years as a student. “Sometimes you could see some really cool images like Greenland or the Red Sea.”

But the work paid off: The team logged the earliest collection of polar sea ice ever recorded.

Once patterns in sea ice edges emerged from the high-resolution scans, composites were stitched into digital maps for further analysis. They showed that the year 1964 had the largest sea ice extent ever measured in Antarctica prior to 2014 — approximately 8 million square miles. The year 1966 had the smallest ever recorded, breaking the record set in 1986.

“The Antarctic may be showing aspects of climate change,” Gallaher says. “New sea ice proxy analyses have been developed, but until the Nimbus data became available, [scientists] had no way to test them. In addition, climate modelers are using this data to improve and test their models.”

All of the team’s work is publicly available at no cost. By the end of last year, more than 200,000 visitors had examined or downloaded data from the project website. A video about the work received 7,000 views the day it was posted.

For scientists, data from 30 years ago is every bit as valuable as data that came today.

Gallaher remains interested in finding and retrieving dark data from satellites and now has his eye on another series of satellites from the 1960s called ESSA.

“There were 250 satellites put up in the 1960s that could see the poles,” he says. “Of those, very little of that data exists. This [Nimbus data] is some of the first to come back.”

Because the CU team scanned every image recorded by the Nimbus satellites regardless of whether they showed the Earth’s poles, their work is helpful to scientists in other fields, especially meteorology. Gallaher and Campbell found they had numerous photos of the disastrous Hurricane Camille, for example, a Category 5 storm that slammed the southern U.S. in the summer of 1969. Until recently, scientists didn’t know the images ever existed. Gallaher’s team has also received inquiries from scientists eager to evaluate whether the recovered photos revealed clear-cutting in Brazil’s rainforests.

“You have to have a compelling reason in science or business to get data like this back,” Gallaher says. “We came up with that reason, and that is why this thing took off. But I don’t think that’s the reason people care. The real reason people are into it is because this is like a treasure hunt. We got in there, we found all this stuff, we did the archaeology, we dusted it off, and people want to know, ‘What did you find?’”

Photography courtesy NSIDC