Berlin, Germany - IBM, Uppsala University and the Swedish Institute of Space Physics are working on a new project to analyze massive volumes of information in real time to better understand 'space weather'.
The scientists plan to sample high frequency radio emissions from space to study and forecast the effect of space weather - plasma eruptions on the sun that can adversely affect energy transmission over power lines, communications via radio and TV signals, airline and space travel, and satellites.
However, the recent advent of new sensor technology and antenna arrays means that the amount of information collected by scientists surpassed the ability to intelligently analyze it. IBM InfoSphere Streams, new software derived from IBM Research project System S, enables large volumes of data to be analyzed in real time making an entirely new level of analytics possible, says the company.
Using InfoSphere Streams to analyze data from sensors that track high frequency radio waves, large amounts of data can be captured and analyzed on the fly. Over the next year, the project is expected to perform analytics on at least six gigabytes a second or 21,600 gigabytes per hour - equivalent to processing all the data on the internet, says IBM.
"InfoSphere Streams is opening up a whole new way of doing science, not only in this area, but any area of e-Science where you have lots of data coming in from external sources and sensors, streaming at such high data rates you can't handle it with conventional technology," says Dr. Bo Thide, professor and head of research, Swedish Institute of Space Physics and director of the LOIS Space Center. "It has helped create a paradigm shift in the area of online observation of the earth, space, sun and atmosphere."
As critical infrastructure such as power grids and telecommunications networks become more digitally aware, instrumented and interconnected, it is increasingly important to understand how these can be affected by influences such as electromagnetic interference or other changes in the atmosphere.
Researchers at Uppsala University and the Swedish Institute of Space Physics worked with the LOIS Space Center facility in Sweden to develop a new type of tri-axial antenna that streams three-dimensional radio data from space, extracting a magnitude more physical information than any other type of antennae array before. Since researchers need to measure signals from space over large time spans, the raw data generated by even one antenna quickly becomes too large to handle or store.
"We've embarked upon an entirely new way of observing radio signals using digital sensors that produce enormous amounts of data," Thide adds. "With this type of research, you have to be able to analyze as much data as possible on the fly. There is no way to even consider storing it. InfoSphere Streams is playing a pivotal role in this project. Without it, we could not possibly receive this volume of signals and handle them at such a high data rate because until now, there was not a structured, stable way of analyzing it."
The technology addresses this problem by analyzing and filtering the data the moment it streams in, helping researchers identify the critical fraction of a percent that is meaningful, while the rest is filtered out as noise. Using a visualization package, scientists can perform queries on the data stream to look closely at interesting events, allowing them not only to forecast, but to nowcast events just a few hours away. This will help predict, for example, if a magnetic storm on the sun will reach the earth in 18-24 hours.
The ultimate goal of the project at Uppsala University with InfoSphere Streams is to model and predict the behavior of the uppermost part of the atmosphere and its reaction to events in surrounding space and on the sun.
The work could have a major impact on future science experiments in space and back on earth. With a unique ability to predict how plasma clouds travel in space, new efforts can be made to minimize damage caused by energy bursts or make changes to sensitive satellites, power grids or communications systems.