A University of Washington team has developed a system it says makes all software effectively open source, allowing users to add custom features to any program.
"Microsoft and Apple aren't going to open up all their stuff. But they all create programs that put pixels on the screen. And if we can modify those pixels, then we can change the program's apparent behavior," said James Fogarty, a UW assistant professor of computer science and engineering.
His approach hijacks the display to customize the user interface. It takes advantage of the fact that almost all displays are made from prefabricated blocks of code such as buttons, sliders, check boxes and drop-down menus. Prefab looks for those blocks as many as 20 times per second and alters their behavior.
"We really see this as a first step toward a scenario where anybody can modify any application," Fogarty said. "In a sense, this has happened online. You've got this mash-up culture on the Web because everybody can see the HTML. But that hasn't been possible on the desktop."
With Prefab, a user wishing to write and listen to music at the same time wouldn't need to click back and forth between Word and iTunes, but could simply add a few iTunes buttons to the Word toolbar.
Having more control over widely used programs would also allow people to benefit from accessibility tools.
Prefab can also produce more advanced effects, for example creating multiple previews of a single image in Photoshop. Behind the scenes, Prefab moves the sliders to different points, captures the output and then displays all of them on a single screen.
The system could also allow programs to move easily from computer screens to mobile devices. "It dramatically lowers the threshold to getting new innovation into existing, complex programs," Fogarty said.
The researchers are continuing to develop Prefab and are exploring options for commercialization. It'll be interesting to see what Microsoft and Apple make of it.
The system will be shown off at the Association for Computing Machinery's Conference on Human Factors in Computing Systems in Atlanta on April 14.