Chicago (IL) – While praised, today's multi-touch input methods are clearly in its infancy. It recognizes only a handful of gestures, it isn’t terribly accurate and not a great tool for precise object manipulation. It seems, however, Apple has many more ideas for multi-touch and is working on adding new technology that could improve multi-touch accuracy, enable more detailed gestures and perhaps change the way you use your computer.
Apple has filed and almost endless number of multi-touch-related patent applications since the iPhone was introduced. Especially noteworthy is gigantic 371-page filing credited to Steve Jobs that covers the iPhone user interface in great detail. Only a handful of the patented inventions are used in Apple products so far, and today we got another taste of what might be cooking in Apple’s labs. The latest patent filed with the US Patent & Trademark Office today is called “Multitouch Data Fusion” and is credited to John Elias and Wayne Westerman, key ex-executives of Fingerworks, the company that Apple acquired during the development of the original iPhone.
The idea proposes a handful of varied input methods designed to improve the accuracy and intelligence of multi-touch interfaces and enable more complex and detailed gestures than what is supported by today's technology. By feeding the system with information collected from other sensing techniques, the multi-touch interpretive capabilities can be significantly improved and deepened. Specifically, gaze vector and voice and facial recognition information could be used to fine-tune data input, while the device movement can be used to resolve possible user interface conflicts.
Advanced recognition technologies are already adding value to everyday applications. For instance, Google's recently updated Picasa application for managing photos offers an almost spooky facial recognition technology that can single out pictures with people's faces with amazing accuracy.
Apple has an interesting explanation how these technologies can be used to improve user experience on an application level and augment multi-touch input. In one paragraph of the patent filing the authors note that voice commands are better suited for quickly changing an object's color, a task that would take longer to perform using multi-touch. However, multi-touch works better for object rotation and resizing.
A video of your fingers in action could be used to fine-tune multi-touch accuracy. Utilizing a built-in device camera (for example, iSight in Mac laptops) and a swing mirror, the system records fingers interacting with the multi-touch surface to better determine the positioning and number of fingers used and enable more specific gestures.
Your gaze can also offer very precise information about objects you intend to manipulate. The patent filing proposes that a built in device camera records your rapid eyes movements in order to locate objects on the screen that are the focus of your attention. Instead of using a mouse, explained Apple, a user simply looks at any object or window to have it selected.
And what do you do if all these augmentative input methods fail and the device recognizes a wrong gesture? You record user's face to detect his frustration. The example provided in the patent depicts hypothetical situation when a user tries to render scroll gesture using three fingers instead of two. Confronted with what is detected as a frustrated expression on a user's face, the system could show a warning or simply accept an input as valid or invalid.
You can even expect games to be enhanced with biometric input information such as fingerprints, skin impedance, heart rate, or body temperature. Of course, biometric input in combination with multi-touch fits security and fitness applications perfectly. "Hand characteristics such as size, shape, and general morphology can be used to identify an individual for the purpose of allowing access to secured areas, including computer systems," the patent filing states. "While hand characteristics alone would not provide a sufficient level of identity verification, it could be the first door through which a user must pass before other security measures are applied."
As with every Apple patent, the scenarios depicted in the Multi-Touch Data Fusion should be taken with a grain of salt. Like other companies, these filings do not necessarily point to upcoming products. What is especially interesting to note is that enhanced multi-touch, especially if a video camera and hand or face rendering is involved is not just a software problem, but a significant hardware hurdle to take. Especially AMD has been showing such concepts for nearly two nears – examples in which your hand is recorded through a webcam and rendered in real-time using a GPGPU application. This virtual hand is then used to manipulate content on a screen.
So, how is it possible that Apple ended up with this patent claim? We don’t know, but we will ask AMD.
Apple has filed and almost endless number of multi-touch-related patent applications since the iPhone was introduced. Especially noteworthy is gigantic 371-page filing credited to Steve Jobs that covers the iPhone user interface in great detail. Only a handful of the patented inventions are used in Apple products so far, and today we got another taste of what might be cooking in Apple’s labs. The latest patent filed with the US Patent & Trademark Office today is called “Multitouch Data Fusion” and is credited to John Elias and Wayne Westerman, key ex-executives of Fingerworks, the company that Apple acquired during the development of the original iPhone.
The idea proposes a handful of varied input methods designed to improve the accuracy and intelligence of multi-touch interfaces and enable more complex and detailed gestures than what is supported by today's technology. By feeding the system with information collected from other sensing techniques, the multi-touch interpretive capabilities can be significantly improved and deepened. Specifically, gaze vector and voice and facial recognition information could be used to fine-tune data input, while the device movement can be used to resolve possible user interface conflicts.
Advanced recognition technologies are already adding value to everyday applications. For instance, Google's recently updated Picasa application for managing photos offers an almost spooky facial recognition technology that can single out pictures with people's faces with amazing accuracy.
Apple has an interesting explanation how these technologies can be used to improve user experience on an application level and augment multi-touch input. In one paragraph of the patent filing the authors note that voice commands are better suited for quickly changing an object's color, a task that would take longer to perform using multi-touch. However, multi-touch works better for object rotation and resizing.
A video of your fingers in action could be used to fine-tune multi-touch accuracy. Utilizing a built-in device camera (for example, iSight in Mac laptops) and a swing mirror, the system records fingers interacting with the multi-touch surface to better determine the positioning and number of fingers used and enable more specific gestures.
Your gaze can also offer very precise information about objects you intend to manipulate. The patent filing proposes that a built in device camera records your rapid eyes movements in order to locate objects on the screen that are the focus of your attention. Instead of using a mouse, explained Apple, a user simply looks at any object or window to have it selected.
And what do you do if all these augmentative input methods fail and the device recognizes a wrong gesture? You record user's face to detect his frustration. The example provided in the patent depicts hypothetical situation when a user tries to render scroll gesture using three fingers instead of two. Confronted with what is detected as a frustrated expression on a user's face, the system could show a warning or simply accept an input as valid or invalid.
You can even expect games to be enhanced with biometric input information such as fingerprints, skin impedance, heart rate, or body temperature. Of course, biometric input in combination with multi-touch fits security and fitness applications perfectly. "Hand characteristics such as size, shape, and general morphology can be used to identify an individual for the purpose of allowing access to secured areas, including computer systems," the patent filing states. "While hand characteristics alone would not provide a sufficient level of identity verification, it could be the first door through which a user must pass before other security measures are applied."
As with every Apple patent, the scenarios depicted in the Multi-Touch Data Fusion should be taken with a grain of salt. Like other companies, these filings do not necessarily point to upcoming products. What is especially interesting to note is that enhanced multi-touch, especially if a video camera and hand or face rendering is involved is not just a software problem, but a significant hardware hurdle to take. Especially AMD has been showing such concepts for nearly two nears – examples in which your hand is recorded through a webcam and rendered in real-time using a GPGPU application. This virtual hand is then used to manipulate content on a screen.
So, how is it possible that Apple ended up with this patent claim? We don’t know, but we will ask AMD.