Interactive design of multimodal user interfaces Reducing technical and visual complexity

• Published in: Journal on multimodal user interfaces Vol. 3; no. 3; pp. 197 - 213
• Main Authors: König, Werner A., Rädle, Roman, Reiterer, Harald
• Format: Journal Article
• Language: English
• Published: Berlin/Heidelberg Springer-Verlag 01-04-2010
• Subjects: Computer Science; Image Processing and Computer Vision; Original Paper; Signal,Image and Speech Processing; User Interfaces and Human Computer Interaction; Natural interaction; Semantic zooming; Squidy; Design environment; Post-WIMP user interface; Multimodal user interfaces; Zoomable user interface; Multimodal interaction
• ISSN: 1783-7677; 1783-8738
• DOI: 10.1007/s12193-010-0044-2

In contrast to the pioneers of multimodal interaction, e.g. Richard Bolt in the late seventies, today’s researchers can benefit from various existing hardware devices and software toolkits. Although these development tools are available, using them is still a great challenge, particularly in terms of their usability and their appropriateness to the actual design and research process. We present a three-part approach to supporting interaction designers and researchers in designing, developing, and evaluating novel interaction modalities including multimodal interfaces. First, we present a software architecture that enables the unification of a great variety of very heterogeneous device drivers and special-purpose toolkits in a common interaction library named “Squidy”. Second, we introduce a visual design environment that minimizes the threshold for its usage (ease-of-use) but scales well with increasing complexity (ceiling) by combining the concepts of semantic zooming with visual dataflow programming. Third, we not only support the interactive design and rapid prototyping of multimodal interfaces but also provide advanced development and debugging techniques to improve technical and conceptual solutions. In addition, we offer a test platform for controlled comparative evaluation studies as well as standard logging and analysis techniques for informing the subsequent design iteration. Squidy therefore supports the entire development lifecycle of multimodal interaction design, in both industry and research.