Research new ways of interacting with complex systems and semi-autonomous agents using speech and dialogue. Identify existing and develop novel capabilities for NASA missions.
The goal of the RIALIST (Research In Advanced Language, Interfaces and Speech Technology) group is to conduct leading edge research on advanced dialogue capabilities for multi-modal dialogue systems and apply these results to applications in NASA programs. Our focus is on Spoken Dialogue Interfaces to semi-autonomous agents and training systems. There are many different types of agents being developed by NASA giving a rich variety of possibilities for experimentation. The research builds on extending and augmenting established speech and language technology such as the Nuance recognizer, state of the art speech synthesizers such as ATT Natural Voices, the SRI Gemini NL system and Open Agent Architecture systems.
The current research focuses on contextual interpretation, portability, asynchronous dialogue management, and natural dialogue designs. Present implementations include Procedure Assistant for use by astronauts on the International Space Station (ISS), a spoken dialogue interface to the Brahms multi-agent modeling system and to the Europa planner. Future research topics include dialogue move based dialogue management, language modeling based on sparse training data, and rational integration of multiple knowledge sources. Our group is actively exploring ways to integrate spoken dialogue interfaces into current and future NASA missions. Areas of interest include spoken dialogue interaction with robotic agents including planetary rovers, dialogue based training, augmentation of air traffic control, multi-media dialogue interfaces including eye tracking, and intelligent cockpit instrumentation.
NASA has a major issue on longer missions including the International Space Station and manned Mars exploration of proving ground assistance to the astronauts and rovers. The old models of 24/7 staff availability from Mission Control are being stretched beyond human capability by these long missions. Clearly a way to provide assistance from the ground which involves fewer people is to use computer systems with spoken dialogue interfaces to provide information and expert advice. These systems need to be so easy and natural to use that special training will not be required. The systems need tohave the dialogue behavior of humans built into them. Our approach is based on mpirical studies of what people say to each other, what people say to actual systems and what they say to simulations of possible future systems.
Speech recognition and natural language processing have been developed over the past ten years so that commercial dialogue systems are used providing airline information, stock quotes and directory assistance. However, NASA's needs in this area differ from the commercial sector in at least two ways:
)
We refined our spoken dialogue interface to the Brahms work practices and knowledge capture modeling system which allows belief addition and revision. We collected speech data in pressurized space suits. At the field tests at JSC and Arizona, our system allowed the suit subject to ask questions about where he was located, and to issue movement and task oriented commands to the robot. The system also allowed the collection and distribution of scientific information to geological experts at JSC and team members at the field test. These efforts were in collaboration with Bill Clancey and Maarten Sierhuis.
)
The Mobile Agents Project addresses issues in planetary exploration with teams of astronauts, scientists and robots. RIALIST has participated in this task by providing a spoken dialogue interface to the Brahms system. This system allows the astronaut to ask questions about where he or she is located, and to issue movement and task oriented commands to the robot. The system also facilitates the collection and distribution of scientific information to experts on earth or team members in a habitation module. The progress of this project is measured by the successful execution of field tests in Mars analog environments such as the Arctic and the desert.
Our current research focuses on developing a spoken dialogue system to assist astronauts in performing procedures on the International Space Station (ISS). This requires researching how astronauts read procedures to each other, developing a dialogue system which mimics the salient features of this interaction, and discussing with astronauts what features and capabilities they want in such a system. A system has been developed based on dialogue moves which reads a procedure to an astronaut, under his direction, giving each step, reading cautions and warnings, allowing voice notes to be added and setting alarms. Advanced features to be studied and implemented include challenge mode to ask if each step has been done in critical operations, fast mode to read only the titles of each large step for the experienced person. The system will be tested by astronauts and feedback from their experience incorporated into subsequent versions of the system. Future research includes indirect confirmations, corrections, clarification dialogues, and task knowledge into the system. Some system architecture issues will be studied in order for the system to be able to be flight certified, yet be able to run several types of procedure within the same system.
Thus far we have implemented a seriew of demonstration systems. In December 2002, 7 astronauts tried the system for an hour each and provided substantial feedback and suggestions. The April 2003 system incorporates many of those suggestions, and will again be tested by astronauts in 2003.