Esprit Project no. P29158 in the Theme:

Information Access and Interfaces






Flexible knowledge-based information access and navigation using multimodal input/output


Project Objectives

The project will provide the technology for bringing the information age into the home for everybody to use. The target product is a new concept for a device called the Home Information Center (HIC) which describes a device for multimodal access to information and services relevant for the everyday life in a family. The FLEX project will provide the FLEX Technology software for supporting intuitive navigation in and visualisation of information and flexible queries for information, as needed for the realisation of a HIC.

The markets for information access to home shopping, home banking, video-on-demand, web-access, traffic information, etc. are expected to explode within the next two years. The HIC addresses the niche of trend setting families who are open to changes and who already have PCs and audio-visual equipment. As the FLEX Technology of intuitive navigation and flexible search meets the users’ strong demand for enhancements of information access systems, the business prospects also include the integration of the FLEX Technology both into the partners’ own products as well as OEM products.

The Home Information Center

Today we see two user situations in the home: The TV in the livingroom as a lean-back situation, and the PC in the office as a lean-forward situation. In the lean back situation the user is being entertained and relaxes, and in the lean-forward situation the user is actively and focused producing. Already a variety of devices brings internet with thousands of programs to the living room through set top boxes, and also services such as home shopping, home banking, video on demand, health care, energy management, traffic information etc. are provided to the home. It is not sure that the lean-forward situation of a PC or the lean-back situation of a TV in a living room will be the right contexts or the right devices for the new services. Instead we try to define a new device HIC which is neither a lean back device as the traditional TV nor a lean forward product like a PC, but a kind of moving around device where users can get and provide information while they are occupied with other house doings. We call this a move-around situation for infotainment. See the illustration in Figure 1.1.

Generically, the Home Information Center is a multimodal input/output device that can be placed in any fixed position in the home and that supports flexible interaction and an intelligent underlying information structure. A vast set of services is relevant, and the design should allow for extensions with no need to redesign the system. The project will focus on services supporting information and messages in the daily life of a family. Examples of such services are:

The FLEX Technology

The utilisation of the phenomenally growing quantity of information and services offered online is hindered by relatively primitive access mechanisms, requiring users to point and click their way through myriads of hyper-links and web sites [Lau et al. 1997]. The HIC requires an improvement to the current praxis. FLEX addresses this problem by producing and integrating two key technologies that together will make up the FLEX Technology, cf. Figure 1.2:

  1. An intuitive navigation support system. The user must be provided a comprehensible mental model of the current information space and the system must be able to interpret concrete references to the system output [Benyon and Höök 1997]. The user understanding is supported by intuitive representations which may be ‘4D’ such as signifying informational distance by blurring and marking referents with different colours.
  2. A flexible query system. FLEX flexible, powerful query mechanisms [Christiansen et al. 1997] supported by an abstract information space adaptable to the user profile and current dialogue.

In addition, a number of supplementary technologies will be adapted and integrated:



With reference to the objectives of the Esprit Theme: Information Access and Interfaces, and to the above descriptions of the Home Information Center and FLEX Technology, it can be stated that:

The main objective of the FLEX project is to establish the technological platform for, and determine the value of, the synergy of intuitive interfaces, intelligent navigation and flexible search in the multimodal environment of a Home Information Center.

FLEX addresses the need for providing easy access to information for ordinary people in their homes. The overall product addresses the modern family's need for a higher and more coordinated information level. The intuitive interfaces, the intelligent navigation, and the flexible search are each exploitable separately, while they also support and complement each other nicely.

In relation to the approach of this project the objectives may be detailed according to users, technology, and business. The user driven objectives for this project are:

The technology driven objectives concern the software necessary for the Home Assistant, including:

The business objectives concern the cost and marketability of the HIC and the FLEX Technology, and include:

The FLEX Technology and the HIC

The overall area of FLEX is Navigation and search in information spaces for non-IT specialists, materialised in the concrete shape of the HIC. The main product of FLEX are software technologies materialised in a HIC demonstrator to validate the navigation and search paradigms. The key software technologies for this to succeed that are produced within the FLEX project are (cf. also Section 1.2):

In addition, the construction of the HIC requires the inclusion of off-the-shelf speech recognition and html rendering modules, and the assembly of the necessary hardware setup.

Usage and scenarios

The scenarios provided in this section serve the purpose to concretise what the system may be expected to do. In the beginning of the project it will be analysed which kinds of applications and combination thereof are most likely to boost user acceptance and commercial success. Together with the architectural description in Sections 2.1 and 2.2 this forms the preliminary specification of the system to be built. For the prototype and validation purposes a smaller selection will be made.

The typical users could be a family with half-grown children, each family member having his/her busy and not too coordinated activities. A larger scenario is provided at the end of this section. Other possible scenarios include:

By the end of the project the contents information will be represented by at least four different contents packages. The packages will represent different aspects of expected HIC uses:

The project will concentrate on the information aspect, but of course commercial trade is possible and expected for several of the above issues, e.g. film guides coupled with video-on-demand or business information coupled with ordering. However, these and other interactive uses such as games are not considered in the project.

The project currently has formal agreement with a cultural provider (CCIS, Edinburgh) and a TV and news provider (Danmarks Radio, Denmark), and informally with a sports results provider (Denmark).

A scenario for a couple in the morning:

Arnold is busy in the kitchen, washing up the dishes from the party he gave yesterday evening. He listens to the morning news in his loudspeakers, connected to his HIC in the kitchen. He is going to be at work within an hour, and the speaker was saying something about a roadblock on the road between his home and his work. He turns his attention to the HIC saying, "Turn on". He could have used touch or the keyboard as well, but his is occupied with his hands right now.

The screen now gives Arnold some possibilities. Arnold has earlier on defined his personal profile in the HIC, very dependent on the traffic situation between his home and his work, so one of the options given is service number 5 "traffic". He could say 5, but he chooses to say "traffic situation". He is now given some more possibilities, and says "local" number 2. The screen now shows the block on the main road, and he realises that he has to leave 10 minutes earlier, because he has to take another road. He hurries up, just leaving anything in the kitchen, taking the pen from the pen holder attached to the HIC, and quickly writes " Good morning dear. Have to leave early, will you finish cleaning up, and walking the dog. Love Arnold". This text is memorised in the HIC’s message heap.

He turns the HIC off (really switching standby mode, ready for new interaction) and leaves.

Jane is woken by the sound of his car, and when she comes down to the kitchen, she sees a flashing note indicator on the HIC. She says, "turn on" and she sees the message from Arnold attached with the local traffic map, and understands why the kitchen is in such a mess. She presses the CD button on the remote and begins cleaning up while she listens to the new record with Placido Domingo.

Finished cleaning up and walking the dog, she goes to the HIC, rolls out the keyboard and writes: "When will the road be cleared for traffic?" The HIC, intelligent and flexible as it is, automatically will know that she is referring to the roadblock on the local road, due to Arnold's request early on in the morning, displays and says "13:30". Jane then writes "mail Arnold", and the HIC then automatically turns on the Internet connection, turns on the mail application, and has filled out the "to:" address ready for keyboard input. Jane now writes "road is clear, take the easy way home? Love Jane" She says "send" and "Turn off", and is already on her way to her job.

Project work plan

This section briefly introduces the overall development process and work plan, including: Overview of work packages, overall process, overall partner roles, risks, person month distribution, duration, and milestones, and resources are given in subsections. Detailed work package (WP) descriptions are provided in Section 5. Detailed Gantt chart is given in Annex B.

The project period is 24 months.

Work package overview

FLEX will achieve its stated goals through the following 7 work packages (WPs), each consisting of a number of tasks (Ts) which will produce deliveries (Ds) of various types. The detailed description of work packages is given in Section 5.

Overall process

The overall development approach will be iterative in the style of evolutionary prototyping, using early WOZ experiments with real users and later complete setups, finally ending with in-real-homes experiments. Early experiments will be employed to support the design of how users should experience the Home Assistant and to get data on what to focus on with the intuitive navigation design, what users are likely to refer to, and what the implied semantics of their input is. The results of the experiments will also be used as input to the analysis of a) how the information space should be organised to support the visualisation and navigation, given the included task domains, and b) how the interaction control should utilise contextual information, and c) the language models. Further, it will be input to the validation work. During the project experiments will be performed on the basis of an increasingly complete prototype, with the aim of getting early feedback from the real users in a realistic application setting. All relevant parameters will be measured, including quality, response time, response functionality, and the users’ satisfaction. The feedback from the ongoing experimental setting will be interpreted and utilised in the further development towards the specific technical objectives.


The overall HIC will be developed as a series of prototypes, leading to a final pilot like complete system. They will be currently improved during the project period. Four distinguished prototypes have special status:

Prototype P0 is delivered by the end of month 6, and will be used for the very first discount engineering experiments with the HIC system for aid in the design specification.

The P0 does not contain software, but presents a first design of appearance and visualisation.

Prototype P1 is delivered by the end of month 11, and will be used for the first laboratory experiments with the HIC system, collecting data on the usage, including user actions and language to be used in constructing the parser.

This prototype includes full monitor setup in the laboratory, a PC running the HIC client and Contents server, a first version of the display visualisation, a simple interaction model (and control module) and simple or dummy versions of the other modules. An operator at the PC (hidden to the user) will start computer actions requested orally by the user, but not yet handled by the software (the bionic wizard-of-Oz, WOZ approach).

Prototype P2 is delivered at the end of month 19 and will be used for full laboratory experiments at first and then in real user homes. The prototype will be used for the final user validation of the HIC and its associated concepts.

This prototype includes full monitor setup including a PC running the client modules and connected to a second PC running the server modules. All modules are present in full form. The prototype is still currently improved according to the laboratory experiments and the user site experiments.

Prototype P3 is the final official version of the HIC, usable for demonstrations and for the possible basis of further exploitation. It corresponds to P2 but with corrections and enhancements made currently with the experiments.