Dr. Debasis Samanta, IIT Kharagpur | Multimodal Interaction

Objectives

Support Internet access to underprivileged communities in India in their mother language. In recent years, with the exponential growth of Information Technology (IT), Internet has spread almost in all applications. However, the Internet advantages are confined to the limited communities only in our country. The majority communities, who have some inherent limitations such as illiteracy, unfamiliarity with foreign languages, cyber ignorance, low-bandwidth communication links etc. are still unable to participate in the IT revolution. This project would give a solution that transcend these barriers and bring Internet services to everyone.
Design and development of Indian languages (Hindi and Bengali) compatible Internet Browser. Currently Internet becomes synonymous with World Wide Web (WWW). The popularity of WWW is largely dependent on the search engines. Search engines are the access point of the huge information repository at the internet. The user interface of the existing search engines takes a query string as an input from the user. Based on the keywords present in the query string they search for relevant documents. This type of interface works fine for well literate people. But for illiterate people or those who has no enough proficiency and expertise to construct a good query string cannot use this. The rural communities in India are a perfect example of this category of users. Hardly, they have the proficiency of constructing a good query strings in English to extract their information from WWW. To serve this type of users, the project would design and develop a user interface, so that users can place their queries in Hindi and Bengali languages and get their information retrieve in Hindi and Bengal languages. Based on the know-how developed in the proposed project, other major Indian languages can be adopted in the system.
Providing multi-modal interactions facilities for user-friendly interaction to Internet, so that people of varieties level of experience can enjoy the system. When backward community comes into consideration, it happens that some of them are illiterate and hence cannot read/write. Further, some of them are novices to the computer. For such users, standard text-based input is not effective. For those people, service can be provided following speech technology, and theory of human-computer interaction (HCI). The proposed project would develop methodology for three modes of interactions: 1) spoken input 2) iconic input and 3) dialog-based input and 4) email-based query retrieval. The output would be in the form of native languages, speech and visual forms.

The Project

The aim of the proposed work would be to develop a gateway to the vast information available in WWW for rural people. An integrated system will be developed so that a user can interact with the Internet through the knowledge of their mother languages only. User can pose their queries in their mother languages as texts, through their spoken languages, with the help of visual displays like icons, dialogue boxes etc. Further, a user can post his query as an email and get retrieved their results as email replies. The later mode is an off-line interaction and best suitable when the Internet is connected through a low-bandwidth communication line. The basic architecture of the proposed system is shown in the following Figure.

The entire project has two main components. A use case diagram of the propose system is shown in the following Figure.

Use-case (context) diagram of the proposed system

Indian languages compatible user interface

The first component would be to develop user interface that would be compatible with Indian languages only. Through this interface, a user can pose his query and see the retrieved information in his native script. In this project, only Hindi and Bengali scripts would be included in the system, and following the know-how developed in the project other major Indian languages scripts would be incorporated. The architecture for this component is shown below.

Architecture of the Indian language compatible browser

Multi-modal interaction service provider

The second component would provide facilities to access Internet with four modes of interactions: text, speech, and visual language. This component allows a user to submit his query in any one of the above forms.

Akshar

Text based query is the conventional way of inputting strings of words but in the proposed system it would be with Bengali/Hindi language. A language translator would translate the user query into English and keywords would be extracted from that, which would be passed to a search engine. The results/web pages returned by the search engine would be translated into user's own language and then displayed.

Dhwani

The speech-based query retrieval is the same way as the text-based except that here the query would be inputted as a spoken sentence of users. The spoken sentence would be converted into text and the later tasks would be same as the text-based interaction. Further, here the retrieved information would be presented in the form of voice. This speech-based interaction would cater the need of novice users.

Chitra

We propose another style of interaction, which is typically meant for intermittent users, that is the users, who have primarily exposed with the computers. This interaction style is with the visual languages like icons and dialogue box based. A library of icons and dialogue boxes would be designed to promote this kind of service. The icons would be designed based on ontology of information available in the Internet. An icon set would be developed for each topic and concepts within the ontology.

Bharati: Integrated System

The dialogue-based interaction would rely on the concept of user profiles. There would be a set of user classes based on user’s profession, for example, farmer, health worker, teacher, student etc. A user would be mapped into one of the existing user classes. The dialogue would be different for each class of users according to their search of interest. Further, object-action based adaptive interface would be incorporated to give the more effective usages to its users. The following Figure shows the architecture of this component.