Automatic Guided Tour of a Museum Using Sign Language

About the project

The object of the project is the development of a standard interactive tour system in a museum for the deaf visitors using mobile devices that will be able to answer visitors' questions on the parent sign language (SL) in relation to the exhibits, and provide additional content using avatar or video also in semantics utilizing techniques from the field of computer vision, machine learning and graphics. Final desired result is the increase of the satisfaction of the deaf visitors and in the long run the better social integration through better dissemination of cultural content, as well as increasing museum attendance by members of that group. SignGuide is expected to help museums and cultural organizations harmonize in the long run their services provided by European legislation obliging them to provide equal access to vulnerable social groups.

Technical golas:

G1: Implementation of a service for mobile devices that will recognize the user's question with SL about the report.

G2: Implementation of a service that will analyze the previous question and will recall in response additional content in SL.

G3: Development of accompanying material in such a way as to support retrieval based on free text questions.

G4: Development of an avatar for the presentation of the additional material through SL.

Financial goals:

G5: The development of a standard tour system for deaf museum visitors, which will be able to penetrate dynamically in the global market of mobile cultural applications, due to its uniqueness in order to increase visitor satisfaction, the increase in traffic and the low cost of installation and operation.

Operational goal:

G6: The development of a museum tour system for the deaf that will take into account their needs and improve their quality of life. Satisfaction of these objectives will be documented during the trial period with a significant increase in satisfaction and duration stay at the museum e.g. by > 50%.

Dissemination goals:

G7: Dissemination of the results of the project with the aim of advertising the benefits and with the ultimate goal of improving the experience of visiting a museum and better integration of deaf people. The target groups are: Museum administration and curators, deaf visitors and deaf associations, general public, researchers, companies in the field of culture.

Objectives will be technically achievable through (a) computer vision and machine learning algorithms for question recognition (b) machine learning to relate the questions to the content and (c) graphical algorithms to implement the virtual guide. The system will be used at the Archaeological Museum of Thessaloniki (AMTH). The integration and commercial operation will be done through BioAssist. The development of the monitoring system and the virtual tour guide will be developed by the Institute of Technology and Research and the Computational Vision laboratories and Robotics (FORTH-EVOR) and Human-Computer Interaction (FORTH-ERAW) respectively. The users will come from the Deaf Education Unit of the University of Patras (PP-MAK). The Signal and Telecommunications Processing Laboratory (PP-EEST) will undertake the machine learning. The Laboratory of Cultural Technology of the University of Patras (PP-EPT) will organize the content and coordinate the project.



Responsive image

The Signal Processing & Communications (SPC) Lab is a Laboratory of the Computer Engineering & Informatics Department (CEID), University of Patras. The SPC Lab presents a research activity in a variety of fields, such as: Signal Processing for Communications, Sensor Networks, System Identification etc.

Responsive image

Institute of Technology and Research - Institute of Informatics (Laboratory of Computational Vision and Robotics, Human-Computer Interaction Lab)

Responsive image

The Archaeological Museum of Thessaloniki is one of the largest museums in the country and the central museum of Northern Greece. With unique collections of ancient objects, with rich activity and extroversion, it is always ready to welcome its visitors.

Responsive image

Founded in 2013, BioAssist is the first Greek ICT services company created for seniors and chronic patients. You only need an Internet connection and our tools will save you time and money by keeping your caring circle into the loop. We believe that mobile Health is the most useful chapter within the book of Internet of Things. Access to health monitoring should be simple, smart, and friendly.

Responsive image

The Deaf Studies Unit (DSU) belongs to the Department of Primary Education of the University of Patras, and is a leading center for Deaf Studies, Greek Sign language (ENG), and Deaf Education in Greece. Established in 1994, is the only academic institution in Greece that provides research in the area of deafness, and programs for teachers of the deaf, sign language tutors, and interpreters.

The project is coordinated by the University of Patras, with Associate Professor Dimitrios Kosmopoulos serving as scientific director- Homepage)

Contact us

Please direct your enquiries and communication using our contact form:
Co-financed by the European Union and Greek national funds through the Operational ProgramCompetitiveness, Entrepreneurship and Innovation, under the call RESEARCH – CREATE - INNOVATE (project code: Τ1EDK-01299).
Responsive image