Download the SignGuide App
The object of the project is the development of a standard interactive tour system in a museum for the deaf visitors using mobile devices that will be able to answer visitors' questions on the parent sign language (SL) in relation to the exhibits, and provide additional content using avatar or video also in semantics utilizing techniques from the field of computer vision, machine learning and graphics. Final desired result is the increase of the satisfaction of the deaf visitors and in the long run the better social integration through better dissemination of cultural content, as well as increasing museum attendance by members of that group. SignGuide is expected to help museums and cultural organizations harmonize in the long run their services provided by European legislation obliging them to provide equal access to vulnerable social groups.
The SignGuide project is implemented within the RESEARCH - CREATE - INNOVATE Action. The main objective of the Action is to connect research and innovation with entrepreneurship and to strengthen the competitiveness, productivity and extroversion of businesses towards international markets, with the aim of transitioning to innovative entrepreneurship and increasing domestic added value.
G1: Implementation of a service for mobile devices that will recognize the user's question with SL about the report.
G2: Implementation of a service that will analyze the previous question and will recall in response additional content in SL.
G3: Development of accompanying material in such a way as to support retrieval based on free text questions.
G4: Development of an avatar for the presentation of the additional material through SL.
G5: The development of a standard tour system for deaf museum visitors, which will be able to penetrate dynamically in the global market of mobile cultural applications, due to its uniqueness in order to increase visitor satisfaction, the increase in traffic and the low cost of installation and operation.
G6: The development of a museum tour system for the deaf that will take into account their needs and improve their quality of life. Satisfaction of these objectives will be documented during the trial period with a significant increase in satisfaction and duration stay at the museum e.g. by > 50%.
G7: Dissemination of the results of the project with the aim of advertising the benefits and with the ultimate goal of improving the experience of visiting a museum and better integration of deaf people. The target groups are: Museum administration and curators, deaf visitors and deaf associations, general public, researchers, companies in the field of culture.
Objectives will be technically achievable through (a) computer vision and machine learning algorithms for question recognition (b) machine learning to relate the questions to the content and (c) graphical algorithms to implement the virtual guide. The system will be used at the Archaeological Museum of Thessaloniki (AMTH). The integration and commercial operation will be done through BioAssist. The development of the monitoring system and the virtual tour guide will be developed by the Institute of Technology and Research and the Computational Vision laboratories and Robotics (FORTH-EVOR) and Human-Computer Interaction (FORTH-ERAW) respectively. The users will come from the Deaf Education Unit of the University of Patras (PP-MAK). The Signal and Telecommunications Processing Laboratory (PP-EEST) will undertake the machine learning. The Laboratory of Cultural Technology of the University of Patras (PP-EPT) will organize the content and coordinate the project.
Hybrid Scientific Workshop "Inclusive Museums - Museum Experiences for the Hearing Impaired"
Second in-person plenary meeting
Completion of GSL question dataset recording
Workshop on accessibility
Sartinas, E.G., Psarakis, E.Z. and Kosmopoulos, D.I., 2023. Motion-Based Sign Language Video Summarization using Curvature and Torsion . arXiv preprint arXiv:2305.16801.
Pikoulis, E.-V., Bifis. A., Arvanitis, N., and Kosmopoulos, D. Kosmopoulos, 2023, September. Automatic Sign Language sentence recognition in the context of guided museum tours for deaf and hard-of-hearing visitors. AMID Workshop, The ACM International Conference on Mobile Human-Computer Interaction (MobileHCI2023).
Drosakis, D. and Argyros, A., 2023, July. 3D Hand Shape and Pose Estimation based on 2D Hand Keypoints. In Proceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments (pp. 148-153).
Arvanitis, N., Sartinas, E. and Kosmopoulos, D., 2023, June. Procrustes-DTW: Dynamic Time Warping Variant for the Recognition of Sign Language Utterances. In 2023 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW) (pp. 1-5). IEEE.
Koulouris, D., Trastelis, F., Menychtas, A., Kosmopoulos, D. and Maglogiannis, I., 2023, June. Utilizing AR and Hybrid Cloud-Edge Platforms for Improving Accessibility in Exhibition Areas. In IFIP International Conference on Artificial Intelligence Applications and Innovations (pp. 171-182). Cham: Springer Nature Switzerland.
K. Papavassileiou, D. I. Kosmopoulos, and G. Owens. 2023. A generative model for the Mycenaean Linear B script and its application in infilling text from ancient tablets. ACM Journal on Computing and Cultural Heritage, 16(3), pp.1-25.
Vasilikopoulos, N., Kolotouros, N., Tsoli, A. and Argyros, A., 2023, April. TAPE: Temporal Attention-based Probabilistic human pose and shape Estimation. In Scandinavian Conference on Image Analysis (pp. 418-431). Cham: Springer Nature Switzerland.
D.I. Kosmopoulos, C. Constantinopoulos, M. Trigka, D. Papazachariou, K. Antzakas, V. Lampropoulou, A. Argyros, I. Oikonomidis, A. Roussos, N. Partarakis, G. Papagiannakis, K. Grigoriadis, A. Koukouvou, A. Moneda. 2022, June. Museum Guidance in Sign Language: the SignGuide project. In Proceedings of the 15th International Conference on PErvasive Technologies Related to Assistive Environments (pp. 646-652). DOI
Tze, C.O., Filntisis, P.P., Roussos, A. and Maragos, P. 2022, June. Cartoonized Anonymization of Sign Language Videos. In 2022 IEEE 14th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP) (pp. 1-5). IEEE. DOI
Tze, C.O., Filntisis, P.P., Dimou, A.L., Roussos, A. and Maragos, P. 2022. Neural Sign Reenactor: Deep Photorealistic Sign Language Retargeting. arXiv preprint arXiv:2209.01470
K. Blekos and D. I. Kosmopoulos. 2022. Quantum Data Reduction with Application to Video Classification. In 2022 IEEE/ACM 7th Symposium on Edge Computing (SEC) (pp. 430-435). IEEE.
Karuzaki, E., Partarakis, N., Patsiouras, N., Zidianakis, E., Katzourakis, A., Pattakos, A., Kaplanidi, D., Baka, E., Cadi, N., Magnenat-Thalmann, N. and Ringas, C., 2021. Realistic virtual humans for cultural heritage applications. Heritage, 4(4), pp.4148-4171.
Blekos, K. and Kosmopoulos, D., 2021. A Quantum 3D Convolutional Neural Network with Application in Video Classification. In Advances in Visual Computing: 16th International Symposium, ISVC 2021, Virtual Event, October 4-6, 2021, Proceedings, Part I (pp. 601-612). Springer International Publishing.
The Signal Processing & Communications (SPC) Lab is a Laboratory of the Computer Engineering & Informatics Department (CEID), University of Patras. The SPC Lab presents a research activity in a variety of fields, such as: Signal Processing for Communications, Sensor Networks, System Identification etc.
Institute of Technology and Research - Institute of Informatics (Laboratory of Computational Vision and Robotics, Human-Computer Interaction Lab)
The Archaeological Museum of Thessaloniki is one of the largest museums in the country and the central museum of Northern Greece. With unique collections of ancient objects, with rich activity and extroversion, it is always ready to welcome its visitors..
Founded in 2013, BioAssist is the first Greek ICT services company created for seniors and chronic patients. You only need an Internet connection and our tools will save you time and money by keeping your caring circle into the loop. We believe that mobile Health is the most useful chapter within the book of Internet of Things. Access to health monitoring should be simple, smart, and friendly.
The Deaf Studies Unit (DSU) belongs to the Department of Primary Education of the University of Patras, and is a leading center for Deaf Studies, Greek Sign language (ENG), and Deaf Education in Greece. Established in 1994, is the only academic institution in Greece that provides research in the area of deafness, and programs for teachers of the Deaf, Sign Language tutors, and interpreters.
