Semantics language analysis on mobile devices

for interpretation with emphasis on health services

About the project

Sign Languages (SLs) are the main means of communication for deaf people. The access to SL is essential for the fulfillment of basic Human Rights, however there is a shortage of interpreters, which undermines these rights and often endangers the lives of the deaf, especially in cases of emergency or serious health incidents.

HealthSign proposes to develop an application for the automated interpretation of the Greek Sign Language (GSL) over internet with focus on the health services, which are the most common reason to seek for an interpreter. The high demand for interpreters is often not met or requires long waiting. Furthermore, the availability of interpreters facilitates the integration of the deaf community.

Vision is probably the only sensor modality that could be of practical use because (a) only vision can capture manual and non-manual cues, which provide essential information for SLR, (b) camera-equipped hand-held devices with powerful processors are a commodity nowadays and (c) recent advances in computer vision and machine learning render mainstream visual SLR a realistic option.

Technical goals:

G1: Develop a database of GSL from native speakers with emphasis on health services

G2: Implement an internet-based platform for synchronous communication and interpretation with health professionals

G3: Develop in parallel a lightweight version which will be able to run on an embedded platform.

Research goals:

G4: Develop algorithms for recognition of SLs, using computer vision and deep learning using the hand and body/facial cues

G5: Implement the algorithms on embedded platforms using FPGAs

Financial goals:

The main goal is the viability of the proposed application which will be achieved by (a) simple off-the-shelf equipment for the users (b) efficient implementation of the proposed algorithms (c) simple installation (d) development of a business plan to facilitate the longevity of the proposed product.

Publications

Consortium

Image

The Signal Processing & Communications (SPC) Lab is a Laboratory of the Computer Engineering & Informatics Department (CEID), University of Patras. The SPC Lab presents a research activity in a variety of fields, such as: Signal Processing for Communications, Sensor Networks, System Identification etc.

Image

The Institute of Computer Science has had a relatively long history and recognised tradition, since its establishment in 1983, in conducting basic and applied research, developing applications and products, providing services, and playing a leading role in Greece and internationally, in the fields of Information and Communication Technologies.

Image

​IRIDA Labs (Patras, Greece) is bridging the gap between a camera and a human eye by bringing visual perception to any device. They develop software in Computer Vision, using Image Processing and Machine Learning techniques made for any CPU, GPU or DSP/ASP platform or a combination of them using Heterogeneous Programming techniques.

Image

Founded in 2013, BioAssist is the first Greek ICT services company created for seniors and chronic patients. The company's goal is to research, design and study remote monitoring, health support and well-being systems.

Image

The Deaf Studies Unit (DSU) belongs to the Department of Primary Education of the University of Patras, and is a leading center for Deaf Studies, Greek Sign language (ENG), and Deaf Education in Greece. Established in 1994, is the only academic institution in Greece that provides research in the area of deafness, and programs for teachers of the deaf, sign language tutors, and interpreters.


The project is coordinated by the University of Patras,
with Associate Professor Dimitrios Kosmopoulos serving as scientific director.

News

Contact Us


Please direct your enquiries and communication using our contact form:

Co‐financed by the European Union and Greek national funds through the Operational ProgramCompetitiveness, Entrepreneurship and Innovation, under the call RESEARCH – CREATE - INNOVATE (project code: Τ1EDK-01299)