According to the European Union of Deaf, more than 750.000 deaf SL users live in EU, while only 12.000 interpreters are registered. This fact highlights the barriers that deaf and hard-of-hearing (HoH) people are facing, specifically when it comes to communication with hearing people and equal opportunities to proper education. For instance, in Greece and Portugal more than 60% of the deaf children have extremely limited reading and writing skills. Hence, it is no coincidence that deaf children, initially educated in SL, usually stop their studies at the elementary school level. Education as a fundamental right is mentioned not only in the Preamble of the UN Convention on the Rights of Persons with Disabilities (UNCRPD), ratified by the EU as a whole in January 2011, but also in a dedicated article on education (Article 24). Deaf children must be able to become active citizens who are employed and contribute to the society, being taxpayers rather than living on social benefits. Those children obviously need a lot of support at school. Therefore, the educators need to receive appropriate training in Sign Languages, the native languages of the Deaf, so that they are able to communicate with them and to teach them how to develop linguistically. Sign Languages in the curricula of Primary Education University Departments or Departments dealing with Special Education definitely contribute towards this goal. However, it is hard for Higher Education Institutes to find and employ as tutors experts in Sign Language, so many educators are not adequately trained as students to face the challenges of communicating with deaf children in the classroom.
To develop an innovative and affordable system/service for interactive SL teaching for students in Special Education/Pedagogical departments and primary school education. It will be based on computer-vision, machine-learning, linguistic technology, and avatars developed by all the involved partners in several national and EU-funded projects that are ongoing or recently completed. Thus, we aim at an exploitation of research results at European level that would not be possible without the contribution of the EU.
To set up the CAT-SL infrastructure in four Higher Education Institutes (one for every partner), and one primary education one, based on open standards; to enable its practical use by teachers and students.
To develop curricula and guides for teaching SL using the CAT-SL system for at least two multilingual courses, in Greece, Cyprus, Portugal and Netherlands; to support the social inclusion of the deaf children.
To actively disseminate the project results through the promotion of workshops with stakeholders in the EU to raise awareness on the main challenges faced by the Deaf.
The project aims to serve different user groups. The users will be required to have a user account for system access. There will be an opportunity for guest accounts for users who wish to verify a limited amount of the system functionality. Target groups are identified in the following:
The students of Special Education university departments, aiming to learn how to use and teach SLs to deaf children. The students will be able to access educational material asynchronously. The students will also be able to access educational material with synchronous feedback.
The primary school pupils who aim to learn how to use the SLs. Occasionally some of them may be deaf. The pupils are able to access educational material asynchronously. They are also able to access educational material with synchronous feedback.
The course administrators, who may be university professors teaching SLs in Special Education departments, as well as educators in public or private organizations involved in SL teaching or Deaf education. The course administrators will be able to access educational material asynchronously, introduce new material, edit previous material, define exams, etc. They will be given access to visual feedback mechanisms through the avatar.
The teachers, who may be university professors or educators, as in the previous case. However they have a restricted role, as they don’t have permission to add or modify the course material and exams. They are allowed to use existing material, assign exams, and collect exam results.
The system administrators, who will be able to set up and maintain the system.
SL courses are essential for deaf children, since it is their primary communication form. The universal lack of SL tutors may undermine the quality of the provided courses. Therefore the CAT-SL aims to complement the practice sessions of related curricula with automated exercises to be performed by students and automatically evaluated by the system. Therefore the needs to be covered are:
An automated system for interactive SL teaching in three national languages, which will be affordable and easy to deploy and maintain.
Related curricula in the SL courses conforming to EU-wide standards in primary and HE.
A community to adopt and support the related technology and motivate further technological developments using open standards.
The education on SLs is quiet fragmented over Europe. There are different groups focusing on SL linguistics, SL didactics, education of the Deaf, others focusing on the related technologies. However it is hard for single national groups to encompass all the related know-how and to engage the culturally diverse local deaf communities. The problem is not of local scale since there are many national SLs and respective users. A more holistic approach is necessary to develop a system and IOs conforming to EU-wide standards. In CAT-SL we gather together five partners with strong ties to local Deaf communities, significant track-record in the development of methods for SL analysis and education as proved by recent EU-funded projects and related publications. In CAT-SL we aim to leverage the acquired knowledge from ten different projects and disseminate the results across the EU and globally.
The results during the project and its completion are:
A prototype system that will be able to interactively teach students the morphology, syntax and semantics of SLs. The system will be able to :
Present a learning environment to the student.
Present SL concepts in morphology, syntax, semantics.
Prompt specific actions to the students, for execution using a specifically designed and customized avatar.
Acquire video of the student signing using RGB and depth cameras (e.g. kinect, realsense etc).
Recognize to what extend the uttered phrases obey the SL morphology (handshapes, hand motion, facial expressions, overall synchronization) by using machine learning software to track human motion.
Analyze the uttered phrases for syntactic and semantic meaning.
Highlight possible errors and give feedback for better execution using the avatar (concerning hand trajectory, hand shape, orientation, as well as facial expressions (eyebrows, mouth, eyes)).
The system will be running on a single PC with a GPU support and will be connected to a sensor including RGB and depth video or could be connected to the cloud and run as a service. The system will be developed using open standards and will be available to the research community for further development. It is among our goals to attract the interest of coding and education community and make the system sustainable.
Intellectual outputs in the form of
Analysis of Sign Language Teaching Methods and Challenges for an Automated Teaching System. It is a reseach study that will extensively analyze the requirements to incorporate specific teaching curricula, to capturing user body motion and level of detail (e.g., hand/body/head motion, facial expressions etc.), and on how to display feedback to the users. It will be made public to the stakeholders and developers aiming to implement similar services.
The CAT-SL Automated Sign Language Teaching System and Service for European SLs, which is expected to significantly affect the SL teaching paradigm. We will develop a standalone system, as well as a server-based learning service. The development will be based on the analysis given in (a). One-by-one the user requirements will be mapped to design requirements. Then the design requirements will be mapped to software modules. The software modules will be integrated into a system. The user will be able to access the service by connecting a depth sensor or a color web camera.
We will develop interactive material two cover at least three different SL courses in three different languages: Dutch, Greek, Portugese. These will include theoretical background and practical exercises providing feedback in an interactive way via a specifically developed avatar. These are intended to SL learners (university special education students, primary schools pupils).
We will develop multilingual teachers' guides (including books and videos) to assist tutors in using the system for most efficient use.
One seminar and training on the use of the system to educators and technical staff that will use the system. There will be participants from all partners. The main purposes of these training activities are to:
provide the necessary knowledge and practice to the staff who will use the CAT-SL platform to use and maintain the system;
extend the curricula already available in the partners-national sign languages with new vocabulary and test cases;
provide a general basic overview of sign languages and communication challenges faced by deaf people;
disseminate the project and attract relevant institutions from local deaf communities to become associated partners.
After the training sessions the participants will be able to:
adapt CAT-SL to new languages;
add new curricula;
operate the platform and understand how the different components integrate.
Four multiplier events targeting the deaf community, the academia - teachers, staff and students. The goal is to promote the CAT-SL approach to inclusive education for the Deaf, that is, the use of an automated tutor as an effective channel of communication between deaf and non-deaf and its implementation in higher education. We will bring together the academic and the deaf communities to raise awareness and discuss the potential of the proposed approach to promote the inclusion of the deaf in higher and primary education, i.e., using the CAT-SL service as a simple and effective way to train SL teachers and pupils and help overcome the communication gap between deaf and nondeaf in education. We will use the course materials to introduce the CAT-SL tools.
Two papers in scientific conferences to disseminate the results in the scientific community. The conferences can be in computer vision/machine learning community such as ECCV, ICIP, ISVC, AIAI or the education community such as EDUCON, EDUNINE, TLE. The costs will be covered by Project Management and Implementation.
Intellectual Output 1 (O1)
The research study will extensively analyze:
Requirements to incorporate specific teaching curricula based on practices followed in four countries and at EU level.
Requirements on capturing user body motion and level of detail (e.g., hand/body/head motion, facial expressions etc.). The morphological, syntactic and semantic rules of the SLs will be extensively considered and encoded.
Requirements on how to display feedback to the users, e.g., how should the avatar visualize the body motion, how should it highlight errors, how should it evaluate the students.
Incorporation of serious games to offer a more playful environment, especially for the younger users.
Investigation of the challenges in the above process as well the challenges for the teacher - student communication.
Needs: The education in Sign Languages is a challenging task. Sign Languages are complete and complex languages, with all the peculiarities and subtleties of a spoken language. Like all languages, Sign Languages are not mastered easily beyond a basic level. Mastery requires extensive exposure and practice. In this report we will analyze the current practices in teaching in various levels and how these can be incorporated into an automated teaching system.
Innovation: The IO is innovative because as far as we know, there is no publicly available study that associates the SL structure to teaching systems that exploit this structure to offer learning services. The impementation of an SL educational interactive service is very recent idea and therefore the analysis proposed in the IO is innovative as well.
Target groups: Educators who want to delve into teaching methods in SLs. Developers that wish to offer innovative educational services. Learners that want to seek deeper the struturing elements of
Impact: The report will be a valuable asset and is expected to pave the way:
for the consortium to help map the user requirements to design decisions and develop the teaching system
for educators involved with special education for the Deaf and seeking new educational services
for developers aspiring to implement accessible and inclusive educational services for the Deaf
for the Deaf community and those members that aim to contribute with requirements into the development of new educational services
Transferrability: Every educator, developer, aspiring to create a service for SL teaching in the future may use as baseline the report of O1. As mentioned, the analysis will be publicly available to every interested individual or organization. Our researchers will be available via the project web portal to provide additional information on the methods used, conclusions or guidelines.
Intellectual Output 2 (O2)
The CAT-SL Automated Sign Language Teaching System for European SLs is proposed to be developed here. We will develop a server-based learning service. The same system will be available as stand-alone as well. The development will be based on the analysis given in O1. One-by-one the user requirements will be mapped to design requirements. Then the design requirements will be mapped to software modules. The software modules will be integrated into a system to offer an educational service for learning SLs. The user will be able to access the service by connecting a depth sensor or a color web camera.
The needs: The educators need to receive appropriate training in Sign Languages, the native languages of the Deaf, so that they are able to communicate with them and to teach them how to develop linguistically. Sign Languages in the curricula of Primary Education University Departments or Departments dealing with Special Education are very useful and contribute towards this goal. However, it is hard for Higher Education Institutes to find and employ as tutors experts in Sign Language, so many educators are not adequately trained as students to face the challenges of communicating with deaf children in the classroom. Similar considerations apply for teachers of SLs in special education schools.
Target groups: SL teachers, SL learners, Software system developers.
Impact: The system is expected to facilitate the teaching process of SL courses in Higher Education institutes, in special education schools, but also to help individuals interested in learning or mastering their SL skills. Today it is almost impossible to find a tutor in a SL different from the official local SL, however this can be partially handled by CAT-SL, since it will be able to cover multiple languages. In the long term this is expected to narrow the communication and cultural gap with the Deaf community and contribute towards a more inclusive society. The approach can be extended to more SLs in the EU and globally.
Transferrability potential: The service will be implemented for a small number of languages (English, Greek, Portugese). However, the same methodology will be applicable to several more languages in Europe and beyond. This will be achieved by focusing on a common approach, without making our approach dependable on a specific language. Furthermore, through the CAT-SL paradigm it will be possible to author more courses in various educational levels and for various proficiency levels in each SL.
Intellectual Output 3 (O3)
The Sign Language Learning Curricula are proposed to be developed here. We will develop interactive material two cover at least two SL courses in three different languages: Greek (including Cypriot dialect), Portugese and Dutch. The courses may be of different focus: one for students of Special education for the Deaf (higher education), one for deaf children (primary or secondary education). The development will be based on the analysis given in O1. One-by-one the user requirements will be mapped to design requirements. Then the design requirements will be mapped to inetractive course material. The material modules will be integrated into the CAT-SL system and service. The user will be able to access the material by connecting to the CAT-SL platform based on moodle. The learning curricula will include theoretical background and practical exercises in the form of serious games. Lessons can consist of concepts, types of words or signs. Such as:finger spelling: A to Z and numbers, signs: People, Objects, Places, etc, communication scenarios (greetings, friendly conversation, hotel, train station, doctor, bank etc.).
The lesson structure may have the following parts:
Avatar to show handshapes and hand motion.
Use games to practice (e.g., pairing a handshape with a corresponding letter).
Students practice scenarios with real-time feedback via the avatar.
At the end of the lesson, students test their knowledge with a quiz. A summary shows what the user still needs to work on.
The presentation style will differ depending on the audience (e.g., more playful for pupils).
The needs: The educators need to receive appropriate training in Sign Languages, the native languages of the Deaf, so that they are able to communicate with them and to teach them how to develop linguistically. Sign Languages in the curricula of Primary Education University Departments, or schools providing Special Education are very useful and contribute towards this goal. Similar courses can be adapted to other education levels in the future. However, it is hard to find multilingual material, which can be used for SL courses and that can offer credits.
Target groups: Educators of special education programmes, SL teachers or SL learners in general, Native users of SL aiming to expand their knowledge on other SLs, Researchers in SLs and Pupils in primary/secondary education.
Impact: The availability of interactive SL courses is expected to facilitate the teaching process of SL courses in Higher Education institutes, in SL schools but also to help individuals interested in learning or mastering their SL skills. It can be adapted and used for courses in any education level. In the long term this is expected to narrow the communication and cultural gap with the Deaf community and contribute towards a more inclusive society.
Transferrability potential: The curriculum will encompass three EU sign languages (Greek, Portuguese, Dutch) in higher/primary education. However, the same methodology will be applicable to several more languages in Europe and beyond. This will be achieved by focusing on a common approach, without making our approach dependable on a specific language. The courses will be publicly available in the CAT-SL portal and open to contributions. The course will also be available as a MOOC for individual study or to be used in face-to-face courses organized by a teacher.
Intellectual Output 4 (O4)
The teaching guide for the CAT-SL Automated Sign Language Teaching System is proposed to be developed here. The guide will be available in four languages (English, Greek, Dutch, Portuguese) and will be a document also referring to videos.
The needs: The educators need to have a guide on how to use the system effectively. Issues to clarify are how to make an account, how to conduct a lesson, how to use system feedback, how to engage the students, how to define the curriculum, how to fix various system problems, how to provide feedback to developers, how to acquaint themselves with the Deaf culture. The guide needs to be multilingual.
Target groups: SL teachers, Software system developers.
Impact: The teachers' guide is expected to facilitate the teaching process of SL courses in Higher Education institutes, in SL schools etc. Teachers will be able to learn faster how to use the system/service efficiently in their native language, since it will be able to cover multiple languages. In the long term this is expected to narrow the communication and cultural gap with the Deaf community and contribute towards a more inclusive society.
Transferrability potential: The guide will be implemented for a small number of languages (Dutch, Greek, Portugese). However, the same methodology will be applicable to several more languages in Europe and beyond. This will be achieved by focusing on a common approach, without making our approach dependable on a specific language.
The final Intellectual Outputs of our project are detailed below. Each output is a culmination of extensive research and development aimed at enhancing Sign Language education across Europe. We invite educators, developers, and learners to explore and utilize these resources:
The Signal Processing & Communications (SPC) Lab is a Laboratory of the Computer Engineering & Informatics Department (CEID), University of Patras. The SPC Lab presents a research activity in a variety of fields, such as: Signal Processing for Communications, Sensor Networks, System Identification etc.
The Deaf Studies Unit (DSU) participates in both the above projects (as partner in T1EDK-01299, jointly with SPC in T2EDK-00982). The Deaf Studies Unit (DSU) investigates the linguistics of the Greek Sign Language, and produces educational material on GSL and multimedia software for the deaf.
The Hellenic Mediterranean University is highly research-oriented in every scientific field covered by its departments and its research laboratories; therefore, it is an Institute that holds a top position in university research rankings and it is recognized for its contribution in the development of the island of Crete and the country in general. The academic and research staff of the Hellenic Mediterranean University is involved in many scientific activities in collaboration with na-tional and international research groups and teams which have led, on the one hand, to important research results and, on the other hand, to the development of technological innovations in a wide range of research areas and innovative products with international recognition.
Kentalis is a national organisation in the Netherlands, specialized in providing diagnostic, care and education to persons who are deaf, hard-of-hearing or deafblind, and to those who have a language impairment. Kentalis' expertise is focused on bridging communication barriers. Royal Dutch Kentalis is a merger organisation established in 2009, consisting of 3 predecessors. Teachers, psychologists, speech and language therapists, sign language specialists and many others work in disciplinary teams and together cater for the needs of our pupils and clients. Kentalis offers day-care and residential care to children, adolescents and young adults.
CUT was established by law in 2004 and enrolled its first students in 2007. CUT’s involvement in research has been recognized from the fact that within the last four years, more than 10M Euro projects have been funded. In this project, CUT participates through its Social Computing Research Center (SCRC), a multidisciplinary center aiming to advance research on various aspects of Social Computing. Recently, a data center was built with more than 20 racks and 400 blade servers, equipped with state-of-the-art NVIDIA GPUs for deep learning applications. CUT’s research facilities are fully independent from other beneficiaries and/or partner organisations.
The Polytechnic of Porto, P. Porto is a public higher education institution created in 1985. With over 18500 students is the biggest Polytechnic of Portugal. P. Porto stands out mainly for its ability to instruct young people with the knowledge necessary for the working market, able to contribute to economical development with a high sense of social responsibility. This institution stands out for its interdisciplinary with 7 Schools that cover a wide spectrum of scientific knowledge, from Engineering (ISEP) to Music and Performing Arts, Education, Accounting and Administration, Management and Industrial Studies, Management and Technology and Allied Health Technology. P. Porto has more than 20 R&D groups, being the institution of the Polytechnic System with more R&D units recognized and funded by the Portuguese Science and Technology Foundation and occupies the first position in the ranking of polytechnics.
The project is coordinated by the University of Patras,with Associate Professor Dimitrios Kosmopoulos serving as scientific director- Homepage.
The European Commission support for the production of this publication does not constitute an endorsement of the contents which reflects the views only of the authors, and the Commission cannot be held responsible for any use which may be made of the information contained therein.
