Kinect-sign: Teaching sign language to “listeners” through a game

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

11 Citations (Scopus)


Sign language is the hearing impaired form of communicating with other people, including listeners. Most cases, impaired people have learned sign language form childhood. The problem arises when a listener comes in contact with an impaired person. For instances, if a couple has a child which is impaired, the parents find a challenge to learn the sign language. In this article, a new playful approach to assist the listeners to learn sign language is proposed. This proposal is a serious game composed of two modes: School-mode and Competition-mode. The first offers a virtual school where the user learns to sign letters and the second offers an environment towards applying the learned letters. Behind the scenes, the proposal contains a sign language recognition system, based on three modules: 1 – the standardization of the Kinect depth camera data; 2 – a gesture library relying on the standardized data; and 3 – the real-time recognition of gestures. A prototype was developed – Kinect-Sign – and tested in a Portuguese Sign-Language school and on eNTERFACE’13 resulting in a joyful acceptance of the approach.

Original languageEnglish
Title of host publicationInnovative and Creative Developments in Multimodal Interaction Systems - 9th IFIPWG 5.5 International Summer Workshop on Multimodal Interfaces, eNTERFACE 2013, Proceedings
EditorsYves Rybarczyk, Tiago Cardoso, João Rosas, Luis M. Camarinha-Matos
Place of PublicationBerlin, Heidelberg
PublisherSpringer Verlag
Number of pages19
ISBN (Electronic)978-3-642-55143-7
ISBN (Print)978-3-662-52571-5
Publication statusPublished - 1 Jan 2014
Event9th IFIPWG 5.5 International Summer Workshop on Multimodal Interfaces, eNTERFACE 2013 - Lisbon, Portugal
Duration: 15 Jul 20139 Aug 2013

Publication series

NameIFIP Advances in Information and Communication Technology
PublisherSpringer Verlag
ISSN (Print)1868-4238


Conference9th IFIPWG 5.5 International Summer Workshop on Multimodal Interfaces, eNTERFACE 2013


  • Gesture Recognition
  • Kinect Sensor
  • Serious Game
  • Sign Language


Dive into the research topics of 'Kinect-sign: Teaching sign language to “listeners” through a game'. Together they form a unique fingerprint.

Cite this