Research | Vastspeech

Virtual Reality As A Therapeutic Tool

VR technology is malleable and can be uniquely designed to fit a certain individual’s needs

Project Summary

I'M AN ORIGINAL CATCHPHRASE

Abstract

Autism Spectrum Disorder (ASD) is a neurodevelopmental communication disorder resulting in functional language and behavioral delays affecting over 3.5 million Americans. These delays vary with the severity of symptoms that present in ASD but often result in limited speech and increased communication challenges. Alongside linguistic acquisition, oral motor coordination is a crucial part of speech production. Current clinical techniques have shown varying degrees of efficacy in improving functional language proficiency. Most techniques follow a drill-like procedure, where the child is made to repeat various sounds and phrases until they are retained. However, such a process requires potentially over twenty therapy sessions to show improvement which may then only be focused on one aspect of speech. This significantly limits the linguistic and social skills a student will acquire. To improve the efficacy of these therapy sessions, new technology must be developed to provide the most effective educational experience.

Video-assisted speech technology (VAST)

is a method of using a video of a close-up model of the mouth and speaking simultaneously with it

Rather than present the individual with a static photograph of the initial phoneme, the entire sequence of oral movements can be presented sequentially via video-recorded segments of the orofacial area producing connected speech, combining best practices, video modeling, and literacy with auditory cues to provide unprecedented support the development of vocabulary, word combinations and communication.

VR offers several benefits as a therapy technique: overcoming sensory difficulties, more effectively generalizing information, employing visual learning, and providing individualized treatment. As a user moves through the stages of the program, they will be immersed in a proactive environment where they will engross themselves with continuous content. Rather than present the individual with a static photograph of the initial phoneme, the entire sequence of oral movements can be presented sequentially via VR-modelled segments of the orofacial area producing connected speech, combining best practices, video modeling, music therapy, and literacy with auditory cues to provide unprecedented support the development of vocabulary, word combinations and communication. The innovation will be a video series of a realistic VR mouth which will require the use of an app on a tablet or a smartphone, VR goggles, and bone conduction headphones. The incorporation of the VAST system into special education classrooms will help children with ASD to enhance their functional and social communicative abilities. The VR system will teach them various motor and word production skills which will grow their vocabulary, reading skills, understanding of speech, and functional language usage as a whole. By building upon standard therapy techniques, this product will provide those with ASD an opportunity to enhance their quality of life by increasing their speech abilities which will enable them to build social networks and handle the events of daily life.

In this SBIR Phase I proposal, iTherapy will develop a personalized educational experience for students with ASD by creating a virtual reality (VR) based VAST program to stimulate engagement and speech production practice.

Project Narrative

Nearly 3.5 million Americans are diagnosed with Autistic Spectrum Disorder (ASD), a communication disorder which causes skill limitations in the areas of language acquisition, sensory integration, and behavior. This lack of functional language ability limits conversation to its most basic parts, making daily tasks difficult for minimally to non-verbal individuals to achieve. iTherapy is developing the VAST platform, a personalized educational experience for students with ASD by creating a virtual reality-based video-modeling program to stimulate engagement and speech production practice, ultimately providing those with ASD an opportunity to enhance their quality of life by increasing their speech abilities which will enable them to build social networks and handle the events of daily life.

A. Specific Aims

Autism Spectrum Disorder (ASD) is a neurodevelopmental communication disorder resulting in functional language and behavioral delays affecting over 3.5 million Americans. These delays vary with the severity of symptoms that present in ASD. Individuals may be verbal and possess stronger functional language abilities whereas 25% of people in the ASD population are minimally verbal or non-verbaliii, having increased communication challenges. ASD limits many aspects of speech such as vocabulary acquisition where the individual retains a limited lexicon. This restriction on word choice leads to further limitations in the development of sentence structure. Conversation is kept to simple requests and commands and only basic topics are maintained for discussion, preventing the expression of abstract ideas. Alongside linguistic acquisition, oral motor coordination is a crucial part of speech production. Students with ASD struggle with acquiring the necessary motor skills needed for speech articulation and activities such as chewing.

Aim 1: Develop a beta product VR-based VAST platform. Currently, the alpha product VAST program utilizes filmed mouths enunciating words and phrases in a 2D setting. Developers will code the VR mouths and the graphic design for all linguistic elements. Coding will comprise of C++ & Unreal Engine 4 where unreal engine utilizes a node-based language called Blueprints Visual Scripting. Further attributes of Unreal Engine include a skeletal animation system combined with higher polygon models. In addition to VR programming, the Dolch Vocabulary list will be integrated into the system as a base word list consisting of frequently used English words. Within the list are 220 easily recognizable service words including nouns and non-nouns which target the pre-school age up to the third grade. Deliverable: The alpha product 2D software will be developed into a 3D VR environment to ensure maximum optimization of the product for client use.

Aim 2: Develop hardware prototype. We will compare the utility of combining off-the-shelf VR goggles & Bone Conduction (BC) headphones with a 3D printing of a single cohesive unit. The strength of the BC headphones will be tested alongside standard headphones to determine if they provide optimal functionality. After testing each option, the product with the highest rate of efficacy will be used to create the integrated prototype. Deliverable: Functional hardware prototype.

Aim 3: Demonstrate feasibility of VR-based VAST platform. We will pilot test the 3D VR VAST app with 30 students with ASD. The recruited population will consist of children from ages 5 to 8 who will participate in a 24-session long study across 12 weeks. The students will be divided into two groups: one which receives the VR-integrated prototype, and one that receives the existing 2D application on a tablet. Sessions will be held twice a week with each lasting approximately 30 minutes. Three main metrics will be measured during the pilot study: Articulation Rate (Syllables per Second), a Speech Intelligibility Score (SIS), and Mean Length Utterance (MLU). These metrics will show whether there has been an improvement in the participant’s functional language abilities as well as their oral motor coordination skills. They will be analyzed by several trained staff members who will observe each recorded and monitored session. Hypothesis: Users of the VR integrated VAST application will show improved functional language abilities and oral motor coordination skills.

Upon successful completion of these Aims, students with ASD will be provided with a way to significantly improve their ability to communicate, and thus enhance their quality of life. The proposed 3D VR VAST app will overcome the limitations of current clinical techniques for those with communicative disabilities. By using 3D VR software, VAST will make speech therapy more individualized and approachable. Further Phase II developments will proceed in adding more lessons and terminology so that users will be given successive opportunities to improve their functional language abilities, as well as implementing a large clinical study to demonstrate the clinical validity of the technology.

Current clinical techniques have shown varying degrees of efficacy in improving functional language proficiency. Most techniques follow a drill-like procedure, where the child is made to repeat various sounds and phrases until they are retained. However, such a process requires potentially over twenty therapy sessions to show improvement which may then only be focused on one aspect of speechiv. This significantly limits the linguistic and social skills a student will acquire. To improve the efficacy of these therapy sessions, new technology must be developed to provide the most effective educational experience. Video-assisted speech technology (VAST) is a method of using a video of a close-up model of the mouth and speaking simultaneously with it.

Research Strategy

B. Significance

Nearly 3.5 million Americans are diagnosed with ASDi, a communication disability which causes skill limitations in the areas of language acquisition, sensory integration, and behavior. Although the symptoms of ASD are generalizable, levels of severity manifest differently in each individual. Certain individuals may display a less severe case of ASD and show stronger functional language and behavioral abilities whereas 25% of people have minimal- to non- verbal communication skills. This lack of functional language ability limits conversation to its most basic parts, making daily tasks difficult for minimally to non-verbal individuals to achieve. Therefore, new treatments and clinical tools must provide those with ASD an ability to enhance their functional language and behavioral skills to provide them with a higher quality of life.

Primarily, ASD affects functional communication abilities such as speech intelligibility, vocabulary acquisition, syntax, language comprehension, and pragmatic language development. The vocabulary set that a minimally verbal individual will acquire is often restricted to simple words and phrases. This lexical simplicity prevents students with ASD from expressing complex thoughts and needs, forcing them to only discuss their immediate environmentii. A decrease in vocabulary leads to greater syntactic difficulties as sentences become telegraphic, ranging from two to three words. Further morphosyntactic challenges can involve a lack of advanced grammar such as limited to no use of embedded clauses as well as misuse of pronoun, verb, and noun morphology. Generally, those who have overall limited language abilities with ASD will show patterns of ungrammatical speech and shorter sentences. The restricted speech patterns lead to consistent challenges while participating in daily communicative activities.

Primarily, ASD affects functional communication abilities such as speech intelligibility, vocabulary acquisition, syntax, language comprehension, and pragmatic language development. The vocabulary set that a minimally verbal individual will acquire is often restricted to simple words and phrases. This lexical simplicity prevents students with ASD from expressing complex thoughts and needs, forcing them to only discuss their immediate environment. A decrease in vocabulary leads to greater syntactic difficulties as sentences become telegraphic, ranging from two to three words. Further morphosyntactic challenges can involve a lack of advanced grammar such as limited to no use of embedded clauses as well as misuse of pronoun, verb, and noun morphology. Generally, those who have overall limited language abilities with ASD will show patterns of ungrammatical speech and shorter sentences. The restricted speech patterns lead to consistent challenges while participating in daily communicative activities.

Limitations of Current Therapy

Modern clinical treatments have demonstrated that advancing functional language proficiency in those with ASD is possible and have proven effective in achieving desired results. However, these techniques are time-consuming and can prevent linguistic growth. Classical treatments such as Applied Behavioral Analysis (ABA) use drill-like techniques to enforce a behavior or a category of behaviors over several treatment sessions, possibly approaching over 20 hours of training per week. The specified linguistic behaviors are narrowed down to their simplest forms and trained until the client has mastered their production. Although these repetitive techniques promote memorization, the client may only learn a small word list over the course of several weeks. This restricts the size of their known vocabulary and further limits their conversational abilities. Treatments such as these may also lead to context dependent memorization, where the client only uses the learned words in the clinical setting they are practiced in, or they are unable to apply these new words to other conversations and environments.

Furthermore, since the symptoms of ASD are unique to each individual, certain treatment types may not be useful. This means that clinicians must test therapies on a trial and error basis, which may lead to a continuous stream of costly clinical sessions. Additionally, one-third of individuals who are diagnosed with ASD are also diagnosed with Attention Deficit and Hyperactivity Disorder (ADHD) vi. The symptoms that rise with ADHD can consist of becoming easily distracted; unable to complete specified tasks; and difficulties focusing and sitting still:

It may thus be difficult for an individual with a dual diagnosis of ASD & ADHD to gain significant success from clinical sessions. Repetitive drill-like tasks with no change in environment can cause the client to be restless and frustrated. This can lead to therapists spending more time helping their client focus than providing therapy. To improve the efficacy of these therapy sessions, new technology needs to be used to provide the most effective educational experience for students with ASD.

Virtual Reality As Therapeutic Tool

VR technology is malleable and can be uniquely designed to fit a certain individual’s needs

Both the principles of motor learning and accepted treatment methodology for motor speech programming disorders adhere to the basic tenet that individuals benefit from visual feedback in addition to auditory cueing when attempting to perform the sequential movements necessary for speech production. It is a widely accepted treatment methodology to provide diagrams, photos, or live face-to-face models of phonemes in order to provide cueing to the individual attempting to initiate and self-monitor speech movements. The technique of using static digital photographs of an individual’s own face when correctly producing phonemes has been used successfully by one of the authors with multiple subjects. This imitation of correct posturing has been found to be successful in training motor movements which has resulted in improved accessibility to these phonemes for individuals attempting to overcome apraxia of speech, but it has the limitation of providing limited cueing for production of connected speech.

Modern clinical techniques and technologies must effectively prepare students with ASD to manage their daily communication and consequently their daily lives. Currently, therapies in the realm of neurocognitive rehabilitation are using VR systems to enhance functional and expressive language abilities in those with ASD. Primarily, VR is able to provide a safe environment for individuals with ASD to practice their functional language skillsix. According to Strickland et al., VR provides several benefits as a therapy technique as it overcomes sensory difficulties, more effectively generalizes information, employs visual learning, and can provide individualized treatmentx. Students with ASD may become overwhelmed with an excess of sensory information – such as a person’s eyes, hair, and body movements - during regular clinical sessions which can lead to stress and a lower retention rate in information. In a VR setting, specific stimuli (i.e. a speaking mouth) can be isolated to fit the current needs of the individual using the technology. This eliminates the sensory overload from other potential environmental distractors, allowing the user of the device to only focus on the lesson at hand.

Individuals with ASD are found to have an enhanced ability for visual processing with a strong ability to remember visual detailsxi. These abilities are best emphasized in a VR setting as the system utilizes visual cues and representations to teach selected skills. This technology can be manipulated to provide a focus on certain aspects of what is being learned, and certain images or games can be programmed into this software to utilize the visual strengths in those with ASD to provide a better opportunity for them to more effectively learn the content presented to them. With this, VR technology is malleable and can be uniquely designed to fit a certain individual’s needs. Since ASD is a disorder where each individual has a different set of symptoms, this ability to manipulate the content of VR software will ensure that each person is able to get the most effective method of treatment that they are ablexii.

C. The iTherapy Innovation

Thanks to recent technology (iPad/iPod Touch), a groundbreaking and exciting therapeutic technique to teach non-verbal students to use spoken language has emerged. Video-Assisted Speech Technology (VAST) is a method of using a video of a close-up model of the mouth and speaking simultaneously with the mouth. Rather than present the individual with a static photograph of the initial phoneme, the entire sequence of oral movements can be presented sequentially via video- recorded segments of the orofacial area producing connected speech, combining best practices, video modeling, music therapy, and literacy with auditory cues to provide unprecedented support the development of vocabulary, word combinations and communication.

students watch a video recording of a mouth speaking on the VAST app

The alpha prototype VAST platform utilizes filmed mouths enunciating words and phrases. Videos are organized into a hierarchy of 5 categories beginning with syllables and ending with sentences. Each video gives a spoken target utterance that is preceded by the written words. Each word, phrase, and sentence is concrete and has meaning that can be generalized and practiced throughout the day. Providing written words will prevent a student from labeling a picture of a frog jumping as “go” or a person lying on a mat as “break time”. Furthermore, there is significant research that suggests pairing picture symbols with words may actually increase confusion, especially when they represent abstract concepts, have multiple meanings, or serve more than one grammatical function.

Early trials using the alpha product VAST platform indicate that students are highly interested in VAST videos and will almost immediately attempt lip movements or touch their lips in response to the models. After a few short weeks, students who were essentially non-verbal began word approximations and word attempts more readily. The best and most unexpected therapeutic improvements have been the student’s ability to generalize skills. Students begin attending to the speaker’s oral motor movements during daily communication and continue learning speech in a more traditional, naturalistic manner.

Although the alpha product 2D VAST platform (Figure 1) has proven to be effective in preliminary studies, there are limiting factors that must be recognized. Where a 2D app provides a video of a mouth, 3D video-modeling software would make for a more naturalistic learning experience. As 3D VR software simulates reality, users are able to receive a more detailed image of the action they are expected to replicate. This will provide them with greater clarity and understanding than a standard 2D app. Moreover, 3D software has a higher rate of adjustability. The 2D video-modeling platform consists of pre-made videos of a mouth enunciating words and phrases. Any desired additions must continuously be recorded and then added. This process is slow and is not individualized to the user. 3D software is able to make up for these limitations as the operating system used to make it can be reprogrammed when necessary, and updated versions can be quickly created.

iTherapy's alpha product has been extraordinarily effective with older (18-22) non-verbal students with autism. In two individual cases, students were attempting word approximations and speaking several one-syllable words after one session of watching the VAST videos. One of those students was diagnosed with severe sensory neural hearing loss and autism. He was able to produce four words by the end of his first session.

In this Phase I proposal, iTherapy will create a VR-based VAST therapy that allows those with ASD to better enhance their functional language abilities.

The innovation will be a video series of a realistic VR mouth which will require the use of an app on a tablet or a smartphone, VR goggles, and bone conduction (BC) headphones. The overall intended use for the VAST system will be in special education classrooms for children who struggle with functional language acquisition and motor abilities. Students will be able to improve upon their oral coordination during their in-school therapy sessions which often last between 30 minutes and 1 hour. The skills they learn will then be applied to their classwork and their daily interactions with peers and teachers. As their language abilities improve, teachers and students will be able to improve their cooperation. By using the VAST system, students will be better able to understand and process the speech of their teachers and become more integrated in the classroom setting. Each student who participates in special education will require the use of a tablet or smartphone, VR goggles, and BC headphones.

The incorporation of the VAST system into special education classrooms will help children with ASD to enhance their functional and social-communicative abilities. The VR system will teach them various motor and word production skills which will grow their vocabulary, reading skills, understanding of speech, and functional language usage as a whole. By building upon standard therapy techniques, this product will provide those with ASD an opportunity to enhance their quality of life by increasing their speech abilities which will enable them to build social networks and handle the events of daily life.

The incorporation of the VAST system into special education classrooms will help children with ASD to enhance their functional and social communicative abilities. The VR system will teach them various motor and word production skills which will grow their vocabulary, reading skills, understanding of speech, and functional language usage as a whole. By building upon standard therapy techniques, this product will provide those with ASD an opportunity to enhance their quality of life by increasing their speech abilities which will enable them to build social networks and handle the events of daily life.

iTherapy's VAST platform is at the forefront of reinventing standard clinical therapies. By implementing video modeling into a VR context, iTherapy will create a learning environment that can be individualized to the needs of the user, surpassing the limitations of current therapy techniques. By using VAST, individuals with motor and communication disorders such as ASD will be able to advance their functional language skills which will allow them to more effectively participate in daily communicative activities. Other video-modeling apps (Table 1) such as Watch Me Learn, iModeling, and Model Me Kids, although providing videos for imitation, do not use a VR setting. VR is able to enhance the efficacy of this therapy method and surpasses the abilities of standard videos. VR is easier to manipulate, making each user’s experience unique to themselves. The aforementioned apps will be more difficult to adapt to the ever-changing needs of the user, making the VAST platform an optimal educational platform.

Prior to the inspiration for the VAST VR product, iTherapy has found success and significant user interest in their current language-learning platform. InnerVoice. InnerVoice is a learning tool that engages its users through several modes of teaching functional expressive, receptive, and pragmatic language usage. The app is a holistic and natural way to use and learn language creating a social learning experience and featuring the capacity to express emotional content such as facial expressions and emotional prosody. iTherapy has been successfully marketing the InnerVoice AAC app, having priced it at $99 dollars to be sold on the Apple App Store. The app continues to grow, continuously developing robust features that surpass the limitations of other apps, and provides better overall functionality than its competitors. InnerVoice is currently being sold to educators and parents of those with ASD, as well as speech pathologists, behavioral analysts, and special educators.

VAST: Autism and Apraxia