Verbal and co-verbal behavior under cognitive pressure: Analyses of speech, gesture, and eye gaze

(The Russian Science Foundation grant № 19-18-00357)

In this project we study the phenomena of verbal and co-verbal behaviour of speakers in contexts of intense cognitive pressure. This issue is part of a larger interdisciplinary research topic concerning professionals who regularly handle a heavy cognitive load under time pressure , such as surgeons conducting an operation, pilots manually trying to stabilize a plane in peril, or simultaneous interpreters. Their work involves a high demand on their working memory. We focus on simultaneous interpreters as their work requires using language as the working material itself, rather than other tools that they manipulate.
So far, we have completed the first stage of the project: conducted a series of experiments (in Russian and English) and analysed the obtained data. Video recordings of Russian interpreters who interpreted a TED talk from English into Russian and vice-versa were annotated and analyzed with the help of ELAN software.

Participants were recorded by three cameras: a video camera on a tripod, looking over their shoulder; a GoPro camera in front of them; a built-in camera in eye-tracking glasses.The videos have been  analysed for moments of difficulties and disfluencies, use of manual gestures, use of gesture space; and head movements (tilts, nods, lifts).

At the moment, our team has started the second stage of the experiment, which includes data collection and further analysis for Russian-German language pair.