The project “Verbal and co-verbal means of event construal across languages”
supported by the Russian Science Foundation (grant № 14-48-00067П) with the participation of researchers from Russia, Germany, Netherlands and France.
In this project we studied the way verbal and co-verbal means interact in event construal in different languages: in Russian, German and French. The focus of the research was on the temporal-aspect structure of an event, which specifies the choice of language material: in German the category aspect is expressed with the help of lexical means, in French – with grammatical structures, while in Russian linguistic aspect is a lexico-grammatical category.
In the project we approached eventfullness and aspectuality from a multimodal perspective. We analyzed these categories in oral discourse considering multimodality of communication. thus we studied both verbal and non- verbal components. Analyzing aspectuality and events from a multimodal perspective, we referred to spontaneous gestures which accompany the speech. The analysis of interaction between gesture parameters and verbal means of aspect representation was performed both for speech production processes and for its interpretation.
During the experiments we discovered a number of gesture parameters with the help of which the lexical and grammatical aspects of these languages are expressed; we also studied in what way verbal and non-verbal components influence speech interpretation.
The study included three stages and on each stage, experimental research was carried out. On the first stage, 30 elicited monologues with elements of dialogue (10 in each language – Russian, French and German) were filmed. The participants were telling each other about random situations from their everyday lives. Each video lasted for about 10 minutes. Then the criteria of speech and gesture analysis were defined. We developed a controlled lexicon for marking verbs considering the peculiarities of time, aspect and modality categories functioning in each language. We used two approaches to study gestures. The basis of the first one was an idea of presence/absence of an energetic impulse in gestures. Due to this idea, a number of schemes with event structures were elaborated. We called such schemes “boundary schemas”. Within the second approach, a kinesiological system was built which considers the type, direction and speed of hand, hand segment and forearm movements.
Then, in the Elan program we marked the corpus according to the defined criteria. First we examined the relevant verb parameters in the corpus, paying particular attention to the past tense forms of verbs which were synchronized with gestures. It should be outlined that in the research we took into account all gestures synchronized with verbs, except for self-adapters.
The data obtained on the basis of three languages displayed an exciting pattern. Results of the French narrative analysis supported the hypothesis of the research: A stable correlation of bounded gestures with perfect tense verbs (passé composé) (p<.01) and non-bounded gestures with imperfect tense verbs (imparfait) (p<.01) was observed. At the same time in the German language
this correlation remains for bounded gestures and the perfect tense (Perfekt) (p<.01), but in the case of the simple past tense (Präteritum), the differences in assignment of bounded and non-bounded gestures were not statistically significant (p=0.35). In the Russian videocorpus the use of bounded gestures did not depend on the choice of verb aspect (both with regard to perfective and imperfective forms). These results can be connected with the choice of gesture type, the versatility of gestures, peculiarities of the aspect categories in the different languages and some parameters of the events the participants discussed.
On the second stage we received the data of the experimental analysis of speech comprehension processes in the three languages in Russia (Moscow), Germany (Frankfurt an der Oder) and France (Paris and Nanсy).
The results of the experiment on speech comprehension in the Russian language confirmed our findings obtained in the narrative production analysis. In all three languages we did not detect differences in response time of the participants to unedited videos in which the perfective verb was accompanied with the bounded gesture and imperfective – with non-bounded; and the response to edited video fragments in which this compatibility was disturbed and the paticipant was shown bounded gestures with imperfective verbs and vice versa. It should be noted though that in the languages analyzed, the following statistically significant tendency was discovered (p=0.05): the speed of verb recognition (in case of the correct answer to the question if the participants heard the verb in the previous video or not) was higher overall for bounded gestures.
The third stage of the research enabled us to examine the differences of aspect representation considering the gesture component of native and foreign language speakers . During the third stage we recorded 20 videos in each language (more than 60 in total) and marked the verb forms occurring with gestures. The outcome from the data allowed us to carry out a comparative analysis and resulted in the development of the pipeline for conducting analysis of gestures using motion capture, as outlined in the documents attached below (the Pipeline Tutorial and Pipeline Documentation).
Within the project we organized a number of events, during which the participants shared the results of their research with other colleagues, Ph.D. candidates and students.