The project “Verbal and co-verbal means of event construal across languages”

the Russian Science Foundation (grant № 14-48-00067П) with the participation of researchers from Russia, Germany, Netherlands and France.

In this project we study the way verbal and co-verbal means interact in event construal in different languages: in Russian, German and French. The spotlight of the research is the temporal-aspect structure of an event, which specifies the choice of language material: in German the category aspect is expressed with the help of lexical means, in French – with grammatical, while in Russian language aspect is a lexico-grammatical category.

In the project we approach eventfullness and aspectuality from a multimodal perspective. We analyze these categories in oral discourse considering multimodality of communication. thus we study both verbal and non- verbal components. Analyzing aspectuality and eventfulness from multimodal perspective, we refer to spontaneous gestures, which accompany the speech. The analysis of interaction between gesture parameters and verbal means of aspect representation is performed both for speech production processes and for its interpretation.

During the experiments we discovered a number of gesture parameters with the help of which the lexical and grammatical aspects of concerned languages are expressed; we also studied in what way verbal and non-verbal components influence speech interpretation.

The study included three stages and on each stage an experimental research was carried out. On the first stage 30 intent monologues with elements of dialogue (10 in each language – Russian, French and German) were filmed. The participant were telling each other about random situatios from their everyday lives. Each video lasts for about 10 minutes. Then the criteria of speech and gesture analysis were defined. We developed a controlled lexicon for marking verbs considering the peculiarities of time, aspect and modality categories functioning and existence of complex forms in each language. We used two approaches to study gestures. The basis of the first one is an idea of presence/absence of an energetic impulse in gestures. Due to this idea a number of schemes with event structure was elaborated. Such schemes are called “boundary schemes”. Within the second approach a kinesiological system was built which considers the type, direction and speed of hand, hand segments and forearm movements.

ELAN-24.09.jpeg

Then, in the Elan program we marked the corpus according to the defined criteria. The received data was processed and analyzed. First we examined verbs’ parameters in the marked corpus paying particular attention to the past form verbs which were synchronized with gestures. It should be outlined that in the research we took into account all gestures synchronized with verbs, but for adepts.

The data obtained on the basis of three languages displayed an exciting pattern. Results of the French narrative analysis proved the hypothesis of the research: A stable correlation of boundary gestures with perfect tense verbs (passé composé) (p<.01) and non-boundary gestures with imperfect tense verbs (imparfait) (p<.01) is observed. At the same time in German language this correlation remains for boundary gestures and perfect tense (Perfekt) (p<.01), but in case with past tense (Präteritum), the differences in assignment of boundary and non-boundary gestures were not statistically accurate (p=0.35). in Russian videocorpus the proportion, demonstrating close quantitative meanings, changes in favor of boundary gestures, not depending on the choice of verb aspect (p<.01 both for perfect and imperfect forms). Such result can be connected with the choice of gesture type, versatility of gestures, peculiarities of aspect category in different languages and with some parameters of the discussed events).

On the second stage we received the data of the experimental analysis of speech perception processes in three languages in Russia (Moscow), Germany (Frankfurt an der Oder) and France (Paris and Nanсy).

The results of the experiment in speech perception in Russian language confirmed our findings obtained in the narrative production analysis. In all the three languages we did not detect differences in response speed of the participants to inedited videos in which the perfective verb was accompanied with the boundary gesture and imperfective – with non-boundary; and the response to edited video fragments in which this compatibility was disturbed and the paticipant was shown boundary gestures with imperfective verbs and vice versa.

The absence of explicit response to the mismatch of boundary schemes of gesture and type of verb form observed in all languages can be caused by the versatility of gestures (functions of representation, discourse structuring and pragmatic functions).  

It should be noted though that in the analyzed languages the following statistically significant tendency was discovered (p=0.05): the speed of verb recognition (in case of the correct answer to the question if the examinees heard the verb in the previous video or not) was higher for boundary gestures.

At the moment the third stage of the research is in progress. It will enable us to examine the differences of aspect representation considering the gesture component of native and foreign language speakers . During the third stage we shot 20 videos in each language (more than 60 in total) and the marking the verb forms with synchronized gestures is being done. The outcome data will let us carry out a comparative analysis and also outline new parameters for evaluating foreign language competences for the learners.

photo21110288_456239165.jpg

Within the project we organized a number of events, during which the participants shared the results of their researches with the colleagues, Ph.D. candidates and students.