EnvisionBox logo EnvisionBox logo

Analyzing Multi-Party Turn-Taking Behavior


James Trujillo (j.p.trujillo@uva.nl)

Updated: 2024-04-23


Info documents

This module gives an introduction to analyzing the turn-taking behavior of a group of individuals. We'll use manual annotations of speech behavior, quantify the timing of turn transitions as gaps and overlaps between speakers, and look at some ways we can perform additional analyses, such as taking the entropy of turn-transitions.

Packages:

Module citation: Trujillo, J.P. (2024-04-23). Analyzing Multi-Party Turn-Taking Behavior [day you visited the site]. Retrieved from: https://github.com/jptrujillo/turntaking_entropy_module/blob/main/TurnTaking_module.html

Location Repository:

https://github.com/jptrujillo/turntaking_entropy_module

Note on automatic utterance extraction: This module is partially based on the processing and analysis pipeline described in a previous publication (see below). The OSF project for this article also contains code for automatically extracting and processing single utterances from the audio stream. Note that this does require separate audio streams (e.g., from separate microphones) for each speaker. Code available at: https://osf.io/bvm56/?view_only=7165d0c25f9344a592685f4bae0c1bf7

Introduction

Turn-taking is a fundamental aspect of human communication, and takes place quite rapidly, with turn transitions (i.e., the time between one speaker's utterance ending and the next speaker's utterance beginning) often on the scale of ~200ms (Stivers et al., doi:10.1073pnas.0903616106%20pna/). This timing is important not only for fast-pace information exchange, but is socially meaningful, as longer-than-expected inter-speaker gaps can be interpreted as signalling a dispreferred response (Kendrick & Torreira 2015 doi: 10.1080/0163853X.2014.955997).
Beyond theoretical interest, quantifying turn-taking behavior can be interesting for understanding group dynamics. One study suggests that smoother turn transitions (i.e., shorter gaps, shorter overlaps) are more associated with collaborative behaviors compared to conflict behaviors (Kim et al., 2015, INTERSPEECH). Similarly, Pincus & Guastello (2005; DOI: 10.1177/1046496405280864) used non-linear dynamics approaches to show that information theoretic patterns in turn-taking behavior correlate with self-report measures of group dynamics (e.g., leadership, conflict). More recently, Hamish et al., (2023: https://escholarship.org/uc/item/2hb5k9fx ) have examined turn-taking entropy and critical instability (also described in this module) in relation to collaborative problem solving.

This function just gets us some summary stats for the turn taking behavior

Descriptive Statistics

Let's see what we can learn just from these turn transition times. Several of these measures can inform us about speaker dominance. Who's taking and holding the floor the most?

The above output shows the turn transition information for each person in our interaction. Remember that each of the values are based on the transitions from this speaker's uterance. So for example, speaker 1 has a gap frequency of 56%, and an overlap frequency of 66%. This means that when speaker 1 finishes an utterance, s/he is "interrupted" 44% of the time (on average by 558ms), but typically there is a gap of 439ms before the next speaker begins their untterance.
This information can already be useful for looking into how individual speakers behave with one another. One way to proceed with this data would be to simply take these summary values, per individual, per group, and statistically compare groups with one another. Do certain groups have smoother turn transition timings? We can also correlate the turn timings with other information, such as KAI scores.

Entropy of Turn Timing

Another way to capture the dynamics of turn-taking behavior is to look at Approximate Entropy. In order to have an idea of what this provides us, let's consider the statistics output above. In a perfectly 'stable' back-and-forth interaction, each turn transition could be approximately 36ms, across the entire interaction. We would then get a mean turn transition time of 4ms, with a SD of 667ms (as seen above for speaker 1 -- reflecting the tendency around 0). However, it may be that these values change over the course of the interaction, and knowing the mean won't necessarily let you accurately predict the next transition time. The latter case is an example of higher entropy.

Aside from the general entropy across the whole session, which we could compare across groups or correlate with personal or group characteristics, we can also look at the change in entropy over time. We'll split the session into 4 minute windows to assess entropy across the session.

For the analysis, we'll use the categorical data corresponding to the "transitioning speaker" and calculate Permutation Entropy. Permutation entropy captures the ordering of values in a time-series. In this case, we want the complexity with which the group switches from one speaker to the next. Is it always speaker 1 followed by speaker 2, and vice versa? This would be low entropy. Or do we see all possible orders; speaker 3, then 1, followed by 3 again, then 2.. The more unpredictable the switching, the higher the entropy.

These values can again be correlated with group level characteristics, or to momentary annotations regarding breakthroughs, changes in topic, etc.

Linking turn-taking entropy to communicative success

Ideally, we will also have some additional information about what is happening during a conversation. For example, we may have annotations regarding certain interaction events (e.g., moments of disagreement, changes in topic), or some indication of communicative success, such as in collaborative problem solving tasks. By relating such annotations to turn-taking entropy, we can learn about how the dynamics of turn-taking relate to what is actually happening during the conversation.
We'll use the open access Weights Task Dataset from Ibrahim and colleagues (2024) as an example. The data can be downloaded from: https://zenodo.org/records/10252341
For this module, we'll focuse on the _Oracle.csv files, which provide transcriptions of all of the interactions within each group. This will allow us to calculate turn-taking (entropy) as shown above.

Now we need to calculate the turn-taking entropy for these groups.

The average entropy over each group's interaction is easy to calculate, and we may want to correlate it with other high-level chacteristics. For example, the "constr" score that we calculated above is a measure of how frequently a group engaged in "constructing shared knowledge". Is frequent co-construction of shared knowledge associated with lower entropy?
We may also be interested in whether groups that are faster at completing the task have overall lower entropy.In this case, we can simply calculate their "time on task" based on the last timestamp in each group's transcript.
Try for yourself:
See if you can plot these associations, or even do some basic statistical analyses on these associations. Are they informative?

Entropy Over Time

While group-level entropy is easy to calculate, we can also use a sliding window approach to see how entropy develops over time. In the previous section of this module, we calculate entropy for several subsequent time windows. This may be useful, particularly if there are particular divisions in a conversational task. However, we may want to have a smoother approach, and use a sliding window, which provides finer temporal resolution.
In this next code block, we'll calculate entropy over 60-second windows that will shift by 30-seconds, until we get to the end of the data.
Important to Note:
While a smaller window provides finer temporal resolution, remember that turn-taking itself occurs on a particular timescale. So, your window should be large enough to capture several turn-exchanges. If the window is so small that it only ever captures 1 or 2 exchanges, the entropy values will be less informative than when you zoom out and take more data into the calculation.

Now, let's plot the data and see how entropy changes over the course of an interaction.
Try for yourself:
Try changing the window_length and shift variables to see how this affects the visualized pattern.

One very useful application to having a time-course of entropy over the entire interaction, is that we can relate the entropy at specific points in the interaction to other (annotated) events. Does higher or lower entropy precede, or follow, particular events?
We'll use a very quick and dirty example, by plotting moments when a team member solved a subtask within the larger CPS interaction.

Calculating Dynamic Instability

While the entropy values themselves can be interesting, we may also be interested in the overal dynamic of the conversation or discussion. One way to do this is to calculate the number of moments of critical instability (Wiltshire, et al., 2021: https://doi.org/10.1109/ICHMS53169.2021.9582454), which we define as peaks in the turn-taking entropy. In other words, these are moments when the turn exchanges experience a sudden spike in instability, when the back-and-forth of conversation is highly unpredictable. These peaks are then followed by a return to a more stable (lower entropy) dynamic.

We can use SciPy's find_peaks function to easily calculate this, although the height, prominence and distance values may need to be adjusted according to the way entropy is calculated. You can try to adjust these values and see how the following plot changes. Note that the orange points are considered the top of these peaks, and thus each represent a critical instability moment.

Here, we'll return to our earlier loop, but now also calculate the number of critical instability moments for each group.

In team-based problem-solving tasks, moments of critical instability may be necessary changes in the dynamic of a conversation/discussion, where a team is (for example) changing strategies or trying to find a stable dynamic. Accordingly, many such changes in dynamics are likely to be associated with a longer time working on a problem. Teams that find a stable, effective dynamic with fewer such changes, will then be faster. Even without looking into what exactly is occurring during a conversation, this type of analysis can provide a first look at whether moments of critical instability are associated with how long it takes to complete a task.
Try for yourself:
Just as above, see if you can plot the group_instability_peaks data against the time-on-task ("ToT") that we calculated earlier. Again, including a fit line and some basic statistics can provide an additional check on whether this critical instability is indeed a useful way to capture (the route to) collaborative communicative success.