A lot of our recent work revolves around working with conversational data, and one thing that’s struck me is that there are no easy ways to create compelling visualizations of conversation as it unfolds over time. The most common form seems to be pixelated screenshots of transcription software not made for this purpose. In the Elementary Particles of Conversations project we’re aiming to change that — see our ACL2022 paper and stay tuned for updates on talkr
. Here I want to give a sneak peek into the kitchen as I experiment with new ways of plotting conversational structure.
Also, this post is my first time blogging to WordPress directly from rmarkdown. I’m using the amazing goodpress
package: https://github.com/maelle/goodpress/.
Ten minutes of conversation
I’m going to walk through an example of plotting a ten minute stretch of conversation. We start by loading some sample data. For privacy reasons I’ll use only the timing data. The most important bits here: every annotation corresponds to a turn at talk by some participant
with a begin
and end
time in milliseconds. This corresponds to the minimal flat data format for diarised conversational data we specify here.
extract <- readr::read_csv('sample_conversation.csv',show_col_types = F) %>%
select(begin,end,duration,participant,uid,nwords,nchar,n,rank,freq,overlap,priorby,FTO,overlapped,talk_all,talk_rel,load,transitions,topturn,focus,scope,participant_int,begin0,end0,participation)
head(extract)
#> # A tibble: 6 × 25
#> begin end duration participant uid nwords nchar n rank freq
#> <dbl> <dbl> <dbl> <chr> <chr> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1 1271 3158 1887 A_agent_text hungari… 1 2 84 40 1.29e-6
#> 2 3158 4830 1672 A_agent_text hungari… 4 20 1 112 1.53e-8
#> 3 6221 8150 1929 A_agent_text hungari… 1 7 1 112 1.53e-8
#> 4 7344 8145 801 A_speaker_text hungari… NA NA NA NA NA
#> 5 8150 9821 1671 A_agent_text hungari… 4 26 1 112 1.53e-8
#> 6 9007 10151 1144 A_speaker_text hungari… 4 17 1 112 1.53e-8
#> # ℹ 15 more variables: overlap <chr>, priorby <chr>, FTO <dbl>,
#> # overlapped <chr>, talk_all <dbl>, talk_rel <dbl>, load <dbl>,
#> # transitions <dbl>, topturn <dbl>, focus <chr>, scope <chr>,
#> # participant_int <dbl>, begin0 <dbl>, end0 <dbl>, participation <chr>
We start by setting a few basic parameters that will help us to divide these ten minutes of conversation into as many lines. Basically, we cut up the conversational turns by their categorizing begin
values into intervals the width of window_size
.
extract_length
(ms): total length of desired stretch of conversationwindow_size
(ms): length of a single linewindow_breaks
: integer vector of extract_length divided by window_size
extract_length <- 600000 # 10 min
window_size <- 60000 # 1 min
window_breaks <- as.integer(c(0:round(extract_length/window_size)) * window_size)
extract <- extract %>%
mutate(end = end - min(begin), # reset timestamps to start from 0
begin = begin - min(begin),
line = cut(begin,window_breaks,right=F,labels=F)) %>%
drop_na(line) %>%
group_by(line) %>%
mutate(begin0 = begin - min(begin), # reset timestamps to 0 for each new line
end0 = end - min(begin)) %>%
ungroup()
Now that our extract
has turns divided into lines, we start with a regular plot in which the conversation flows from left to right and from top to bottom. We number the lines so that you can see how turns get assigned to them.
There’s a bunch of things you might note about this plot. We reverse the y scale because we want the first turns (line 1) to be on top. We plot the actual turns (or at least their timing) with geom_rect()
. And there’s another layer of items plotted with geom_point()
; these are interjections, one-word turns like uh-huh and yeah.
plot.base <- extract %>%
ggplot(aes(y=participant_int)) +
ggthemes::theme_tufte() + theme(legend.position = "none",
strip.text = element_blank(),
axis.text.y = element_blank(),
axis.ticks.y = element_blank(),
plot.title.position = "plot") +
ylab("") + xlab("time (s)") +
viridis::scale_fill_viridis(option="plasma",direction=1,begin=0.2,end=0.8) +
scale_y_reverse(breaks=seq(1,max(extract$line,1)),
labels=seq(1,max(extract$line,1))) +
scale_x_continuous(limits=c(0,window_size),
breaks=seq(0,window_size,10000),
label=seq(0,window_size/1000,10),
oob = scales::oob_keep)
plot.simple_grid <- plot.base +
theme(axis.text.y = element_text()) +
ggtitle("Ten minutes of conversation",
subtitle="Points are interjections, time moves left-right and top-bottom") +
geom_rect(aes(xmin=begin0,xmax=end0,ymin=line-0.5+participant_int/3-0.2,ymax=line-0.5+participant_int/3+0.2),
linewidth=0,colour=NA,fill="lightgrey") +
geom_point(data=. %>% filter(nwords == 1, topturn == 1),
aes(x=begin0+200,fill=rank,y=line-0.5+participant_int/3),colour="white",size=2,shape=21,stroke=1)
plot.simple_grid
A grid is boring though. Can we make it more interesting visually? Let’s try a polar plot. One lovely thing about ggplot is that this is as easy as adding coord_polar()
:
plot.simple_grid +
coord_polar()
Make it spiral
It would be much nicer of course if this were a spiral instead of concentric rings. For that, we’re going to have to take a slightly different approach. Recall that the y coordinate of individual turns is currently determined by their line
number. If we want to make the end of one line meet the beginning of the next, we need a variable that increments over the window_length
. Let’s call it line_polar
:
extract <- extract %>%
mutate(line_polar = line+((1+begin0)/window_size))
# we'll need to update our plot.base with that new dataset
plot.base <- extract %>%
ggplot(aes(y=participant_int)) +
ggthemes::theme_tufte() + theme(legend.position = "none",
strip.text = element_blank(),
axis.text.y = element_blank(),
axis.ticks.y = element_blank(),
plot.title.position = "plot") +
ylab("") + xlab("time (s)") +
viridis::scale_fill_viridis(option="plasma",direction=1,begin=0.2,end=0.8) +
scale_y_reverse(breaks=seq(1,max(extract$line,1)),
labels=seq(1,max(extract$line,1))) +
scale_x_continuous(limits=c(0,window_size),
breaks=seq(0,window_size,10000),
label=seq(0,window_size/1000,10),
oob = scales::oob_keep)
We’ll also need a line to guide our eyes along the spiral. We make a grid of points that we can plot as a geom_line()
. We now recreate our plot using the updated dataframe and position the turns on the line_polar
variable:
spiral <- expand.grid(seconds = seq(0,window_size,1000), line = 1:max(extract$line))
spiral$line_polar <- spiral$line+(spiral$seconds/window_size)
plot.polar <- plot.base +
coord_polar() +
geom_rect(aes(xmin=begin0,xmax=end0,ymin=line_polar-1+participant_int/3-0.2,ymax=line_polar-1+participant_int/3+0.2),
linewidth=0,colour=NA,fill="lightgrey") +
geom_line(data=spiral,aes(x=seconds,y=line_polar-0.5,group=line),
color="darkgrey",linewidth=0.5) +
geom_point(data=. %>% filter(nwords == 1, topturn == 1),
aes(x=begin0+200,fill=rank,y=line_polar-1+participant_int/3),colour="white",size=2,shape=21,stroke=1)
plot.polar
Cool! It starts to look like a spiral. But wait — now perhaps the time dimension doesn’t make a lot of sense anymore: it starts at the top and rolls… inward? It would be nicer if things moved outward, giving a sense of conversation unrolling. That’s as easy as un-reversing the y-axis (which, you recall, we reversed because in plot.simple_grid
we wanted it to run from top to bottom).
We get a warning here for setting the y-axis for a second time but that’s okay.
plot.polar + scale_y_continuous()
#> Scale for y is already present.
#> Adding another scale for y, which will replace the existing scale.
Bonus plot. The line_polar
trick we use to weld line ends to next line beginnings is easy to see if we unroll this plot. Commenting out coord_polar()
gives us:
plot.base +
# coord_polar() +
theme(axis.text.y=element_text()) +
geom_rect(aes(xmin=begin0,xmax=end0,ymin=line_polar-0.5+participant_int/3-0.2,ymax=line_polar-0.5+participant_int/3+0.2),
linewidth=0,colour=NA,fill="lightgrey") +
geom_line(data=spiral,aes(x=seconds,y=line_polar,group=line),
color="darkgrey",linewidth=0.5) +
geom_point(data=. %>% filter(nwords == 1, topturn == 1),
aes(x=begin0+200,fill=rank,y=line_polar-0.5+participant_int/3),colour="white",size=2,shape=21,stroke=1)
That’s it for today. Note that none of these experimental plots are proposed as serious scientific visualizations. In particular, the polar plot has the obvious drawback of deforming time. So really the point of this blog post is just to test a new workflow for blogging straight from R, which will be useful if I have more plots and code to share. For more information on conversational structure and its importance for NLP, linguistics and the cognitive sciences, check out Elementary Particles of Conversation. Thanks for reading!