Playing with R: unrolling conversation

A lot of our recent work revolves around working with conversational data, and one thing that’s struck me is that there are no easy ways to create compelling visualizations of conversation as it unfolds over time. The most common form seems to be pixelated screenshots of transcription software not made for this purpose. In the Elementary Particles of Conversations project we’re aiming to change that — see our ACL2022 paper and stay tuned for updates on talkr. Here I want to give a sneak peek into the kitchen as I experiment with new ways of plotting conversational structure.

Also, this post is my first time blogging to WordPress directly from rmarkdown. I’m using the amazing goodpress package: https://github.com/maelle/goodpress/.

Ten minutes of conversation

I’m going to walk through an example of plotting a ten minute stretch of conversation. We start by loading some sample data. For privacy reasons I’ll use only the timing data. The most important bits here: every annotation corresponds to a turn at talk by some participant with a begin and end time in milliseconds. This corresponds to the minimal flat data format for diarised conversational data we specify here.


extract <- readr::read_csv('sample_conversation.csv',show_col_types = F) %>%
  select(begin,end,duration,participant,uid,nwords,nchar,n,rank,freq,overlap,priorby,FTO,overlapped,talk_all,talk_rel,load,transitions,topturn,focus,scope,participant_int,begin0,end0,participation)

head(extract)
#> # A tibble: 6 × 25
#>   begin   end duration participant    uid      nwords nchar     n  rank     freq
#>   <dbl> <dbl>    <dbl> <chr>          <chr>     <dbl> <dbl> <dbl> <dbl>    <dbl>
#> 1  1271  3158     1887 A_agent_text   hungari…      1     2    84    40  1.29e-6
#> 2  3158  4830     1672 A_agent_text   hungari…      4    20     1   112  1.53e-8
#> 3  6221  8150     1929 A_agent_text   hungari…      1     7     1   112  1.53e-8
#> 4  7344  8145      801 A_speaker_text hungari…     NA    NA    NA    NA NA      
#> 5  8150  9821     1671 A_agent_text   hungari…      4    26     1   112  1.53e-8
#> 6  9007 10151     1144 A_speaker_text hungari…      4    17     1   112  1.53e-8
#> # ℹ 15 more variables: overlap <chr>, priorby <chr>, FTO <dbl>,
#> #   overlapped <chr>, talk_all <dbl>, talk_rel <dbl>, load <dbl>,
#> #   transitions <dbl>, topturn <dbl>, focus <chr>, scope <chr>,
#> #   participant_int <dbl>, begin0 <dbl>, end0 <dbl>, participation <chr>

We start by setting a few basic parameters that will help us to divide these ten minutes of conversation into as many lines. Basically, we cut up the conversational turns by their categorizing begin values into intervals the width of window_size.

  • extract_length (ms): total length of desired stretch of conversation
  • window_size (ms): length of a single line
  • window_breaks: integer vector of extract_length divided by window_size

extract_length <- 600000 # 10 min
window_size <- 60000 # 1 min
window_breaks <- as.integer(c(0:round(extract_length/window_size)) * window_size)

extract <- extract %>%
  mutate(end = end - min(begin), # reset timestamps to start from 0
         begin = begin - min(begin),
         line = cut(begin,window_breaks,right=F,labels=F)) %>%
  drop_na(line) %>%
  group_by(line) %>%
  mutate(begin0 = begin - min(begin), # reset timestamps to 0 for each new line
         end0 = end - min(begin)) %>%
  ungroup()

Now that our extract has turns divided into lines, we start with a regular plot in which the conversation flows from left to right and from top to bottom. We number the lines so that you can see how turns get assigned to them.

There’s a bunch of things you might note about this plot. We reverse the y scale because we want the first turns (line 1) to be on top. We plot the actual turns (or at least their timing) with geom_rect(). And there’s another layer of items plotted with geom_point(); these are interjections, one-word turns like uh-huh and yeah.


plot.base <- extract %>%
  ggplot(aes(y=participant_int)) +
  ggthemes::theme_tufte() + theme(legend.position = "none",
                        strip.text = element_blank(),
                        axis.text.y = element_blank(),
                        axis.ticks.y = element_blank(),
                        plot.title.position = "plot") +
  ylab("") + xlab("time (s)") +
  viridis::scale_fill_viridis(option="plasma",direction=1,begin=0.2,end=0.8) +
  scale_y_reverse(breaks=seq(1,max(extract$line,1)),
                  labels=seq(1,max(extract$line,1))) +
  scale_x_continuous(limits=c(0,window_size),
                     breaks=seq(0,window_size,10000),
                     label=seq(0,window_size/1000,10),
                     oob = scales::oob_keep) 

plot.simple_grid <- plot.base +
  theme(axis.text.y = element_text()) +
  ggtitle("Ten minutes of conversation",
          subtitle="Points are interjections, time moves left-right and top-bottom") +
  geom_rect(aes(xmin=begin0,xmax=end0,ymin=line-0.5+participant_int/3-0.2,ymax=line-0.5+participant_int/3+0.2), 
                      linewidth=0,colour=NA,fill="lightgrey") +
  geom_point(data=. %>% filter(nwords == 1, topturn == 1),
             aes(x=begin0+200,fill=rank,y=line-0.5+participant_int/3),colour="white",size=2,shape=21,stroke=1)
plot.simple_grid

A grid is boring though. Can we make it more interesting visually? Let’s try a polar plot. One lovely thing about ggplot is that this is as easy as adding coord_polar():


plot.simple_grid +
  coord_polar() 

Make it spiral

It would be much nicer of course if this were a spiral instead of concentric rings. For that, we’re going to have to take a slightly different approach. Recall that the y coordinate of individual turns is currently determined by their line number. If we want to make the end of one line meet the beginning of the next, we need a variable that increments over the window_length. Let’s call it line_polar:

extract <- extract %>% 
  mutate(line_polar = line+((1+begin0)/window_size)) 

# we'll need to update our plot.base with that new dataset
plot.base <- extract %>%
  ggplot(aes(y=participant_int)) +
  ggthemes::theme_tufte() + theme(legend.position = "none",
                        strip.text = element_blank(),
                        axis.text.y = element_blank(),
                        axis.ticks.y = element_blank(),
                        plot.title.position = "plot") +
  ylab("") + xlab("time (s)") +
  viridis::scale_fill_viridis(option="plasma",direction=1,begin=0.2,end=0.8) +
  scale_y_reverse(breaks=seq(1,max(extract$line,1)),
                  labels=seq(1,max(extract$line,1))) +
  scale_x_continuous(limits=c(0,window_size),
                     breaks=seq(0,window_size,10000),
                     label=seq(0,window_size/1000,10),
                     oob = scales::oob_keep) 

We’ll also need a line to guide our eyes along the spiral. We make a grid of points that we can plot as a geom_line(). We now recreate our plot using the updated dataframe and position the turns on the line_polar variable:


spiral <- expand.grid(seconds = seq(0,window_size,1000), line = 1:max(extract$line))
spiral$line_polar <- spiral$line+(spiral$seconds/window_size)

plot.polar <- plot.base +
  coord_polar() +
  geom_rect(aes(xmin=begin0,xmax=end0,ymin=line_polar-1+participant_int/3-0.2,ymax=line_polar-1+participant_int/3+0.2),
            linewidth=0,colour=NA,fill="lightgrey") +
  geom_line(data=spiral,aes(x=seconds,y=line_polar-0.5,group=line),
            color="darkgrey",linewidth=0.5) +
  geom_point(data=. %>% filter(nwords == 1, topturn == 1),
             aes(x=begin0+200,fill=rank,y=line_polar-1+participant_int/3),colour="white",size=2,shape=21,stroke=1)

plot.polar

Cool! It starts to look like a spiral. But wait — now perhaps the time dimension doesn’t make a lot of sense anymore: it starts at the top and rolls… inward? It would be nicer if things moved outward, giving a sense of conversation unrolling. That’s as easy as un-reversing the y-axis (which, you recall, we reversed because in plot.simple_grid we wanted it to run from top to bottom).

We get a warning here for setting the y-axis for a second time but that’s okay.


plot.polar + scale_y_continuous() 
#> Scale for y is already present.
#> Adding another scale for y, which will replace the existing scale.

Bonus plot. The line_polar trick we use to weld line ends to next line beginnings is easy to see if we unroll this plot. Commenting out coord_polar() gives us:


plot.base + 
  #  coord_polar() +
  theme(axis.text.y=element_text()) +
  geom_rect(aes(xmin=begin0,xmax=end0,ymin=line_polar-0.5+participant_int/3-0.2,ymax=line_polar-0.5+participant_int/3+0.2),
            linewidth=0,colour=NA,fill="lightgrey") +
  geom_line(data=spiral,aes(x=seconds,y=line_polar,group=line),
            color="darkgrey",linewidth=0.5) +
  geom_point(data=. %>% filter(nwords == 1, topturn == 1),
             aes(x=begin0+200,fill=rank,y=line_polar-0.5+participant_int/3),colour="white",size=2,shape=21,stroke=1)

That’s it for today. Note that none of these experimental plots are proposed as serious scientific visualizations. In particular, the polar plot has the obvious drawback of deforming time. So really the point of this blog post is just to test a new workflow for blogging straight from R, which will be useful if I have more plots and code to share. For more information on conversational structure and its importance for NLP, linguistics and the cognitive sciences, check out Elementary Particles of Conversation. Thanks for reading!

Leave a Reply

Your email address will not be published. Required fields are marked *