Via Language Log, a nice tutorial titled Interactive Visualization for Computational Linguistics [PDF, 13,1 Mb] by Christopher Collins, Gerald Penn, and Sheelagh Carpendale. Includes not only lots of wonderful visualizations, but also a lot of background information on Gestalt perception, visualizations as ‘external cognition’, preattentive processing, info on a case study (slide 196ff.), and ample examples of different kinds of visualization software. See also InfoVis:Wiki — Linguistic Visualization.

Great news for those who are into visual corpus linguistics but don’t work on SAE languages: since July, Wordle handles alphabets in the Extended Latin ranges; and today its maker, Jonathan Feinberg, added support for combining diacritics. That means that you can now feed Wordle texts from languages that use tone marks and other diacritics in their orthographies. Like Siwu.

Wordle based on some ten minutes of spontaneous conversation in Siwu.

The Wordle above displays the most common words in some ten minutes of spontaneous conversation in Siwu, one of the fruits of my last fieldtrip. The conversation has four participants. Nothing groundbreaking about this particular Wordle, it’s just a nice word cloud starring:

• kùɖu ‘gunpowder’, the main topic of the conversation since this was actually videotaped while the four participants were manufacturing a local type of gunpowder;
• sɔ, the all-purpose quotative/complementizer — (X) (says) that Y…;
• gɔ, the relative pronoun for the animate singular class, as in ɔ̀turi gɔ lokpi ‘the man who died’ — this indicates that a lot of the talk is about persons;
• mm, a backchannel cue signalling involvement and attention
• fɔ, an emphatic 2nd person pronoun;
• kɔ̃rɔ ‘right now’, a stoplap roughly used as English now in ‘now we went there and guess what happened…’

Things this Wordle cannot show is the relative differences in conversational strategies of the participants. Mr. Orange, for example, as I call him in my ELAN transcript of the conversation, is by far the main supplier of mm and its cousin m-hm; in fact his repertoire is not much bigger than that — rather than doing the talking he prefers to have a supporting role in this particular conversation.

The three others are much more vocal and varied, not to mention much more expressive (ideophones are sprinkled all over the place!). But that’s all for another occassion. For now, cheers to the new Wordle!

A visualization of the previous two posts on Many Eyes and Siwu ne

Because recursivity is a Good Thing, here is a visualization of the previous two posts on visualizing linguistic data with Many Eyes. The astute reader will note that the strange loop is not perfect since I didn’t use Many Eyes for the visualization — that is because nothing can do a simple visualization as beautifully as Wordle.

Unfortunately, Wordle doesn’t seem to handle Unicode outside the basic Latin range very well (probably because of the fancy fonts), otherwise I would’ve fed it some Siwu text, too. (I think Wordle could be made to work with SIL’s freely available Unicode fonts.)

Queensland grammar scandal at a glance

As an added bonus, here are the 75 most common content words from the recent discussion of the Queensland grammar scandal (sampled from three verbose posts at Language Log and from matjjin-nehen, including comments). It won’t help the debate, but it does give you the brouhaha at a glance. Another different something function. Grammatical grammar. Australian grammar errors indeed.

The 75 most common content words about the Queensland grammar brouhaha in the linguablogosphere

(Link to Wordle found in Cornelis Puschmann‘s del.icio.us feed.)

Lots of readers looked at the challenge I posted last week (my blog statistics say more than 450 views for the post alone, so that’s many eyes indeed). A few of you were even daring enough to come up with a story on the various functions of Siwu ne. The challenge was probably a bit too difficult (involving an untranslated text in an as of yet undescribed Niger-Congo language), which makes those few attempts all the more heroic. So what did they see?

ne and its right periphery; “ne, …” accounts for almost half of the tokens

Brett was the first to bite the bullet, providing some statistics on the use of ne. He noted that “it occurs sentence initially 158 times (out of 1161) and sentence terminally 83 times. (…) It often seems to bracket a whole clause and it can even be doubled. The ne kama ne string is quite common.” Ray Girvan didn’t trust the visualization and inspected the raw text instead. He discovered that the text contains some dialogues “as well as as a complete song/poem with multiple uses of “ne” in a question”; that the construct Si …. ne occurs frequently in it; and that the text probably consisted of several different text types. On came Jason with a number of rather detailed observations:

• “ne” is the most common word in the sample data. It is more common than the most common punctuation marks, “.” and “,”. It is the most common word following a period (”. ne” gives 158 hits; the runner-up “. si” has 115) and the most common word preceding a period (”ne .”: 84; “ame .”: 72).
• “ne” usually appears before or after a “.” or “,”:
  • – 42.9%: immediately before “,”
  • 13.6%: sentence-initial (”ne” after “.” “?” or “!”)
  • 7.8%: sentence-final (”ne” before “.” “?” or “!”)
  • 4.1%: immediately after “,”
  • 31.5%: other
• There are 9 hits for “ne , ne” and one for “ne ne”. (“Si maɔsɛgu ne ne, ɔrɔ̃go kpakpa gɔ mpia…”)
• “ne kama ne” appears 31 times, almost always after punctuation.
• The most common word after a sentence ending in “ne .” is “ne”.

Then there were a couple of anonymous comments on the visualizations themselves, perhaps from the above gentlemen, perhaps from others. One contrasted the left and the right periphery of ne, noting that the first was “very very branchy… indicating perhaps (a) sentence-final “ne” attaches to the preceding clause or sentence, rather than the preceding word or phrase; (b) all these words belong to some small number of open classes– they’re all nouns or adjectives, for example.” By comparison, this commenter noted, the visualization of sentence-initial “ne” is not nearly as branchy.

So what does ne do?

So much for the frequency-based observations. Based on this, what can we say about the possible functions of ne? Brett’s guess was that “it has a grammatical function. Perhaps a negation (though I may be influenced by the spelling) or a question marker.” Ray didn’t place a bet and stuck to his observations. Jason said: “I’m guessing it performs some function that is not served by any English word. Maybe it indicates past tense. (…) For a while I was thinking preposition, but it seems really inconvenient for a language to prefer that prepositions appear at the extremes of sentences.”, and came back later to add “Maybe it’s a topic marker.”

Let me first disclose my own story before considering the above hypotheses. Ne is extremely common sentence- and clause-finally. In this context, its main function is to mark stuff as the topic on which a comment will follow. That is, it is a discourse particle used by speakers to signal ‘this was background information, more will come’. Because of this, it also functions as a floorholder, signalling that this was not yet the end of the interactional turn (even though the sentence may have been grammatically complete). Here is a glossed and translated example of clause-final ne:

Si ma-ɔ-ro ɔturi ɔkpa ne,

If they-PERF-finish person parting TOPIC

ka mà-a-sɛ mà-a-pie wũ ndu

IMM they-FUT-go they-FUT-wash him water

When they have finished taking leave of the person [who died, MD], they will go and wash him

The Si … ne construction noted by Ray also is a part of this picture. You didn’t know of course, but si means ‘if’, and therefore si … ne sentences are roughly equivalent to if-then clauses. Note however that Siwu ne does not correspond to ‘then’ but rather (I would say) to the rising intonation of the ‘if’ clause in English — it backgrounds the condition and announces the ‘then’ clause.

The anonymous commenter’s note that sentence-final ne is ‘very branchy’ nicely accords with this view of ne as a discourse marker. Basically, you can tack it on any type of sentence; there are no grammatical restrictions. This makes Jason’s preposition hypothesis less likely (prepositions usually occur in well-defined contexts). His other proposal that ne ‘performs some function not served by any English word’ is not without problems; we do have topicalizing and floor-holding devices in English of course (politicians cannot utter a sentence without them). On the other hand, one could argue that these devices (front-shifting, ‘uhm‘, intonation, etc.) are not nearly as wordlike as Siwu ne (as shown for example by the fact that the former do not show up as regularly in written texts as the latter).

So there we have function number one: ne as a topicalizing discourse particle. Jason’s last guess turns out to be spot on.

Sentence-initial ne

Sentence-initial ne, arranged by frequency.

There is also a sizable number of sentences starting with ne (13,6% of all occurrences according to Jason’s statistics). About 60% of these are taken up by two collocations: ne ɔso (ne) and ne kama ne. The latter, first identified by Brett, is essentially a lexicalized construction combining clause-initial and clause-final ne, and it means something like “And after that, …”, kama meaning ‘back, behind’. The former, a combination of ne with ɔso ‘reason’ can be translated as “Therefore”. Thus, both constructions are rather similar to conjunctive adverbs in English, sometimes with a final ne added as a topicalizer announcing the comment that is to follow.

Overall, my feeling is that sentence-initial ne is very similar in function to sentence-initial and in English (for a good overview see Dorgeloh 2004). That is, it contributes to the overall textual cohesion by coordinating idea structures in discourse. It is also the most unmarked mode of connection in stories. Here is an interlinearized example of sentence-initial ne:

Ne sɔ lo-karɛ mi sɔ nda kaa se?

ne COMP I-ask you.pl COMP how home is

And he said, ‘I asked you, how is home?’

One remaining question of course is: what do sentence-initial and sentence-final ne have to do with each other, if anything? My first pass at an answer would be that both uses have to do with speaker continuation, or the ‘speaker-imposed structuring of ideas’, as Dorgeloh (2004:1763) calls it; but this is an issue that needs to be investigated more closely.

Let me just end with a final note on the nature of the corpus. It consists of a lengthy text on traditional funeral rites in Kawu (Ray, this is the part that includes the poem, actually a funeral dirge); a long narrative about Tete Kalai, son of a smith; a personal narrative by a retired preacher; and some ten shorter stories from fast track literacy participants. All writers are adult native speakers of Siwu. The texts are set in the official orthography, which doesn’t mark tone. All in all, it seems that even this first corpus, though relatively small, is large and varied enough to bring out salient grammatical structures, just as I had hoped.

Thanks to the fellow language bloggers at Language Log for bringing the challenge to the attention of its readers; and above all, a big thank you to Brett, Ray, and Jason for your invaluable input! The Siwu for ‘thank you (pl)’ literally means ‘you are the ones who did the work’. Very true. Mìlobrara!

References

1. Dorgeloh, Heidrun. 2004. Conjunction in sentence and discourse: sentence-initial and and discourse structure. Journal of Pragmatics 36, no. 10 (October): 1761-1779.
2. Sacks, Harvey, Emanuel A. Schegloff, and Gail Jefferson. 1974. A Simplest Systematics for the Organization of Turn-Taking for Conversation. Language 50, no. 4 (December): 696-735.

I recently came across Many Eyes, a nifty data visualisation tool by IBM’s Visual Communication Lab. It has lots of options to handle tabular data, but —more interesting to linguists— it can also handle free text. The two visualization options it currently offers for text are a tag cloud and a so-called ‘word tree’. The former visualizes simple token frequency, the latter displays the occurences of a given word (or phrase) in a branching view. It is the latter that I find the most exciting feature, because it allows for rapid visual exploration of linguistic patterns in a text.

Take for instance the Siwu locative marker i. Before today I vaguely knew where it usually occurs (before an NP and after a VP, more or less). Now I know (1) that it also occurs sentence initially, as in I Ɔtuka ame, ... {LOC Lolobi inside} 'In Lolobi country, ...'; (2) that it often precedes a deictic, as in ...i mmɔ {LOC there} 'over there'; and (3) that one can have nested occurrences, as in ma-sɛ ma-a-su kaku i ngbe-gɔ i ɔturi ɔ-kpi mmɔ {they-HAB they-FUT-take funeral LOC here-REL LOC person he-died there} 'they usually will hold the funeral there were ('in the place in which') the person died'. The next step is to look more carefully into these particular constructions and improve my grammatical analysis. I might conclude, for example, that the distal deictic mmɔ is more nouny than I had taken it to be.

Of course, I would have discovered these facts eventually after carefully analyzing enough Siwu texts — but the point is that right now, finding and comparing these patterns took only five minutes of playing around with the word tree above. Cool, isn't it? Let's call it visual corpus linguistics.

Some of the best options, by the way, are not available in the embedded view above. Take a look at the full version to order alphabetically or by frequence, or try the start/end radio buttons which enable you to view preceding and following words. And if you don't happen to read Siwu, why don't you check out Shakespeare's All's Well That Ends Well?

Collaborative data mining

Wait, let me retract that. It doesn't even matter that you don't know any Siwu — the power of frequency-based visualization is that you should be able to spot salient grammatical patterns anyway. It would be interesting to test this. Can you, based on this dataset of 21,891 Siwu words and punctuation marks, work out what kind of functions the word ne might have?

Perhaps I'm making it too difficult by not providing the meaning of the Siwu text, but I do think that this type of collaborative pattern hunting is an interesting addition to the toolkit for linguistic analysis. In the future, we will use visualizations like this to get a feeling for the relative frequencies of different constructions in which an item occurs, and to quickly test each other's hypotheses. The philosophy behind Many Eyes is interesting in this respect. As the About page says: Many Eyes is a bet on the power of human visual intelligence to find patterns. Our goal is to "democratize" visualization and to enable a new social kind of data analysis.

I am quite sure that there are specialized corpus linguistic applications out there that have far more sophisticated data-munching and searching capabilities. But the simplicity of Many Eyes may be its secret power. It is totally hasslefree: feed it a text and you can instantly play around with different types of visualizations. The intuitive interface makes it possible to quickly traverse the data in search of patterns, or do some quick testing of constructional hypotheses. In fact, you do not even need to upload your own dataset, because you can create visualizations based on any existing dataset (all the data is publicly available). And did I mention it is free?

Limitations

Despite all the goodness, Many Eyes is not perfect, although to be fair, it wasn't exactly made for linguistic analysis to begin with. After some playing around, you will inevitably hit upon the limitations; for me, it was a bit disappointing for instance to see that the tag cloud chokes on Unicode characters outside the basic Latin range. The following few relatively simple improvements would hugely enhance the linguistic uses of Many Eyes:

• tag cloud: proper handling of Unicode characters outside the basic Latin range.
• visualization over multiple data sets. (Right now, a workaround is to make a new dataset combining other data sets. However, this is not straigtforward.)
• word tree: positional wildcards to enable searching for patterns like Siwu "i * ame" (LOC X inside) or English "I * know" (capturing 'I don't know', 'I damn well know', etc.) . This would involve a branching tree inside two focus words, very interesting visually. It should be possibly to limit the search to N intervening words, or to search for an arbitrary number of intervening words (respecting sentence boundaries, I guess).
• word tree: show target word in full context (i.e. show the words left and right of it). Note that this is not trivial, as there is the problem of how to connect left and right parts of sentences on potential regroupings. The sort by occurrence option would no longer work in any case. Perhaps highlighting the relevant connector lines on mouseover would be a good solution.
• word tree: some way to do partial word searches, e.g. "*kɛlɛ" to find all occurrences of the verb kɛlɛ 'go' regardless of subject agreement and tense/aspect prefixes. This would involve non-trivial ordering and visualization problems (how to visualize the difference between a preceding word and a preceding part of a word? Perhaps this would add too much clutter).

Until these are implemented (either in Many Eyes or in a specialized linguistic cousin), those of us working on radically isolating languages will probably be more happy with the tool than those working on the polysynthetic end of the continuum. Be that as it may, Many Eyes remains an exceptionally useful application, offering a glance into the bright future of visual corpus linguistics. Thank you, VCL people!

(Hat tip: Ethan Zuckerman.)

Links

• About Many Eyes
• Visual Communication Lab
• Many Eyes Development Blog