How Academia.edu promotes poor metadata and plays to our vanity… and how it could improve.

Update: Now also on Academia.edu

giphyA while back some low quality citations started showing up on Google Scholar. They had titles like “CHAPTER 2 draft — email xyz@ab.edu” and it was hard find actual bibliographic metadata. Google Scholar seemed to have scraped random PDFs uploaded on Academia.edu and decided it was worth counting the citations in them even in the absence of proper metadata. I shared this on Twitter and promptly forgot about it.

Then I got an email from someone asking me to say a bit more about my concerns with poor metadata. I decided to write it up in a blog post. I’m afraid it turned into a a bit of rant about how Academia.edu seems built not so much for sharing scientific information as for playing to our vanity. Sorry about that. Let’s start with the poor metadata issue, which turns out to be rather pervasive.

Academia.edu has a massive metadata problem

  • Academia.edu doesn’t record any metadata except title and author for the bulk of papers, and doesn’t expose any metadata using standard formats like RDF/unAPI.

Ever tried to figure out how to cite a paper on Academia.edu? This is hard because most of the metadata is missing. Reference managers like Zotero or Mendeley cannot detect and save papers for citing. For those hoping to cite works uploaded there, this makes life more difficult than it needs to be. For users of academia.edu, this hurts citability. Yes, I know academia.edu touts a 69% citation advantage. See here for a discussion of some concerns about that study. My point here is simply that whoever wants to cite papers found on Academia.edu currently has to get the metadata from elsewhere.

Google Scholar has started to index the heap of PDFs on Academia, but has to resort to scraping only the most superficial info available, usually from the PDF, because there is no metadata. This is the number one reason for the junk citations in Google Scholar that started this story. It means people’s work is misrepresented and makes it harder to figure out how to cite it  — bad news, because Google Scholar is very widely used. For users of academia.edu, this hurts both the findability and the citability of their work. For users of Google Scholar, this adds more noise in an already noisy system.

Screenshot 2016-08-26 09.30.36

  • Academia.edu is built for single-authored papers, and its handling of multi-authored papers is surprisingly poor. 

The default way of scraping author names leads to many errors and they can only be fixed manually. Take the paper Academia.edu staff published on ‘discoverability’ — the authors are all jumbled up. Only the original uploader owns the item and can add or fix bibliographic metadata, and for other authors, it’s hard to see who’s the owner. There is no system for duplicate detection and resolution. It is too easy for multiple authors to upload the same paper with slight differences in bibliographic metadata. It is too hard to clean up the mess and make sure there is only one good  version of record. This affects people’s profiles and has undesirable knock-on effects for the points above.

  • The process of adding papers is geared towards enriching Academia.edu content rather than towards promoting the sharing of correct and complete scientific information.

After Academia.edu gets your PDF (and that’s a requirement for adding a paper), there are very few opportunities for providing metadata, and the primary upload interface cares more about fluff like ‘research interests’ than about getting the basic bibliographic metadata right. There is no way to import via DOI or PMID (which would prevent many errors), or even to record these identifiers — a fatal lack of concern for interoperability which is quite surprising. Essentially, a user interface should make it easy for people to get things right, and hard to get them wrong. The current user interface for adding papers does the exact opposite (see annoted screenshot).

Screenshot 2016-08-25 10.19.35
  • It is surprisingly (and needlessly) hard to add crucial bibliographical information like journal, DOI and URL.

More details can only be added after importing papers, which simply means most users won’t do it. As far as I can see, the only way to do it is to go back to your publication list, hover over the Edit button, and find other fields to edit. Even here, there appears to be no place for identifiers like DOI or PMID. Page numbers and so on are hidden in an “Other” field.  Any user interface designer will tell you that stuff buried this deeply might as well be left out: only a negligible amount of users will find and use it. Anyway, if you’ve succeeded in adding some of this metadata, congratulations for completing a futile exercise. The information you painstakingly entered is nowhere exposed and so cannot be reused or exported, except, again, manually. Quite remarkable in the age of APIs and interoperability.

How Academia.edu nudges us towards narcissism

My conclusion from these points: Academia.edu seems not to care about promoting the curation and sharing of correct and high quality metadata of scientific publications. One might counter that this is not the goal of the network, and that the content of the papers is what’s most important anyway. But peer-reviewed publications are still the main vehicle for advancing scientific results, and citations are still the main currency of cumulative science. So getting bibliographic metadata right is key to promoting science as a cumulative enterprise. Nor should this be hard in the era of DOIs and PMIDs, making it all the more surprising Academia.edu doesn’t care about them.

If there really is a problem, why do relatively few people complain and why are so many users seemingly happy about the service? There are several reasons. Not every academic has access to personal website or an open academic repository, and Academia.edu presents itself as one of the easiest options to make one’s work visible online (never mind the fact that it doesn’t actually make it easily citable, and laces it with ads to boot). It may be a  way to keep up with colleagues. Also, I’ve heard people are happy with its “sessions” as a way to get interactive feedback on a paper. But there’s one important reason that I haven’t seen commented upon often: Academia.edu plays to our vanity. Many elements of its design are built to satisfy and amplify our craving for external validation.

Judging from the navigation menu, “analytics” is one of the most important elements of Academia.edu. Upload papers, tag them with research interests, and they generate paper views. Follow people and they’ll follow you back, generating profile views. Tomorrow your paper may be in the top 5%! Next week you might be crowned as the 1%! Look, your paper was just read by someone from Vienna! Your work is being read in 27 countries! You’re being followed by someone you barely know! All those things are nicely presented in spiffy graphs — evidently a part of Academia.edu that a lot of design resources have been devoted to.

And note some of the design here is cynical. The only two time windows offered are 30 days and 60 days, inviting you to come back at least this often to keep up with the stats (yes, you can download a CSV for more, but once again that is one of those power user features that will rarely be used). Views are promoted over actual downloads while bounce rates (basically, how many people are gone after a quick glance, usually the majority) are concealed. The most important metadata for papers (again, just taking the design as a measure of what Academia.edu promotes as important) is this mostly meaningless view count. Not where it was published, not how to cite it, certainly not where to find it off Academia.edu — just how many people had a look.

Academia.edu doesn’t take academia seriously

Does this mean everybody on Academia.edu is a narcissist? Of course not. My point is not about users; it is about the design of the service. User interface design is not innocent: as a recent Medium essay noted, technology hijacks our minds, constraining our options and nudging us in ways that often elude our awareness. Not everybody on Academia.edu is a narcissist, but many aspects of its design make it easy to become one. (The emails! Don’t get me started about the emails. By default, Academia.edu will send you an email whenever someone stumbled upon your profile or one of your papers. Just look at this Twitter feed to see how creepy people find that feature. You might even spot a few who have come to like it, Stockholm syndrome-style.)

On balance, I feel Academia.edu doesn’t really take us seriously as academics. It takes our work to make a profit (for instance by putting advertisements around it), totally botches the metadata and tries to appease us by offering stats and social rankings that promote constant comparison. And nothing in its design suggests a regard for getting even the most basic bibliographic information about our scientific work right — even though that would be one way to turn page views into citations. This is one of the reasons the only paper I’ve uploaded there for years has been one pointing people to where they can find all my papers freely and without hassle.

To end on a slightly more optimistic note: at least the poor metadata problem can be solved. As far as I can see, nothing in Academia.edu’s business model turns on proliferating poor and incomplete metadata. The citation advantage it likes to claim could significantly increase if it started exposing metadata in ways that are compatible with widely used tools like Google Scholar and Zotero. It still won’t be a service I’m keen on using, but I do hold hope it will become better at promoting cumulative science rather than cynically playing to our vanity.

Sound symbolism boosts novel word learning: comparing ideophones and adjectives

We have a new paper out. It’s actually been available since February in an online-first version, but for those of us who love page numbers and dead trees, the journal has now printed it in its August issue on pages 1274-1281. Citation:

Lockwood, G., Dingemanse, M., & Hagoort, P. (2016). Sound-symbolism boosts novel word learning (PDF). Journal of Experimental Psychology: Learning, Memory, and Cognition, 42(8), 1274-1281. doi:10.1037/xlm0000235

This is another one for which we’ve made available the stimuli —word lists and sound files— through OSF, contributing to our mission to make research from our lab replicable. Also, we have since replicated the results in a follow-up study where we also took EEG and individual difference measures.

I provide a quick summary in 2×3 points below. For a write-up that’s much more fun and has great illustrations, check out Gwilym Lockwood’s Sound symbolism boosts novel word learning: the MS Paint version.

  1. We test how people learn real words from an unfamiliar language. If you give people a choice (does bukubuku mean ‘slim’ or ‘fat’?), most will get it right. But in real life we are rarely given such choices; instead, we learn associations, some of which may stick better than others. Does sound symbolism boost word learning?There is some evidence it does, but this is mostly from studies in which people learn artificial words. The real litmus test of the power of sound symbolism is to give people a learning task with real, existing words. We test how well Dutch people learn the meanings of words from Japanese, a language that is unfamiliar to them.
  2. We contrast ideophones and adjectives. Ideophones are said to be sound-symbolic, i.e. to show iconic associations between sound and meaning. However, evidence for this so far has mostly come from studies without a strong control condition (which compare ideophones with nonwords). We want to be sure that we are tapping into whatever may be special about ideophones as a word class, so we compare the learning of ideophones and adjectives from one and the same language. This helps us tease apart general ease (or difficulty) of learning from a boost that may be specifically due to sound symbolism. We expected that ideophones are significantly easier to learn than adjectives.
  3. We swap meanings to see if we can make and break the effect of sound symbolism. Prior work has shown that sound-symbolic cues may get you in the rough ballpark of the semantic domain (e.g., size) but may not be enough to distinguish between opposite meanings (e.g., small vs. large). We have people learn some words with their real translation and others with their opposite translation. If there is an arbitrary connection between form and meaning (as in most adjectives), this should make no difference to how easy or hard they are to learn. But if there is an iconic connection that helps you learn a word (as in most ideophones), breaking this connection may well make it harder to learn them.
JEP2

Key figures from the paper

And here are our main findings:

  1. Sound-symbolism boosts word learning. The sound-symbolic bootstrapping effect, as it has come to be known, extends to real lexical words from existing languages. This is an important validation of the earlier findings based on pseudowords.
  2. Ideophones are easier to learn than adjectives. Of the ideophones, 86% are recalled correctly; for the adjectives, this is only 79%. So all words can be learned (fortunately! otherwise what would be the point of having them?), but some are significantly easier to learn than others. The ones that are easier to learn are ideophones, i.e. those words that have iconic associations between form and meaning.
  3. Ideophones with their opposite meaning are harder to learn than adjectives. If we sever the connection between form and meaning, this doesn’t affect the learning of adjectives: it doesn’t make it easier or harder. This is what we expect of arbitrary words. But doing the same thing to ideophones does affect learning: ideophones presented with their opposite meaning are harder to recall than ideophones with their real meaning. So not only does the special association between form and meaning in ideophones provide a learning advantage; reversing the association results in a learning difficulty.

Since this study came out, it’s been featured on the Lingspace vlog as well as in several other media — see its altmetric.com page.

Why PLOS ONE needs page proofs

Note: I prepared this posting in August 2015, when PLOS ONE was due to publish a paper by us and I wanted to make sure they’d avoid the stupid typesetting errors they made in our 2013 paper. I used the numbers to convince them to show us proofs beforehand. To my surprise, they did, and I never got around to finishing the draft piece I had in the making. This week the issue flared up again following a comment by Dorothy Bishop, so I’ve decided to unearth my draft blog post and put it online.

Update: thanks Retraction Watch for giving some attention to this issue: PLOS ONE’s correction rate is higher than average. Why?

PLOS ONE notoriously and astonishingly does not have a proofs stage: authors do not get to see how their work is typeset until the very day it appears. If they spot typesetting errors then, it is too late: PLOS ONE has a policy of not correcting formatting errors (recent example here). Note that I’m not talking about typos but about typesetting errors: problems introduced by the publisher deviating from the authors’ proofed manuscript. In the most egregious cases, they will republish the article, slapping a correction notice on it. The correction notice may mention that the errors were “introduced during the typesetting process” and that “the publisher apologizes for this error” (as for our 2013 paper). But nobody reads correction notices, while everybody gets to see the article was corrected. For many readers a correction is an index of sloppiness at least and bad science at worst. Unnecessary corrections hurt authors.

Let me be clear: Corrections can be useful and important. If there are true errors of fact in a publication, a correction is an important way to set things straight while also keeping the scientific record intact. Readers can trace an earlier version, but the new version is designated as the definitive one. However, at PLOS ONE, that useful function of corrections is fast being diluted by many corrections that simply fix errors introduced in the typesetting stage. PLOS ONE thereby punishes authors for errors not introduced by the authors and makes the publication record unnecessarily messy.

Many corrections are for publisher errors

In the period May-July 2015, PLOS ONE published a total of 474 corrections. Over a quarter of these (132) indicate that the error was introduced at the typesetting stage, i.e. beyond the control of the authors.. The remainder did not include that phrase, but most were trivial errors that would have been caught by the authors if there had been a proofs stage: missing information from Funding sections, swapped figures, and incorrectly typeset equations. Actually some of these may also be publisher errors. Here is a correction of a correction noting, “This article was republished on July 27, 2015, to change the byline of the correction, which should have been The PLOS ONE Staff, and to note the publisher as responsible for the error.” The original error was one of omitting Funding information.

In the same three-month period in 2015, PLOS ONE published 8466 articles whose title does not include “Correction:”. Ignoring for a moment that some corrections in this period may be for articles that go further back, we see that in this quarter of 2015, PLOS ONE issued corrections for over 5% of its publication output. That seems rather a lot. Was this a special period? In the first six months of 2015, there were 925 corrections, 193 (or 21%) of which indicate publisher error. In that period, the rate of corrections to real publications is 7% (925 over 14252). The rate of publisher errors seems to be rising. In 2014, 13% of corrections indicate publisher error (and the ratio of corrections to publications was 6%). In 2013, 4% of corrections indicate publisher error (and the ratio of corrections to publications was 5%).

Update Aug 2016: Here are the numbers for the whole of 2015: 30970 research articles across all PLOS journals, 1939 corrections (6.3% of publication output), of which 415 acknowledge publisher error (21.4% of corrections). And here’s 2016 so far: 15162 articles, 794 corrections (5.2%) of which 154 are publisher error (19.4% of corrections). So over the last 1.5 years, a full 6% of all PLOS publication output has received corrections, and at least one fifth of these are due to publisher errors beyond the control of authors. Keep in mind authors are essentially powerless and many don’t request corrections, so the problems are likely much worse.

Authors deserve better

PLOS ONE asks publication fees “to offset expenses—including those of peer review management, journal production and online hosting and archiving”. It seems to me that for a fee of $1495, authors can expect to get a modicum of quality control in typesetting, which would fall under journal production. They haven’t been getting it so far. This ought to change.

I love PLOS ONE for its visionary publishing model and its open access philosophy. Though some have pointed out quirks in its editorial processes (e.g. “Creatorgate”), for the papers I’ve had under review, editors were fast and reviewers tough yet fair. It pains me to conclude that they are letting authors down when it comes to the final stage of publication. Our work deserves to be published according to the same standards of rigour that hold prior to publication. By not exercising due diligence in the journal production process, PLOS ONE is hurting its authors and ultimately damaging its own reputation.

What sound symbolism can and cannot do: new paper in Language

What sound symbolism can and cannot doWe have a new paper out in Language:

Dingemanse, Mark, Will Schuerman, Eva Reinisch, Sylvia Tufvesson, and Holger Mitterer. 2016. “What Sound Symbolism Can and Cannot Do: Testing the Iconicity of Ideophones from Five Languages.” Language 92 (2): e117–33. doi:10.1353/lan.2016.0034

The basic finding is this: people are sensitive to the meaning of ideophones they’ve never heard, even when they are produced out of context by a computer voice in a difficult forced choice task. Yet there is also reason for caution: the effect is not nearly as strong as what people have found for pseudowords like bouba and kiki.

As we note in the introduction, “there appears to be a tendency to either underplay or exaggerate the significance of iconicity in the study of language and mind”. In this paper we chart a middle way between these extremes. Here’s a quick summary in 3×3 points:

What we did:

  1. Sound symbolism (iconicity in spoken language) is usually studied using hand-crafted pseudowords in binary forced choice experiments (think bouba and kiki, as reviewed here), but there are three problems with such experimental designs: (i) they run the risk of inflating effect sizes, (ii) it is unclear how they relate to natural languages, and (iii) they usually don’t control for prosody.
  2. We designed a study to tackle these problems by (i) adjusting the binary choice task to be more realistic and harder, (ii) using real words and meanings from natural languages, and (iii) teasing apart prosody and segmental features. Essentially, we bring linguistic insights to bear on the psychological study of sound symbolism.
  3. We take 203 ideophones —lexical sound-symbolic words— from 5 languages and 5 semantic domains and present them to 80 participants in 4 versions: (i) full original recording, (ii) full speech synthesized version, (iii) prosody-only condition and (iv) phonemes-only condition. The versions help us control for variation due to different speakers and help us examine the contributions of prosody and segmental features.

What we found:

  1. People can choose the correct translation of ideophones at a level significantly above chance. So ideophones in Japanese, Korean, Semai, Ewe and Siwu are not fully arbitrary, as is normally assumed of words; they contain iconic cues that even people who don’t speak these languages can pick up.
  2. Sound ideophones are easiest to guess, but the other semantic domains (movement, texture, color/visual appearance, and shape) come out significantly above chance as well. However, the effect is much more modest than most bouba/kiki studies: in the best versions, people score 57.2% on average (where 50% would be chance level) — quite different from the 95% that has sometimes been claimed for pseudoword studies.
  3. Performance for the original and resynthesised stimuli is indistinguishable, so our speech synthesis method works. Performance is significantly better for the full versions (i-ii) than for the reduced versions (iii-iv), so both prosody and phonemes contribute to the effect (and neither alone is sufficient).

What we conclude:

  1. Findings based on pseudowords like bouba/kiki cannot be automatically translated into claims about natural languages. Ideophones combine iconicity and arbitrariness, and lexical iconicity in ideophones is best characterised as a weak bias, which is supported by multimodal performances in actual use and which may be amplified in cultural evolution (cf our TiCS paper).
  2. Prosody is just as important as segmental information in supporting iconic interpretations (as predicted here). Prior work, which has rarely controlled for prosody, likely overestimates the role of speech sounds and underestimates the role of intonation, duration and other prosodic cues.
  3. Speech synthesis offers a viable way to achieve experimental control in the study of sound symbolism. To stimulate its wider use we’re making available all stimulus materials, including the diphone synthesis source files we used to create them. Get them at MUSE or OSF.
Here’s the abstract:

Sound symbolism is a phenomenon with broad relevance to the study of language and mind, but there has been a disconnect between its investigations in linguistics and psychology. This study tests the sound-symbolic potential of ideophones—words described as iconic—in an experimental task that improves over prior work in terms of ecological validity and experimental control. We presented 203 ideophones from five languages to eighty-two Dutch listeners in a binary-choice task, in four versions: original recording, full diphone resynthesis, segments-only resynthesis, and prosody-only resynthesis. Listeners guessed the meaning of all four versions above chance, confirming the iconicity of ideophones and showing the viability of speech synthesis as a way of controlling for segmental and suprasegmental properties in experimental studies of sound symbolism. The success rate was more modest than prior studies using pseudowords like bouba/kiki, implying that assumptions based on such words cannot simply be transferred to natural languages. Prosody and segments together drive the effect: neither alone is sufficient, showing that segments and prosody work together as cues supporting iconic interpretations. The findings cast doubt on attempts to ascribe iconic meanings to segments alone and support a view of ideophones as words that combine arbitrariness and iconicity. We discuss the implications for theory and methods in the empirical study of sound symbolism and iconicity.

Some things you need to know about Google Scholar

Summary: Google Scholar is great, but its inclusiveness and mix of automatically updated and hand-curated profiles means you should never take any of its numbers at face value. Case in point: the power couple Prof. Et Al and Dr. A. Author, whose profiles I created following Scholar’s recommended settings (and a bit of manual embellishment). If you have a Scholar profile, make sure you don’t let Scholar update the publication list automatically without checking and cleaning up regularly. If you’re looking at somebody else’s profile, take it with a big pinch of salt, especially when they have a reasonably common name or when duplicate entries or weird citation distributions indicate that it is being automatically updated. 

Update July 1st: Google Scholar has now manually blocked Prof. et al. from appearing in top rankings for her disciplines. They probably thought her too prominent a reminder of the gameability of their system (how long will it take before they silence her next of kin?). This doesn’t solve the real problem, noted below, of auto-updating profiles like Yi Zhang and John A. Smith diluting top rankings. In fact, even in scientometrics, it looks like there are at least 3 or 4 auto-updating profiles in the top 10.

I love Google Scholar. Like many scientists, I use it all the time for finding scientific literature online, and it is more helpful and comprehensive than services like PubMed, Sciencedirect, or JSTOR. I like that Google Scholar rapidly delivers scholarly papers as well as information about how these papers are cited. I also like its no-nonsense author profiles, which enable you to find someone’s most influential publications and gauge their relative influence at a glance. These are good things. But they are also bad things. Let’s consider why.

Three good things about Google Scholar

  1. Google Scholar is inclusive. It finds scholarly works of many types and indexes material from scholarly journals, books, conference proceedings, and preprint servers. In many disciplines, books and peer-reviewed proceedings are as highly valued and as influential as journal publications. Yet services like Web of Science and PubMed focus on indexing only journals, making Google Scholar a preferred tool for many people interested in publication discovery and citation counts.
  2. Its citation analysis is automated. Citations are updated continuously, and with Google indexing even the more obscure academic websites, keeping track of the influence of scholarly work has become easier than ever. You can even ask Scholar to send you an email when there are new citations of your work. There is very little selection, no hand-picking, and no influence from questionable measures like impact factor: only citations, pure and simple, determine the order in which papers are listed.
  3. Its profiles are done by scholars. No sane person wants to disambiguate the hundreds of scholars named Smith or clean up the mess of papers without named authors, titles or journals. Somebody at Google Scholar had the brilliant idea that this work can be farmed out to people who have a stake in it: individual scholars who want to make sure their contributions are presented correctly and comprehensively. So while citations are automated, the publication lists in Google Scholar profiles are at least potentially hand-curated by the profile owners. Pretty useful. But wait…

Three bad things about Google Scholar

The classic 'Title of paper', 1995

  1. Google Scholar is inclusive. It will count anything that remotely looks like an article, including the masterpiece “Title of article” (with 128 citations) by A. Author. It will include anything it finds on university web domains, so anyone with access to such a domain can easily game the system. Recently it has started to index stuff on academia.edu, a place without any quality control where anybody can upload anything for dissemination.
  2. Its citation analysis is automated. There are no humans pushing buttons, making decisions and filtering stuff. This means rigorous quality control is impossible. That’s why publications in the well-known “Name of journal” are counted as contributing bona fide citations, and indeed how “Title of article” can have 128 citations so far. It’s also why the recent addition of academia.edu content has resulted in an influx of duplicate citations due to poor metadata.
  3. Its profiles are done by scholars. Scholars have incentives to appear influential. H-indexes and citation counts play a role in how their work is evaluated and enter into funding and hiring decisions. Publications and co-authors can be added to Google Scholar manually without any constraints or control mechanism, an opportunity for gaming the system that some may find hard to resist. But forget malicious intent: scholars are people, and people are lazy. If Google Scholar tells them it can update their publications lists automatically, they’ll definitely do so — with consequences that can be as hilarious as harmful, as we’ll see below.

To illustrate these points, let’s have a look at the Google Scholar profiles of two eminent scholars, Dr. Author and Prof. Et Al.

Dr. Author

Enter dr. A. AuthorRanking second in the field of citation analysis, his h-index is 30 and he has over 3500 citations. Among his most influential papers are “Title of article” with 159 citations and “Title of paper” with 128 citations to date. It is a matter of some regret to him that his 1990 “Instructions to authors” has been less influential, but perhaps its time is yet to come. Dr. Author is active across a remarkable range of fields. He likes to write templates, editorials, and front matter but has also been known to produce peer-reviewed papers as well. His first name is variously spelled Andrew, Albert or Anonymous, but most people just call him “A.” and Google Scholar happily accepts that.

Dr. Author reminds us that Google Scholar citations are done by an automated system, and so will be necessarily noisy. His profile simply gathers anything attributed to “A. Author”, a listing that is automatically updated in accordance with Google Scholar’s recommended settings. How pieces like “Title of article” can accrue >100 citations is a bit of a mystery, especially since only a few of the citing articles are other templates. Some of A. Author’s highly cited papers seem to be due to incomplete metadata from the source; others seem to be simply misparses; some are correct in the sense that editorials are often authored by “anonymous author”. At any rate, this shows there are a lot of ghost publications and citations out there, some of which may easily be attributed to people or publications they don’t belong to.

But surely these are just quirks due to bad data — garbage in, garbage out, as they say. Actual scientists maintaining a profile can count on more reliable listings. Or can they?

Prof. Et Al

Enter prof. Et Al. With an h-index of 333 and over 2 million citations, she is the world’s most influential scientist, surpassing such highly cited scholars as Freud, Foucault, and Frith (what is it with F?). She has an Erdős number of 1 and ranks first in the disciplines of scientometrics, bibliometrics, quality control and performance assessment; in fact in any discipline she would care to associate herself to. How did she reach this status? Simply by (i) creating a profile under her name, (ii) blindly adding the publications that Google Scholar suggested were hers; (iii) allowing Scholar to update her profile automatically, as recommended. Oh, and just because Google Scholar allows her to, she also manually added some more papers she was sure she wrote (including with her good friend Paul Erdős).

Prof. Al reminds us that Google Scholar profiles are made by scholars. Scholars, being people, are mostly well-intentioned — but they can also be unsuspecting, lazy or worse. Prof. Al started out by simply doing what most scholars do when they create a new profile: following the instructions and recommended settings. If you do this blindly, Google Scholar will just add anything to your profile that comes remotely close to your name, and there is almost a guarantee that you’ll end up with a profile that way overestimates your scientific contributions.

Real-life examples

It is not that hard to find real examples of profiles getting a lot of extra padding because of Scholar’s automatic updating feature. Take Yi Zhang at Georgia Tech, who must surely be the most accomplished PhD student ever with 40.000+ citations and an h-index of 70. This is Google Scholar’s recommended “automatic updating” feature going bananas with what must be a very common name. Indeed, there is another Yi Zhang, ranking 4th in syntax just after Chomsky, Sag, and Kiparsky. His top cited paper has 306 citations and yet the sum of his work —a well-rounded total of 1000 publications— has somehow received over 23,000 citations. (Note that #5 and #6 in syntax are also auto-updating profiles.)

All this is mostly harmless fun, until you realise that a profile may be claiming the publications and citations of another one without either of them noticing. Case in point: the profile of Giovanni Arturo Rossi, an expert on respiratory diseases, is consistently hoovering up publications by my colleague Giovanni Rossi, who works on social interaction. Scholar auto-links author names to profiles in search results, preventing people from finding the real Rossi from his publications unless he actively and manually adds those Arturo-claimed publications to his profile.

Bottomline: if you have a common name, you’ll have to take control of every new publication manually, since otherwise Rossi (or Smith, or Zhang) is going to get it added automatically to their profile. Also, if you have a common name and you blindly follow Google Scholar’s recommended settings, you may be very pleased with your h-index, but probably for the wrong reasons (hello there John A. Smith, independent scholar, 23428 citations, h-index 64!). So my most general recommendation would be: don’t let Google Scholar update your profile automatically, and if you must, clean up regularly to avoid looking silly.

Know what you’re doing

So far, the examples arise simply from Google Scholar’s recommended setting to automatically update publication lists. It doesn’t look like any of these authors (well, except maybe dr. Author and prof. Et Al) have done anything like actively adding publications that aren’t theirs, or claiming they’ve worked with Paul Erdős. But here’s the thing: these things are not just possible, they are really easy, as prof. Et Al’s superstar profile shows. And with hundreds of thousands of active profiles, there’s bound to be some bad apples there.

What are the consequences? Nothing much if you take Google Scholar for what it is: a useful but imperfect tool. Yet many take it more seriously. If you’re in the business of comparing people (for instance while reviewing job applications or when looking for potential conference speakers), the metrics provided by Google Scholar are some of the first ones you’ll come across and it will be very tempting to use them. There is even an r package that will help you extract citation data and compare scholars based solely on citation numbers and h-indexes. All this is perilous business, considering these ranks are diluted with auto-updating ghost profiles.

Let me end by reiterating that I love Google Scholar and I use it all the time. It can be a tremendously useful tool. Like all tools, it can also be misinterpreted, misused and even gamed. If you know what you’re doing you should be fine. But if you think you can blindly trust it, take another look at the work of dr. A. Author and prof. dr. Et Al.

Notes

The “A. Author” and “Et Al” profiles were created in June 2016 by Mark Dingemanse to illustrate the points made in this post. Thanks to Seán Roberts for suggesting that A. Author should co-author with Et Al. Just in case Google Scholar follows up with some manual quality control and some of these profiles or publications disappear, screenshots document all the relevant profiles and pages.

There is something of a tradition of creating Google Scholar profiles to make a point; see here and here, for example. While my goal here is simply to promote mindful use of technology by noting some problems with Google Scholar profiles (as opposed to citations, the focus of most prior research), let me note there is of course a large scholarly literature in bibliometrics and scientometrics on the pros and cons of Google Scholar. Google Scholar Digest offers a comprehensive bibliography.

Arbitrariness, iconicity and systematicity in language

arbicosysJust out in Trends in Cognitive Sciences: a review paper by yours truly with Damián Blasi, Gary Lupyan, Morten Christiansen and Padraic Monaghan. It is titled “Arbitrariness, iconicity and systematicity in language”. You can download it here (PDF). Here is a simple summary:

An important principle in linguistics is that words show no predictable relation between their form and their meaning (arbitrariness). Yet this principle does not have exclusive reign. Some words have forms that suggest aspects of their meaning (iconicity). Some groups of words have subtle statistical properties that give away something about their grammatical function (systematicity). To fully explain how words work, we need to recognise that the principle of arbitrariness is not the whole story, and that words can additionally show degrees of iconicity and systematicity.

Here are some of the main points made in the paper:

  1. Often, arbitrariness is thought to be not just necessary but also sufficient to explain how words work. The paper shows this is not the case: non-arbitrary patterns in language are more common than assumed, and they have implications for how we learn, process and use language.
  2. Often, arbitrariness and iconicity are pitted against each other. The paper shows this is an oversimplification: iconic words have a degree of arbitrariness and the two do not exclude each other.
  3. Often, the role of iconicity in language is thought to be minimal. The paper shows that can differ dramatically across languages and also varies as a function of meaning and modality (e.g. signed or spoken).
  4. Sometimes, iconicity and systematicity have been confused. The paper shows that distinguishing them helps us to better understand vocabulary structure, by showing why we may expect iconicity to show certain universal patterns while systematicity allows more language-specific patterns.
  5. Sometimes, we may forget that words are not abstract ideas but tools that have their own history. The paper argues that the way words are learned and used influences their form, and that this may help explain how arbitrariness, iconicity and systematicity pattern the way they do.
  6. Sometimes, language scientists make far-reaching claims based on studying a small portion of the vocabulary, or a small number of (typically Western) languages. The paper argues that we can get a better picture of language by looking at a wider range of evidence.
Dingemanse, Mark, Damián E. Blasi, Gary Lupyan, Morten H. Christiansen, and Padraic Monaghan. 2015. “Arbitrariness, Iconicity and Systematicity in Language.” Trends in Cognitive Sciences 19 (10): 603–15. doi:10.1016/j.tics.2015.07.013.

Folk Definitions in Linguistic Fieldwork

Folk definitionsAnother extensively revised chapter from my thesis sees the light: Folk definitions in linguistic fieldwork. In which I discuss a procedure that is part of many field work routines, but seldomly appreciated as a method of its own. Abstract:

Informal paraphrases by native speaker consultants are crucial tools in linguistic fieldwork. When recorded, archived, and analysed, they offer rich data that can be mined for many purposes, from lexicography to semantic typology and from ethnography to the investigation of gesture and speech. This paper describes a procedure for the collection and analysis of folk definitions that are native (in the language under study rather than the language of analysis), informal (spoken rather than written), and multi-modal (preserving the integrity of gesture-speech composite utterances). The value of folk definitions is demonstrated using the case of ideophones, words that are notoriously hard to study using traditional elicitation methods. Three explanatory strategies used in a set of folk definitions of ideophones are examined: the offering of everyday contexts of use, the use of depictive gestures, and the use of sense relations as semantic anchoring points. Folk definitions help elucidate word meanings that are hard to capture, bring to light cultural background knowledge that often remains implicit, and take seriously the crucial involvement of native speaker consultants in linguistic fieldwork. They provide useful data for language documentation and are an essential element of any toolkit for linguistic and ethnographic field research.

Dingemanse, Mark. 2015. “Folk Definitions in Linguistic Fieldwork.” In Language Documentation and Endangerment in Africa, edited by James Essegbey, Brent Henderson, and Fiona McLaughlin, 215–38. Amsterdam: John Benjamins. (PDF)

Bruner on language learning

Quote

Jerome Bruner (who turns 100 today!) writes in his 1983 autobiography (emphasis in original):

“How puzzling that there should be so much emphasis … on the underlying genetic program that makes language acquisition possible and so little on the ways in which the culture, the parents and more “expert” speakers (including other, older children) help the genetic program to find expression in actual language use. The educational level of parents deeply affects how well, richly and abstractly their children will talk (and listen). It is not just the grammar of sentences that is at issue, but discourse, dialogue, the capacity to interpret spoken and written language.

In the end, I came to the conclusion that the need to use language fully as an instrument for participating in a complex culture (just as the infant uses it to enter the simple culture of his surround) is what provides the engine for language acquisition. The genetic ‘program’ for language is only half the story. The support system is the other half.”

Three decades later, proposals for the other half, what Bruner calls “the engine of language acquisition”, have become increasingly well-articulated and supported by rich empirical data (cf., for instance, all the research reviewed in Tomasello’s (2008) Constructing a language). But the two halves (genetic underpinnings and cultural scaffolding) are still not regularly talking to each other. Indeed they’re frequently pretending that the other half has no story at all… Why?

Bruner, Jerome S. 1983. In Search of Mind: Essays in Autobiography. Alfred P. Sloan Foundation Series. New York u.a: Harper [and] Row.

Pragmatic Typology: invited panel at IPrA 2015 in Antwerp

Together with Giovanni Rossi I’ve organised an invited panel at the 14th International Pragmatic Conference in Antwerp, July 2015. Contributors include Jörg Zinken & Arnulf Deppermann; Sandy Thompson & Yoshi Ono; Stef Spronck; Giovanni Rossi, Simeon Floyd, Julija Baranova, Joe Blythe, Mark Dingemanse, Kobin Kendrick & N.J. Enfield; Ilana Mushin; and Mark Dingemanse. More information here.

IPRA Pragmatic Typology Panel

Hockett on open-mindedness in the language sciences

Hockett's design features (1960 version)Charles F. Hockett (1916-2000) is well-known for his work on the design features of language. Many linguists will know his 1960 article in Scientific American in which thirteen design features are nicely illustrated (though Hockett himself preferred the more developed 1968 version co-authored with Altmann).

Hockett worked in many areas of linguistics, from phonology to morphology and from linguistic anthropology to semantics. One of his later books — which I came across while doing research for our new book series Conceptual Foundations of Language Science — has the intriguing, slightly cumbersome title  “Refurbishing our Foundations: Elementary Linguistics from an Advanced Point of View”.

In this book, written towards the end of a long career, Hockett takes a birds’ eye view of the field of linguistics and presents his own perspective, which is often sensible, sometimes a bit idiosyncratic, and always interesting. The introduction is pleasantly constructive, in contrast to some other approaches (Hornstein, “Against open-mindedness” comes to mind). Hockett’s observations on the “eclipsing stance” are as relevant today as they were three decades ago. So here is Hockett on open-mindedness:

No one in any culture known to us denies the importance of language. Partly because it is important, partly just because, like Mount Everest, it is there, we should like to know how it works. To that end, people from time immemorial have examined it or speculated about it, trying to come up with cogent commentary.

What one sees of language, as of anything, depends on the angle of view, and different explorers approach from different directions. Unfortunately, sometimes they become so enamored of their particular approach that they incline to scoff at any other, so that instead of everybody being the richer for the variety, everybody loses. That attitude has been called the “eclipsing stance.”

The early followers of Noam Chomsky adopted this stance, but they were by no means the first: some of us post-Bloomfieldians came close to it in the 1940s (though Leonard Bloomfield himself never did), and so, apparently, did the Junggrammatiker in the late 1870s. But it is a wrong position to take, even toward those who have themselves assumed it. It is obviously impossible to see all of anything from a single vantage point. So it is never inappropriate to seek new perspectives, and always unseemly to derogate those favored by others. Or, to use a different figure: the blind man touching the tail has reason to say an elephant is like a rope, but no right to claim an elephant is not also like a wall or a tree-trunk or a snake.

I don’t mean we shouldn’t be critical. I do mean we should try to be most wary just of those propositions that we ourselves hold, or have held, closest to our hearts — above all, those we come to realize we have been taking for granted. Scientific hypotheses are formulated not to be protected but to be attacked. The good hypothesis defends itself, needing no help from enthusiastic partisans.

References

  • Hockett, Charles F. “The Origin of Speech.” Scientific American 203, no. 3 (1960): 89–96.
  • Hockett, Charles F., and Stuart A. Altmann. “A Note on Design Features.” In Animal Communication: Techniques of Study and Results of Research, edited by Thomas Sebeok, 61–72. Bloomington: Indiana University Press, 1968.
  • Hockett, 1987, Refurbishing our Foundations. Amsterdam: John Benjamins.