Jekyll2024-08-26T09:33:53-04:00https://hackingsemantics.xyz/feed.xmlHacking semanticsA Natural Language Processing blog: research findings, methodology squibs, and thoughts on peer review.Anna RogersOn AI-assisted writing in graduate school2024-08-24T18:00:00-04:002024-08-24T18:00:00-04:00https://hackingsemantics.xyz/2024/writingThis post is specifically about AI-assisted writing in graduate school. Not covered: writing by experts. Also not covered: other uses of AI assistance, such as coding.

LLMs have certainly ‘disrupted’ education. The historically underpayed and under-respected teachers now have to contend with the fact that cheating on their assignments has become a lot easier, and currently impossible to detect. What is worse, it is even kind of implicitly encouraged: the high-ranking techies talk about how the school teaches outdated skills - not even the coding needs to be learned any more. What follows is that students should stop learning how to do things that LLMs (seem to) do, and “get ahead by using AI”.

Where graduate student writing is concerned, this just entirely misses the point. I found myself repeating the points below so often for my students that I wrote them up. I hope that this might be of use to others.

The product of your education is not your thesis, it’s yourself

The point of school, including graduate school, has always been to produce not more texts, but more highly-skilled people. Not just more knowledgeable, but also more skilled at thinking, synthesizing, assessing evidence. The texts are primarily a byproduct of this process. Yes, we are writing research papers and theses – but the main product of a Master and especially a PhD thesis is the student with a new set of knowledge and cognitive skills. That is why the higher-paying jobs often require more education.

Aren’t we also meant to advance research? Yes, but if that was the only goal, we would just have more professional research institutions, with full-time experts writing most of the research papers. They’d work faster, formulate more mature questions, and probably implement them better. As it stands, PhD students are a big part of the both author and reviewer pool in NLP and machine learning conferences. Hence, their papers need to perform both functions.

In machine learning in particular, sometimes graduate degrees are also confused with acquiring a set of practical skills like modeling in PyTorch or experience with some specific kind of model or task. That, too, is only a byproduct, the specific case study that the student has worked on while acquiring their new thinking skills. This field is moving so fast that by the time the student finds a job, they will likely need to update any such skills they have. What matters is building up the mental muscle for how to approach such tasks and use the current tools for solving them.

From the learning perspective, automating the work meant for building up your mental muscles makes as much sense as having someone to go to the gym instead of you. Technically, some pushups will be performed, but you would miss the point of the exercise by a few light years. Andrej Karpathy makes a similar analogy with respect to student learning in general: you’re supposed to ‘sweat’, to get a ‘real workout’. What goes for technical content, also goes for learning to think clearly.

One problem for a new student is that it’s harder to notice that your thinking isn’t clear, than the fact that you don’t know some technical concept. For technical courses there are clear curricula and textbooks, and it looks like you just need to go through this much material and then you’re done with this subject. ‘Clear thinking’ doesn’t have such a curriculum. It’s similar to the gym exercises in that you need to keep doing it your whole life, and you can/should always try to get better than your current level. But although there isn’t a curriculum and a clear way to measure progress, this still needs constant work.

Another problem is the avalanche of daily tasks, which is clearly spread across the continuum of “things I know to be meaningless bureaucracy” to “things that I know I should learn”. For a beginner, many things may be hard to recognize as being in the second category. Say, if you identify your career goal as “machine learning engineering” – what should you do about that introductory philosophy of science course? Well, that course might just be the difference between an engineer with and without a graduate degree.

Example: my university has recently made me take a teaching certification course. As someone with a lot of teaching experience, I was annoyed, and highly tempted to delegate my final essay to ChatGPT. But the point of this assignment was not to add to the world supply of teaching philosophy essays (already approaching infinity). It was to make me think through what I am trying to do for my own students and why. For my personal career, it probably would have been better to spend those 30 minutes on another research paper. From my students’ perspective – it is better that I gave that time to them.

If you are a new student facing courses that don’t immediately make sense – maybe they really don’t. But do consider the description of your future degree that convinced you to apply in the first place. Chances are, your university has a team of professionals who thought through your coursework load, and the interplay between those courses. Yes, this is all rough, and things are moving fast, and people sometimes talk past each other, and there will be imperfections and repetitions here and there – but do you generally trust that the designers of your study program know what they’re doing, and it corresponds to the degree description that attracted you? If so, perhaps some of those seemingly useless courses do actually contribute to a graduate version of you. And if you wish your program had more focus on something else – then when you think about how to achieve that, do consider what in your curriculum you are willing to sacrifice, and what set of thinking (not just practical) skills this will leave you with.

The contribution of your courses to the post-graduation version of you does not always literally correspond to course titles. Say, a course titled “research methods” should give you a solid set of evidence assessing skills, useful for almost any kind of job, and not just for professional researchers. Most professors also discuss the course goals in the introductory lectures, and actually mean what they say. They might actually appreciate clarification questions about this: you’d be telling them that you are seriously thinking about the place of their course in your education.

With all this in mind, let us consider the possible AI writing assistance workflows for graduate students.

Approaches to AI-assisted research writing

I talked to dozens of people doing research in machine learning, in particular NLP, asking them about their own writing practices, and I also try to observe how they talk publicly about their work. In my sample of researchers in this area, there are roughly three kinds of approaches to AI-assisted writing.

“Writing = nuisance”. This approach tends to go with mostly experimental work. Writing is considered as the final step in the lifecycle of the project, which is done after all the experiments are done, and it is secondary to the experimental work. The people adopting this approach are more likely to say that they are ok with producing entire texts with ChatGPT, as long as they check that it’s correct. Another line of argumentation for this approach centers on possible difficulties with text input, e.g. for the people who have physical issues that make keyboard work difficult (note the conceptual difference between assisted input and assisted writing, which is what this post focuses on).

“Writing = thinking”. Writing is the tool with which thinking is done. I will illustrate the approach by the following story about Feynman (a famed Nobel laureate in theoretical physics):

Richard Feynman once had a visitor in his office, a historian who wanted to interview him. When he spotted Feynman’s notebooks, he said how delighted he was to see such “wonderful records of Feynman’s thinking.”
“No, no!” Feynman protested. “They aren’t a record of my thinking process. They are my thinking process. I actually did the work on the paper.”
“Well,” the historian said, “the work was done in your head, but the record of it is still here.”
“No, it’s not a record, not really. It’s working. You have to work on paper, and this is the paper.” (Sönke Ahrens, ‘How to Take Smart Notes’)

For myself, I can attest that I have never been able to first do all the experiments, and then just write them up. I have an idea, experiments are done, then I start writing – and this forces me to rethink the idea, the reasons why I thought it would even work, and what my results actually say. The moment that my motivation, my results, my idea lands on the overleaf page – they magically stop being as watertight as I thought they were. This usually forces some new experiments, some rewriting, and so on. The earlier I start writing, the better the project will be.

By the way, a key aspect of peer review process is exactly this – only it is the other people finding the holes in your reasoning, which you failed to find yourself, because you didn’t spend enough time writing and reading what you wrote. It can be really painful to receive such feedback, but it is fair. This is also why it is so helpful to ask colleagues to read your draft, once you have exhausted your own understanding of where the holes might be. They could see some holes that you can’t see yourself, and you could fix them before getting external reviews.

Learning to clearly say what you mean is the mental ‘gym’ of graduate schoool. Note that the idea that you first generate the text, and then just “check” it, assumes that you already have the ability to notice what is wrong. That ability is exactly the muscle that gets exercised by writing and rewriting and rewriting. When you do it yourself, you work in short stretches (sentence, even a phrase), and you gradually build up the ability to see and fix different types of issues. When you get a page of AI-generated text, the effort of reading and noticing/fixing everything there scales up exactly like the effort of debugging one function vs a few hundred lines of code - exponentially harder and less enjoyable. And that’s before we get into the publication ethics, and discussion of who is actually the author of that synthetic text, and how much your own thinking got influenced.

“Rewriting = thinking”. I have so far met two people in this camp. My interpretation is that they have such a severe case of writer block that they prefer to start from a draft of even utter nonsense, or a graveyard of random text snippets cut from other papers, than from a blank page. Perhaps a special case of that is just for inspiring belief in your own writing ability:

People are of course all different and may have different reasons to prefer different workflows. Again, I personally have never been able to do the writing as some kind of auxiliary step for doing the experimental work, but I allow for the possibility that some researchers have such clear thinking that everything is perfectly conceptualized by the time they start writing.

What I absolutely don’t believe is that most new students can skip the “writing = thinking” phase. And, for myself, I don’t think this phase will ever end. Perhaps the dangers of stopping the ‘mental gym’ training are even higher for a tenured professor: we are subject to a higher recognition of our past ideas, which may encourage a kind of intellectual fossilization.

But what about non-native speakers?

I am myself a non-native speaker of English, and I do appreciate how difficult it is to climb that hill. However, ChatGPT can only help you write in English, and that is far from your only problem: you will need to pass oral exams and interviews, present your work at conference, generally talk to your colleagues and peers (at this point, I would actually struggle to talk about research not in English). There’s no shortcut here. If the job you’re aspiring to involves graduate-level intellectual exchanges in international teams or events – you need to put in work to speak professional English fluently, engagingly and convincingly. If somehow you fake it and get a job – it will hardly help you to learn the new organization, new tasks, and make a good impression if you’re also struggling to somehow repay your language learning debt.

Writing is arguably the least stressful way to practice producing largely the same words that you need to be able to produce fluently in these other professional situations. If you feel strongly that English is such an impediment that you’d rather write in your own language and translate – you need to come up with a plan B on how to catch up with the language, asap. Which is non-trivial: applied linguistics is its own field for a reason, and just practicing in a language app is unlikely to help you enough.

That being said, both native and non-native speakers would of course be wise to use tools that offer grammar and stylistic checks, such as Grammarly. These tools have been powered by language models for a long time, and nobody is objecting to them. Such tools can now also help to spot repetitions and extraneous words. But edits like shortening and simplifying are also precious mental gym exercises. If you can’t explain something simply and clearly, it means that you don’t understand it well enough. And shortening is truly a key skill to develop: you only have X pages, and you have to pick what to say so as to preempt the possible criticisms from the reviewer. The same skill will serve you well later, e.g. when you try to pitch your ideas to your boss.

Also, keep in mind that “polishing” your draft with AI assistants by no means guarantees to actually improve the text. Your own original voice, even with slight linguistic imperfections, is actually your asset.

Tips for using your advisor’s time well

  • Your advisor is not the person who will teach you to write (though they will give you some tips and feedback). Most of the work of teaching you to write will be done by yourself, in the process of reworking your own drafts. It will be painful and slow, but it will make you better – just like push-ups.
  • Finding a fellow student to critique each other’s drafts with is also a very good idea. If they didn’t get something, the reviewer probably won’t either.
  • Rubber duck debugging works here too: explaining your motivation/reasoning out loud, even to a rubber duck, may help to notice problems.
  • When you read research papers, try to notice which ones seem clear and well-written to you, and what makes them so. Keep a collection of good ideas for visualization, presentation etc.
  • Do not bring your advisor something you just wrote: even if you are a beginner, if you just read it again, you will probably see some problems that you can fix yourself. When you bring your advisor a draft with such problems – it’s like hammering in nails with a microscope. Use their time for the problems that you can’t yet see/fix yourself. Then overall more problems will get fixed, and the project will be more likely to withstand other people’s scrutiny.
  • Always plan your work so that you have a draft at least a day before you send it, so that you can sleep on it, and fix at least the obvious problems first.
  • Focus on the clarity and text structure first, the minor spelling/grammar checks are less important.
  • Leave comments to mark the parts of text which aren’t ready to be read yet, so that they don’t waste time.
  • If some parts of the text aren’t written yet, put in a sentence saying what will be the main point of that section.
  • I ask my students to formulate clearly and early in the project (a) what the problem is, (b) why it is important, (c) what has been done about it (roughly, not full literature review), (d) what is the new thing they propose to do, and how. This is the skeleton for the introduction section, and this should be updated as their understanding of these points evolves.
]]>Anna RogersAI ‘News’ Content Farms Are Easy to Make and Hard to Detect2024-08-12T18:00:00-04:002024-08-12T18:00:00-04:00https://hackingsemantics.xyz/2024/ai-newsHow bad is the synthetic news problem?

Do you feel like most sources of information that you used to rely on, have gotten worse? You’re right: Internet is already flooded with AI-generated content. I’m a NLP researcher, so I’m mostly monitoring the cases involving text, but here are just a few examples within that domain:

To be clear, the above problems are not new: review farms, SEO ‘content’ etc have existed before LLMs went mainstream. E.g. here’s a 2017 discussion of the problem with fake reviews on customer review websites like Yelp. But the scale of the problem has changed, as generating such ‘content’ today now does not even require technical knowledge.

This year I contributed to a study published at ACL 2024, led by Giovanni Puccetti from CNR Pisa. This study focused on the specific problem of synthetic ‘news’: websites generated to resemble legitimate news outlets, but entirely filled with synthetic text without any clear traces of human editors involved. Here is a screenshot of such a website:

Example of a website filled with synthetic text, but formatted and presented as a legitimate news website

This particular example appears to be aimed simply at serving ads, to people who visit because they think it’s a legitimate information source. Here the model that was used to generate the content is clearly bad enough that the synthetic nature of the text is obvious - and still this site must have been making its owners enough money, since it’s been up for at least a year. The main harms are waste of user time and resources, and potentially also misinformation (if the reader does not realize that this source is scammy, and comes away misinformed). There are also websites that paraphrase & republish content of real news sites. The main victim in this case is to the original outlet, from which the content is stolen, and the reader may also come away with mis/disinformation. Finally, synthetic ‘news’ may disseminate propaganda narratives, potentially harming not just the individual reader, but the society overall. The disinformation campaigns are trying to mix such narratives with real stories, so that they are perceived as more credible.

At present, the only organization I’m aware of that tries to estimate the prevalence of such ‘news’ is NewsGuard, a for-profit organization that also provides ‘reputation management’ services. Newsguard provides as a service their ratings of various news outlets, including their list of unreliable websites manually identified by their team. In April 2023 they reported 49 such websites. In August 2024, the latest count is 1,021. At present this is the only tracker I’m aware of, and Newsguard list is not public (it is a for-profit company that sells access to that list through a browser extension). So it is hard to tell how accurate this is as an estimate of the size of the problem, but the examples I could find check out. Their criteria for identifying such sites are quite narrow (e.g. AI-generated website with minimal human editing would not count), and so there probably are a lot more spammy sites that are just masking it better.

Importantly, the issue already went far beyond English. At present, Newsguard reports that they identified such websites in 16 languages: Arabic, Chinese, Czech, Dutch, English, French, German, Indonesian, Italian, Korean, Portuguese, Russian, Spanish, Tagalog, Thai, Turkish. A separate, though related, issue is the use of audio/image/video deep fakes to provide seeming legitimacy to fake stories: e.g. in a recent Slovakia elections fake audio recordings went viral two days before elections, damaging the progressive candidate who then lost to a populist with a pro-Russian policy on Ukraine (Meaker, n.d.)).

At present, there is little recourse. Even when spammy sites are identified and called out as such, this does not automatically damage their standing in search engine results, which is what they’re after. Voice of America attempted to reach out to owners of several such websites, but received either no or uninformative responses (Guess, 2024). They conclude that ‘part of the problem in accountability is the difficulty in tracing the owners or producers of the sites’.

According to Newsguard, 90% of ads supporting these websites are through the Google ad services. In response to request from Voice of America (Guess, 2024), Google said that they could not verify that since Newsguard does not share their list of sites (which of course it would not share, since it is their main business asset). But if their list can be considered as an independent test for whatever strategies Google itself is employing to detect spammy websites - these strategies do not appear to work well at present.

Plausible synthetic ‘news’ is (too) easy to generate, even beyond English

For our study, we used a relatively old LLM - the first generation Llama (7B and 65B parameters) (Touvron et al., 2023). It is a ‘mostly-English’ model: we know it was not trained to be multilingual, but its training data included Italian Wikipedia (about 500K articles). It could have also seen some other Italian web texts, but there were deliberate efforts to exclude non-English sources. We further used 40K Italian news articles from an open dataset CHANGE-it (Mattei, Cafagna, Dell’Orletta, Nissim, & Gatt, 2020). We fed the model the first thirty tokens of real news articles as a prompt, and generated its continuation. Then we recruited 93 native speakers of Italian to determine whether the text is synthetic or not. They were shown either the original news articles, or their versions with synthetic endings. For the best model (fine-tuned 65B Llama), the accuracy of our native speakers was only 64% (vs 50% random chance).

This result is in line with the experiments OpenAI did to show that already GPT-3 could generate news articles in English that could not be easily detected by humans (Brown et al., 2020). The new aspect of our work is showing that the problem goes beyond English. We are not claiming that any language + model combination would be equally successful, but the later models (especially multilingual ones) could be expected to do even better than the old Llama we used.

We note that creating a model that could power a scammy news-like website now comes with very few technical or financial difficulties, which could be expected increase the number of bad actors – and would be in line with the proliferation of such websites discussed above. Renting GPUs on cloud providers such as Amazon is now relatively cheap, and it would require as little as 100$ to replicate one of our LLM training sessions and data generation. The technical barrier to fine-tuning LLMs is also low now, thanks to tools like Huggingface’s Autotrain. We do not criticize open-sourcing such tools, but we hope that our results would highlight the problem with detecting synthetic text, and necessity of more research on synthetic text detection.

Synthetic ‘news’ is currently nearly-impossible to detect ‘in the wild’

Supervised detection.

Methodology. The core idea behind this approach is to train a binary classifier on a collection of human-written and synthetic texts. The core problem is that one needs to collect such a dataset, ideally balanced, and that this dataset should represent the distribution of all possible human- and machine-written texts. This seems conceptually non-feasible, even within a given domain such as news. We experimented with training datasets of various sizes (2K, 4K and 8K) and with various distributions of human-written texts:

  • in-domain: all human samples come from one dataset (CHANGE-it (Mattei, Cafagna, Dell’Orletta, Nissim, & Gatt, 2020))
  • mixed: 50% human samples come from CHANGE-it, and 50% from DICE, a different Italian news dataset (Bonisoli, di Buono, Po, & Rollo, 2023) This allows us to see whether the task becomes more difficult for classifier when it has to represent a more complex distribution of human-authored texts. We also experimented with 3 models used for the machine-generated text: llama2-7b, llama2-13b, and mistral-7b. The classifier was based on xlm-roberta-large.

What we found: for all generator models, the classifier trained on mixed distribution gets consistently worse performance than in-domain (and our mix of just two datasets is a lot simpler than the real distribution of human-authored news). Also, at least 4K samples are needed to get accuracy in the range of 80%.

Accuracy of a classifier based on xlm-RoBERTa-large for human/synthetic text classification task, for synthetic texts generated by three LLMs that were fine-tuned on CHANGE-it.

Verdict: not a practical solution. In reality, the distribution of all possible human-authored texts (even just for Italian news) is much more complex than our settings with 1-2 datasets for Italian. Perhaps this could be the core reason for why OpenAI pulled its own classifier ‘due to low accuracy’. And also we need to collect at least 4K samples of texts. That takes time and effort, and ideally we would like to take action on the scammers before they publish that much.

Approaches based on token likelihoods

Methodology. The methods based on token likelihoods assume that we have access to the token likelihood information from the model. We experimented with log-likelihood and DetectGPT (Mitchell, Lee, Khazatsky, Manning, & Finn, 2023) approaches (DetectGPT normalizes the log-likelihood score based on sentence modifications generated by another model, IT5-large (Sarti & Nissim, 2022) in our case). The task is to determine whether a given text came from our llama1, either original version or after fine-tuning on CHANGE-IT news dataset.

What we found. The simple log-likelihood does not seem to react to fine-tuning at all: the ROC curve is about the same for pre-trained and fine-tuned models. Detect-GPT has a consistently higher ROC curve than log-likelihood, and does rise 5-6 points for fine-tuning. This is counter-intuitive, because fine-tuning should have made the task more rather than less difficult: by qualitative analysis we found that without Italian fine-tuning the original llama tends to switch to English in the middle of the text, which is highly unnatural and was noticed by our human raters.

ROC curve for DetectGPT and log-likelihood. In (a) for Llama 65B measured over 100 sentences from the CHANGE-it data-set (Italian), in (b) the same measure for Llama 65B model after 20,000 fine tuning steps on CHANGE-it training set and in (c) after 60,000 fine-tuning steps.

We also experimented with using token likelihood information from other models to detect an unknown model. The good news is that even a little fine-tuning data, similar to data that the scammers could have used, is enough. The bad news is that this only seems to work well when the models used for detection and generation are the same. See the paper for details.

Verdict: not a practical solution. If we are trying to detect whether a given suspicious text was written by a model, we do not have access to the token likelihoods in the first place. And there are already too many ‘open’ models to check them all – not to mention the commercial models provided via API without access to token likelihodd information, or custom non-public models.

What about watermarking?

Watermarking assumes that the scammers used a model that comes with a watermark. If the watermark is applied at generation time to an ‘open’ model, such as red-green watermarking (Kirchenbauer et al., 2023), it is safe to assume that they would try to remove it. If it is embedded in model weights, it is also relatively simple to remove it by fine-tuning (which they would likely do anyway). If the scammers used a commercially available model, tampering at the source could be ruled out, but the many current commercially available models do not appear to be watermarked (Gloaguen, Jovanović, Staab, & Vechev, 2024). And finally, even if the text was originally watermarked, the watermark could be weakened and even removed by further manipulating the text by a non-watermarked model, e.g. paraphrasing or human editing (Kirchenbauer et al., 2023).

Moreover, even if we assume that the watermark is intact (i.e. if the scammers were too lazy to remove it), we have the same problem as with approaches based on token likelihoods: simply too many models to test. For watermarking solutions to be practical, there would have to be a centralized repository of all known watermarks, where one could submit a text sample to check against them all in one step. At present, there do not seem to be any efforts in that direction.

Conclusion

Where do we go from here? The methods relying on token probability distributions currently seem to be the best bet for synthetic text detection methods - but even among the open source models there are already too many options to check, given a suspicious text sample.

This does not entail that open source models should be banned, but we do need to consider ways forward that encourage their development that is responsible by design. Ideally watermarking would be built into model weights, so that it would be at least difficult to remove, and there would ideally be a centralized service enabling anybody to check whether a given text contains any of the known watermarks. For ‘closed’ models watermarking should be even easier, since they control the generation process and ensure that the watermark is applied.

Should we bother at all though, if none of the strategies is likely to be fully successful? Any watermark can probably be removed, if you try hard enough, and the most villainous agents (e.g. content farms acting on behalf of hostile governments) are likely to be sufficiently well-funded to develop their own models if they need to. That is all true, but I will borrow an analogy from Prof. Hany Farid (UCB): we do lock our doors, although we know that there are people who can unlock most common household locks. But since we know that it is a relatively rare skill, it is still rational to use household locks as defense. Similarly, measures that significantly raise the barrier for scammers should at least lower the volume of scammy websites.

Assuming that the research community does mananage to develop better measures for identifying spammy websites - there should be a clear, transparent, and public-accessible way to trigger action on scammy websites. Their owners are now hard to track, and the sites are hard to take down, and it is hard to change any of that without changing Internet in a way that would also endanger people who need the protection of anonymity (e.g. activists in authoritarian regime countries). But at least for the scammy websites that only try to serve ads by misrepresenting their content as human-authored - perhaps there should be a process to at least report them and cut off their monetization. This is their raison d’être.

]]>Anna RogersA Sanity Check on ‘Emergent Properties’ in Large Language Models2024-07-15T13:00:00-04:002024-07-15T13:00:00-04:00https://hackingsemantics.xyz/2024/emergenceOne of the often-repeated claims about Large Language Models (LLMs), discussed in our ICML’24 position paper, is that they have ‘emergent properties’. Unfortunately, in most cases the speaker/writer does not clarify what they mean by ‘emergence’. But misunderstandings on this issue can have big implications for the research agenda, as well as public policy.

From what I’ve seen in academic papers, there are at least 4 senses in which NLP researchers use this term:

  1. A property that a model exhibits despite not being explicitly trained for it. E.g. Bommasani et al. (2021, p. 5) refers to few-shot performance of GPT-3 (Brown et al., 2020) as “an emergent property that was neither specifically trained for nor anticipated to arise’”.
  2. (Opposite to def. 1): a property that the model learned from the training data. E.g. Deshpande et al. (2023, p. 8) discuss emergence as evidence of “the advantages of pre-training’’.
  3. A property “is emergent if it is not present in smaller models but is present in larger models.’’ (Wei et al., 2022, p. 2).
  4. A version of def. 3, where what makes emergent properties “intriguing’’ is “their sharpness, transitioning seemingly instantaneously from not present to present, and their unpredictability, appearing at seemingly unforeseeable model scales” (Schaeffer, Miranda, & Koyejo, 2023, p. 1)

For a technical term, this kind of fuzziness is unfortunate. If many people repeat the claim “LLLs have emergent properties” without clarifying what they mean, a reader could infer that there is a broad scientific consensus that this statement is true, according to the reader’s own definition.

I am writing this post after giving many talks about this in NLP research groups all over the world - Amherst and Georgetown (USA), Cambridge, Cardiff and London (UK), Copenhagen (Denmark), Gothenburg (Sweden), Milan (Italy), Genbench workshop (EMNLP’23 @ Singapore) (thanks to everybody in the audience!). This gave me a chance to poll a lot of NLP researchers about what they thought of emergence. Based on the responses from 220 NLP researchers and PhD students, by far the most popular definition is (1), with (4) being the second most popular.

The idea expressed in definition (1) also often gets invoked in public discourse. For example, you can see it in the claim that Google’s PaLM model ‘knew’ a language it wasn’t trained on (which is almost certainly false). The same idea also provoked the following public exchange between a US senator and Melanie Mitchell (a prominent AI researcher, professor at Santa Fe Institute):

What this exchange shows is the idea of LLM ‘emergent properties’ per definition (1) has implications outside the research world. It contributes to the anxiety about the imminent takeover by super-AGI, to calls for pausing research. It could push the policy-makers in the wrong directions, such as banning open-source research – which would further consolidate resources in the hands of a few big tech labs, and ensure they won’t have much competition. It also creates the impression of LLMs as entities independent on the choices of their developers and deployers – which has huge implications for who is accountable for any harms coming from these models. With such high stakes for the research community and society, shouldn’t we at least make sure that the science is sound?

How much do these notions of ‘emergence’ contribute to the scientific understanding of LLMs?

Much in the above versions of ‘emergence’ in LLMs is still debatable: how much do they actually advance the scientific discussion, with respect to other terms and known principles that are already in use? I would like to stress that this discussion is completely orthogonal to the question of whether LLMs are useful or valuable. Countless models have been and will be practically useful without claims of emergence.

Let us start with definition 2: something that a model learned from the training data. Since this is exactly what a machine learning model is supposed to do, does this version of ‘emergence’ add much to ‘learning’?

For the definition (3) (something that only large models do), the better performance of larger models is to be expected, given basic machine learning principles: the larger model simply has more capacity to learn the patterns in its training data. Hence, this version of ‘emergence’ also does not add much. Unless we expect that the larger models, but not the small ones, do something they weren’t trained for – but then this definition depends on definition (1).

For the definition (4), the phenomenon of sharp change in performance turned out to be attributable to non-continuous evaluation metrics (e.g. for classification tasks like multi-choice question answering), rather than LLMs themselves (Schaeffer, Miranda, & Koyejo, 2023). Furthermore, J. Wei himself acknowledges that the current claims of sharp changes are based on results from models that are only available in relatively few sizes (1B, 7B, 13B, 70B, 150B…), and if we had more results for intermediate model sizes, the increase in performance would likely turn out to be smooth (Wei, 2023).

The unpredictability part of definition (4) was reiterated by J. Wei (Wei, 2023) as follows:

the “emergence” phenomenon is still interesting if there are large differences in predictability: for some problems, performance of large models can easily be extrapolated from performance of models 1000x less in size, whereas for others, even it cannot be extrapolated even from 2x less size.
However, the cited predictability at 1,000x less compute refers to the GPT-4 report (OpenAI, 2023), where the developers knew the target evaluation in advance, and specifically optimized for it. Given that, predictable scaling is hardly surprising theoretically (though still impressive from the engineering point of view). This is in contrast with the unpredictability at 2x less compute for unplanned BIG-Bench evaluation in (Wei et al., 2022). This unpredictability is expected, simply due to the unknown interaction between (a) the presence of training data that is similar to test data, and (b) sufficient model capacity to learn some specific patterns.

Hence, we are left with the definition (1): emergent properties are properties that the model was not explicitly trained for. This can be interpreted in two ways:

  1. A property is emergent if the model was not exposed to training data for that property.
  2. A property is emergent even if the model was exposed to the relevant training data -- as long as the model developers were unaware of it.

Per def. 6, it would appear that the research question is actually ‘what data exists on the Web?’ (or in proprietary training datasets of generative AI companies), and we are training LLMs as a very expensive method to answer that question. For example, ChatGPT can generate chess moves that are plausible-looking (but often illegal). This is surprising if we think of ChatGPT as a language model, but not if we know that it is a model trained on a web corpus, because such a corpus would likely include not only texts in a natural language, but also materials like as chess transcripts, ascii art, midi music, programming code etc. The term ‘language model’ is actually a misnomer - they are rather corpus models (Veres, 2022).

Per def. 5, we can prove that some property is emergent only by showing that the model was not exposed to evidence that could have been the basis for the model outputs in the training data. And it cannot be due to lucky sampling in the latent space of the continuous representations. If we are allowed to generate as many samples as we want and cherry-pick, we are eventually going to get some fluent text even from a randomly initialized model – but this should arguably not count as an ‘emergent property’ on definition (5).

For commercial models with undisclosed training data such as ChatGPT, such a proof is out of the question. But even for the “open” LLMs this is only a hypothesis (if not wishful thinking), because so far we are lacking detailed studies (or even a methodology) to consider the exact relation between the amount and kinds of evidence in the training text data for a particular model output. On definition 5, emergent properties are a machine learning equivalent of alchemy – and the bar for postulating that should be quite high.

Especially in the face of evidence to the contrary.

Counter-evidence to ‘emergent properties’ in LLMs

Here are some of the empirical results that make it dubious that LLMs have ‘emergent properties’ by definition (5) (the model was not exposed to training data for that property):

  • Phenomenon of prompt sensitivity (Lu, Bartolo, Moore, Riedel, & Stenetorp, 2022; Zhao, Wallace, Feng, Klein, & Singh, 2021): LLMs responding differently to prompts that should be semantically equivalent. If we say that models have an emergent property of answering questions, slightly different ways of posing these questions, and especially different order of few-shot examples, should not matter. The most likely explanation for the prompt sensitivity is that the model responds better to prompts that are more similar to its training data in some way that helps the model.
  • Liang et. al evaluate 30 LLMs and conclude that “regurgitation (of copyrighted materials) risk clearly correlates with model accuracy’’ (2022, p. 12). This suggests that models which ‘remember’ more of training data perform better.
  • McCoy et al. (McCoy, Yao, Friedman, Hardy, & Griffiths, 2023) show that LLM performance depends on probabilities of output word sequences in web texts.
  • Lu et al. (Lu, Bigoulaeva, Sachdeva, Madabushi, & Gurevych, 2023) show that emergent abilities of 18 LLMs can be ascribed mostly to in-context learning. Instruction tuning facilitates in-context learning, but does not seem to have an independent effect.
  • For in-context learning itself (first shown in GPT-3 (Brown et al., 2020), and used as the example of ‘emergence’ by Bommasani et al. (2021, pp. 5 % }) , the results of {% cite ChanSantoroEtAl_2022_Data_Distributional_Properties_Drive_Emergent_In-Context_Learning_in_Transformers) suggest that it happens only in Transformers trained on sequences, structurally similar to the sequences in which in-context learning would be tested.
  • Liu et al. (Liu et al., 2023) report that ChatGPT and GPT-4 perform better on older compared to newly released benchmarks, suggesting that many evaluation results may be inflated due to data contamination. OpenAI itself went to great lengths in the GPT-3 paper (Brown et al., 2020) showing how difficult it is to mitigate this problem. Since we know nothing about the training data of the latest models, external evaluation results may not be meaningful, and internal reports by companies that sell their models as a commercial service have a clear conflict of interest.

A well-known effort to propose a methodology that would avoid at least the data contamination problem is the ‘sparks of AGI’ study (Bubeck et al., 2023). Using the methodology of newly constructed test cases, checked against public web data, and their perturbations, the authors notably concluded that GPT-4 possesses “a very advanced theory of mind’’. At least two studies have come to the opposite conclusion (Sap, Le Bras, Fried, & Choi, 2022; Shapira et al., 2023). The most likely reason for the failure of this methodology is that while we can check for direct matches on the web, we could still miss some highly similar cases (e.g. the well-known example of unicorn drawn in tikz from that paper could be based on the stackoverflow community drawing other animals in tikz). Furthermore, the commercial LLMs such as GPT-4 could also be trained on data that is not publicly available. In case of OpenAI, hundreds of researchers and other users of GPT-3 have submitted a lot of data though the API, before OpenAI changed their terms of service to not use such data for training by default.

This is not to say that it is absolutely impossible that LLMs could work well out of their training distribution. Some degree of generalization is happening, and the best-case scenario is that it is due to interpolation of patterns that were observed in training data individually, but not together. But at what point we would say that the result is something qualitatively new, what kind of similarity to training data matters, and how we could identify it - these are all still-unresolved research questions.

NLP researchers are actually NOT convinced about LLM emergent properties

As I mentioned, I had a chance to give a talk about this in several NLP research groups. In the very beginning of these talks, before I presented the above discussion, I asked the audience a few questions, including whether they personally believed that LLMs had emergent properties (according to their preferred definition, which, as shown above, was predominantly (1)). I also asked them about their perception of the consensus in the field - what did they think that most other NLP researchers thought about this? For the first question I have answers from 259 researchers and PhD students, and for the second - from 360 (note to self: give people more time to connect to the poll).

The results were striking: while most respondents were sceptical or unsure about LLM emergent properties themselves (only 39% agreed with that statement), 70% thought that most other researchers did believe this.

This is in line with several other false sociological beliefs: e.g. most NLP researchers don’t think that NLP leaderboards are particularly meaningful, or that scaling will solve everything, but they do think that other NLP researchers believe that (Michael et al., 2023). In my sample, the idea that LLM have emergent properties is similarly held by a minority of researchers, but it is misperceived to be the majority. And even for that minority the conviction is not very firm. In four of my talks, after presenting the above discussion, I also asked the audience what they thought now. In this sample of 70 responses, 83% of those who originally agreed with the statement “LLMs have emergent properties”, changed their belief to either disagreeing (13.9%) or being unsure (69.4%).

In retrospect, “agree/disagree/unsure” is not the best choice of options for this poll. As scientists, we can hardly ever be 100% sure: as Yann LeCun put it in the Munk debate, we cannot even prove that there is no teapot orbiting Jupiter right now. Our job is not to fall into such distracting rabbit holes, but to formulate and test hypotheses that would advance our understanding of the phenomenon we are studying. For ‘emergence’ in LLMs, I think we are still at the ‘formulation’ stage – since even after all the above work with clarifying ‘emergence’ we still don’t have a research question, for which it is clear how to obtain empirical evidence.

The key unresolved question is what kind of interpolation of existing patterns would even count as something new enough to qualify as an ‘emergent phenomenon’ in the domain of natural language data. This domain is particularly hard, because it mixes different kinds of information (linguistic, social, factual, commonsense), and that information may be present differently (explicit in context, implicit, or requiring reasoning over long contexts). See (Rogers, Gardner, & Augenstein, 2023, pp. sec. 8.2) for a discussion of different skills involved in just the question answering task.

:wave: If you will attend ICML or ACL’24, and would like to chat about this, let me know! Also, I am recruiting (PhD and postdoc level).

This post is based on a part of the ICML 2024 position paper Key Claims in LLM Research Have a Long Tail of Footnotes, by Anna Rogers and Sasha Luccioni. The poll results are not there, but most of the other points can be cited as follows:

@inproceedings{
rogers2024position,
title={Position: Key Claims in {LLM} Research Have a Long Tail of Footnotes},
author={Anna Rogers and Sasha Luccioni},
booktitle={Forty-first International Conference on Machine Learning},
year={2024},
url={https://openreview.net/forum?id=M2cwkGleRL}
}

The paper also discusses what we even mean by ‘large language model’ (as opposed to ‘foundation’ and ‘frontier’ models), and several other often-repeated claims that come with a lot of footnotes: LLMs are robust, LLMs are state-of-the-art, (LLM) scale is all you need, LLMs are general-purpose-technologies.

:pray: Acknowledgements:

References

  1. Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., … Liang, P. (2021). On the Opportunities and Risks of Foundation Models. ArXiv:2108.07258 [Cs]. 
    @article{bommasani2021opportunities,
  title = {On the {{Opportunities}} and {{Risks}} of {{Foundation Models}}},
  author = {Bommasani, Rishi and Hudson, Drew A. and Adeli, Ehsan and Altman, Russ and Arora, Simran and {von Arx}, Sydney and Bernstein, Michael S. and Bohg, Jeannette and Bosselut, Antoine and Brunskill, Emma and Brynjolfsson, Erik and Buch, Shyamal and Card, Dallas and Castellon, Rodrigo and Chatterji, Niladri and Chen, Annie and Creel, Kathleen and Davis, Jared Quincy and Demszky, Dora and Donahue, Chris and Doumbouya, Moussa and Durmus, Esin and Ermon, Stefano and Etchemendy, John and Ethayarajh, Kawin and {Fei-Fei}, Li and Finn, Chelsea and Gale, Trevor and Gillespie, Lauren and Goel, Karan and Goodman, Noah and Grossman, Shelby and Guha, Neel and Hashimoto, Tatsunori and Henderson, Peter and Hewitt, John and Ho, Daniel E. and Hong, Jenny and Hsu, Kyle and Huang, Jing and Icard, Thomas and Jain, Saahil and Jurafsky, Dan and Kalluri, Pratyusha and Karamcheti, Siddharth and Keeling, Geoff and Khani, Fereshte and Khattab, Omar and Koh, Pang Wei and Krass, Mark and Krishna, Ranjay and Kuditipudi, Rohith and Kumar, Ananya and Ladhak, Faisal and Lee, Mina and Lee, Tony and Leskovec, Jure and Levent, Isabelle and Li, Xiang Lisa and Li, Xuechen and Ma, Tengyu and Malik, Ali and Manning, Christopher D. and Mirchandani, Suvir and Mitchell, Eric and Munyikwa, Zanele and Nair, Suraj and Narayan, Avanika and Narayanan, Deepak and Newman, Ben and Nie, Allen and Niebles, Juan Carlos and Nilforoshan, Hamed and Nyarko, Julian and Ogut, Giray and Orr, Laurel and Papadimitriou, Isabel and Park, Joon Sung and Piech, Chris and Portelance, Eva and Potts, Christopher and Raghunathan, Aditi and Reich, Rob and Ren, Hongyu and Rong, Frieda and Roohani, Yusuf and Ruiz, Camilo and Ryan, Jack and R{\'e}, Christopher and Sadigh, Dorsa and Sagawa, Shiori and Santhanam, Keshav and Shih, Andy and Srinivasan, Krishnan and Tamkin, Alex and Taori, Rohan and Thomas, Armin W. and Tram{\`e}r, Florian and Wang, Rose E. and Wang, William and Wu, Bohan and Wu, Jiajun and Wu, Yuhuai and Xie, Sang Michael and Yasunaga, Michihiro and You, Jiaxuan and Zaharia, Matei and Zhang, Michael and Zhang, Tianyi and Zhang, Xikun and Zhang, Yuhui and Zheng, Lucia and Zhou, Kaitlyn and Liang, Percy},
  year = {2021},
  month = aug,
  journal = {arXiv:2108.07258 [cs]},
  eprint = {2108.07258},
  primaryclass = {cs},
  url = {http://arxiv.org/abs/2108.07258},
  urldate = {2021-08-18},
  archiveprefix = {arXiv}
}
  2. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., … Amodei, D. (2020). Language Models Are Few-Shot Learners. Advances in Neural Information Processing Systems 33 (NeurIPS 2020). 
    @inproceedings{BrownMannEtAl_2020_Language_Models_are_Few-Shot_Learners,
  ids = {BrownMannEtAl_2020_Language_Models_are_Few-Shot_Learnersa},
  title = {Language {{Models}} Are {{Few-Shot Learners}}},
  booktitle = {Advances in {{Neural Information Processing Systems}} 33 ({{NeurIPS}} 2020)},
  author = {Brown, Tom B. and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and Agarwal, Sandhini and {Herbert-Voss}, Ariel and Krueger, Gretchen and Henighan, Tom and Child, Rewon and Ramesh, Aditya and Ziegler, Daniel M. and Wu, Jeffrey and Winter, Clemens and Hesse, Christopher and Chen, Mark and Sigler, Eric and Litwin, Mateusz and Gray, Scott and Chess, Benjamin and Clark, Jack and Berner, Christopher and McCandlish, Sam and Radford, Alec and Sutskever, Ilya and Amodei, Dario},
  year = {2020},
  month = jun,
  eprint = {2005.14165},
  url = {https://papers.nips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html},
  urldate = {2020-06-04},
  archiveprefix = {arXiv}
}
  3. Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., … Zhang, Y. (2023). Sparks of Artificial General Intelligence: Early Experiments with GPT-4. https://doi.org/10.48550/arXiv.2303.12712 
    @misc{BubeckChandrasekaranEtAl_2023_Sparks_of_Artificial_General_Intelligence_Early_experiments_with_GPT-4,
  title = {Sparks of {{Artificial General Intelligence}}: {{Early}} Experiments with {{GPT-4}}},
  shorttitle = {Sparks of {{Artificial General Intelligence}}},
  author = {Bubeck, S{\'e}bastien and Chandrasekaran, Varun and Eldan, Ronen and Gehrke, Johannes and Horvitz, Eric and Kamar, Ece and Lee, Peter and Lee, Yin Tat and Li, Yuanzhi and Lundberg, Scott and Nori, Harsha and Palangi, Hamid and Ribeiro, Marco Tulio and Zhang, Yi},
  year = {2023},
  month = apr,
  number = {arXiv:2303.12712},
  eprint = {2303.12712},
  primaryclass = {cs},
  publisher = {arXiv},
  doi = {10.48550/arXiv.2303.12712},
  url = {http://arxiv.org/abs/2303.12712},
  urldate = {2023-04-29},
  archiveprefix = {arXiv}
}
  4. Deshpande, V., Pechi, D., Thatte, S., Lialin, V., & Rumshisky, A. (2023). Honey, I Shrunk the Language: Language Model Behavior at Reduced Scale. Findings of the Association for Computational Linguistics: ACL 2023, 5298–5314. Toronto, Canada: Association for Computational Linguistics. 
    @inproceedings{deshpande-etal-2023-honey,
  title = {Honey, {I} Shrunk the Language: Language Model Behavior at Reduced Scale.},
  author = {Deshpande, Vijeta and Pechi, Dan and Thatte, Shree and Lialin, Vladislav and Rumshisky, Anna},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2023},
  month = jul,
  year = {2023},
  address = {Toronto, Canada},
  publisher = {Association for Computational Linguistics},
  pages = {5298--5314},
  url = {https://aclanthology.org/2023.findings-acl.326}
}
  5. Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., … Koreeda, Y. (2022). Holistic Evaluation of Language Models. https://doi.org/10.48550/arXiv.2211.09110 
    @misc{LiangBommasaniEtAl_2022_Holistic_Evaluation_of_Language_Models,
  title = {Holistic {{Evaluation}} of {{Language Models}}},
  author = {Liang, Percy and Bommasani, Rishi and Lee, Tony and Tsipras, Dimitris and Soylu, Dilara and Yasunaga, Michihiro and Zhang, Yian and Narayanan, Deepak and Wu, Yuhuai and Kumar, Ananya and Newman, Benjamin and Yuan, Binhang and Yan, Bobby and Zhang, Ce and Cosgrove, Christian and Manning, Christopher D. and R{\'e}, Christopher and {Acosta-Navas}, Diana and Hudson, Drew A. and Zelikman, Eric and Durmus, Esin and Ladhak, Faisal and Rong, Frieda and Ren, Hongyu and Yao, Huaxiu and Wang, Jue and Santhanam, Keshav and Orr, Laurel and Zheng, Lucia and Yuksekgonul, Mert and Suzgun, Mirac and Kim, Nathan and Guha, Neel and Chatterji, Niladri and Khattab, Omar and Henderson, Peter and Huang, Qian and Chi, Ryan and Xie, Sang Michael and Santurkar, Shibani and Ganguli, Surya and Hashimoto, Tatsunori and Icard, Thomas and Zhang, Tianyi and Chaudhary, Vishrav and Wang, William and Li, Xuechen and Mai, Yifan and Zhang, Yuhui and Koreeda, Yuta},
  year = {2022},
  month = nov,
  number = {arXiv:2211.09110},
  eprint = {2211.09110},
  primaryclass = {cs},
  publisher = {arXiv},
  doi = {10.48550/arXiv.2211.09110},
  urldate = {2023-07-28},
  url = {http://arxiv.org/abs/2211.09110},
  archiveprefix = {arXiv},
  keywords = {!}
}
  6. Liu, H., Ning, R., Teng, Z., Liu, J., Zhou, Q., & Zhang, Y. (2023). Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4. https://doi.org/10.48550/arXiv.2304.03439 
    @misc{LiuNingEtAl_2023_Evaluating_Logical_Reasoning_Ability_of_ChatGPT_and_GPT-4,
  title = {Evaluating the {{Logical Reasoning Ability}} of {{ChatGPT}} and {{GPT-4}}},
  author = {Liu, Hanmeng and Ning, Ruoxi and Teng, Zhiyang and Liu, Jian and Zhou, Qiji and Zhang, Yue},
  year = {2023},
  month = may,
  number = {arXiv:2304.03439},
  eprint = {2304.03439},
  primaryclass = {cs},
  publisher = {arXiv},
  doi = {10.48550/arXiv.2304.03439},
  url = {https://arxiv.org/abs/2304.03439},
  urldate = {2023-06-21},
  archiveprefix = {arXiv}
}
  7. Lu, S., Bigoulaeva, I., Sachdeva, R., Madabushi, H. T., & Gurevych, I. (2023). Are Emergent Abilities in Large Language Models Just In-Context Learning? https://doi.org/10.48550/arXiv.2309.01809 
    @misc{LuBigoulaevaEtAl_2023_Are_Emergent_Abilities_in_Large_Language_Models_just_In-Context_Learning,
  title = {Are {{Emergent Abilities}} in {{Large Language Models}} Just {{In-Context Learning}}?},
  author = {Lu, Sheng and Bigoulaeva, Irina and Sachdeva, Rachneet and Madabushi, Harish Tayyar and Gurevych, Iryna},
  year = {2023},
  month = sep,
  number = {arXiv:2309.01809},
  eprint = {2309.01809},
  primaryclass = {cs},
  publisher = {arXiv},
  doi = {10.48550/arXiv.2309.01809},
  urldate = {2023-11-22},
  url = {https://arxiv.org/abs/2309.01809},
  archiveprefix = {arXiv}
}
  8. Lu, Y., Bartolo, M., Moore, A., Riedel, S., & Stenetorp, P. (2022). Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 8086–8098. https://doi.org/10.18653/v1/2022.acl-long.556 
    @inproceedings{LuBartoloEtAl_2022_Fantastically_Ordered_Prompts_and_Where_to_Find_Them_Overcoming_Few-Shot_Prompt_Order_Sensitivity,
  title = {Fantastically {{Ordered Prompts}} and {{Where}} to {{Find Them}}: {{Overcoming Few-Shot Prompt Order Sensitivity}}},
  shorttitle = {Fantastically {{Ordered Prompts}} and {{Where}} to {{Find Them}}},
  booktitle = {Proceedings of the 60th {{Annual Meeting}} of the {{Association}} for {{Computational Linguistics}} ({{Volume}} 1: {{Long Papers}})},
  author = {Lu, Yao and Bartolo, Max and Moore, Alastair and Riedel, Sebastian and Stenetorp, Pontus},
  year = {2022},
  month = may,
  pages = {8086--8098},
  publisher = {Association for Computational Linguistics},
  address = {Dublin, Ireland},
  doi = {10.18653/v1/2022.acl-long.556},
  url = {https://aclanthology.org/2022.acl-long.556},
  urldate = {2022-06-15}
}
  9. McCoy, R. T., Yao, S., Friedman, D., Hardy, M., & Griffiths, T. L. (2023). Embers of Autoregression: Understanding Large Language Models Through the Problem They Are Trained to Solve. https://doi.org/10.48550/arXiv.2309.13638 
    @misc{McCoyYaoEtAl_2023_Embers_of_Autoregression_Understanding_Large_Language_Models_Through_Problem_They_are_Trained_to_Solve,
  title = {Embers of {{Autoregression}}: {{Understanding Large Language Models Through}} the {{Problem They}} Are {{Trained}} to {{Solve}}},
  shorttitle = {Embers of {{Autoregression}}},
  author = {McCoy, R. Thomas and Yao, Shunyu and Friedman, Dan and Hardy, Matthew and Griffiths, Thomas L.},
  year = {2023},
  month = sep,
  number = {arXiv:2309.13638},
  eprint = {2309.13638},
  primaryclass = {cs},
  publisher = {arXiv},
  doi = {10.48550/arXiv.2309.13638},
  url = {https://arxiv.org/abs/2309.13638},
  urldate = {2024-02-06},
  archiveprefix = {arXiv}
}
  10. Michael, J., Holtzman, A., Parrish, A., Mueller, A., Wang, A., Chen, A., … Bowman, S. R. (2023). What Do NLP Researchers Believe? Results of the NLP Community Metasurvey. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 16334–16368. Toronto, Canada: Association for Computational Linguistics. 
    @inproceedings{MichaelHoltzmanEtAl_2023_What_Do_NLP_Researchers_Believe_Results_of_NLP_Community_Metasurvey,
  title = {What {{Do}} {{NLP}} {{Researchers Believe}}? {{Results}} of the {{NLP}} {{Community Metasurvey}}},
  booktitle = {Proceedings of the 61st {{Annual Meeting}} of the {{Association}} for {{Computational Linguistics}} ({{Volume}} 1: {{Long Papers}})},
  author = {Michael, Julian and Holtzman, Ari and Parrish, Alicia and Mueller, Aaron and Wang, Alex and Chen, Angelica and Madaan, Divyam and Nangia, Nikita and Pang, Richard Yuanzhe and Phang, Jason and Bowman, Samuel R.},
  year = {2023},
  month = jul,
  pages = {16334--16368},
  publisher = {Association for Computational Linguistics},
  url = {https://aclanthology.org/2023.acl-long.903/},
  address = {Toronto, Canada}
}
  11. OpenAI. (2023). GPT-4 Technical Report. https://doi.org/10.48550/arXiv.2303.08774 
    @misc{OpenAI_2023_GPT-4_Technical_Report,
  title = {{{GPT-4 Technical Report}}},
  author = {OpenAI},
  year = {2023},
  month = mar,
  number = {arXiv:2303.08774},
  eprint = {2303.08774},
  primaryclass = {cs},
  publisher = {arXiv},
  doi = {10.48550/arXiv.2303.08774},
  url = {http://arxiv.org/abs/2303.08774},
  urldate = {2023-06-18},
  archiveprefix = {arXiv}
}
  12. Rogers, A., Gardner, M., & Augenstein, I. (2023). QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension. ACM Computing Surveys, 55(10), 197:1–197:45. https://doi.org/10.1145/3560260 
    @article{RogersGardnerEtAl_2023_QA_Dataset_Explosion_Taxonomy_of_NLP_Resources_for_Question_Answering_and_Reading_Comprehension,
  title = {{{QA Dataset Explosion}}: {{A Taxonomy}} of {{NLP Resources}} for {{Question Answering}} and {{Reading Comprehension}}},
  shorttitle = {{{QA Dataset Explosion}}},
  author = {Rogers, Anna and Gardner, Matt and Augenstein, Isabelle},
  year = {2023},
  month = feb,
  journal = {ACM Computing Surveys},
  volume = {55},
  number = {10},
  pages = {197:1--197:45},
  issn = {0360-0300},
  doi = {10.1145/3560260},
  urldate = {2023-05-22}
}
  13. Sap, M., Le Bras, R., Fried, D., & Choi, Y. (2022). Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs. In Y. Goldberg, Z. Kozareva, & Y. Zhang (Eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (pp. 3762–3780). https://doi.org/10.18653/v1/2022.emnlp-main.248 
    @inproceedings{SapLeBrasEtAl_2022_Neural_Theory-of-Mind_On_Limits_of_Social_Intelligence_in_Large_LMs,
  title = {Neural {{Theory-of-Mind}}? {{On}} the {{Limits}} of {{Social Intelligence}} in {{Large LMs}}},
  shorttitle = {Neural {{Theory-of-Mind}}?},
  booktitle = {Proceedings of the 2022 {{Conference}} on {{Empirical Methods}} in {{Natural Language Processing}}},
  author = {Sap, Maarten and Le Bras, Ronan and Fried, Daniel and Choi, Yejin},
  editor = {Goldberg, Yoav and Kozareva, Zornitsa and Zhang, Yue},
  year = {2022},
  month = dec,
  pages = {3762--3780},
  publisher = {Association for Computational Linguistics},
  address = {Abu Dhabi, United Arab Emirates},
  doi = {10.18653/v1/2022.emnlp-main.248},
  url = {https://aclanthology.org/2022.emnlp-main.248},
  urldate = {2024-07-15}
}
  14. Schaeffer, R., Miranda, B., & Koyejo, S. (2023). Are Emergent Abilities of Large Language Models a Mirage? In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, & S. Levine (Eds.), Advances in Neural Information Processing Systems (Vol. 36, pp. 55565–55581). Curran Associates, Inc. 
    @inproceedings{schaeffer2023emergent,
  title = {Are Emergent Abilities of Large Language Models a Mirage?},
  booktitle = {Advances in Neural Information Processing Systems},
  author = {Schaeffer, Rylan and Miranda, Brando and Koyejo, Sanmi},
  editor = {Oh, A. and Naumann, T. and Globerson, A. and Saenko, K. and Hardt, M. and Levine, S.},
  year = {2023},
  volume = {36},
  pages = {55565--55581},
  publisher = {Curran Associates, Inc.},
  url = {https://proceedings.neurips.cc/paper_files/paper/2023/file/adc98a266f45005c403b8311ca7e8bd7-Paper-Conference.pdf}
}
  15. Shapira, N., Levy, M., Alavi, S. H., Zhou, X., Choi, Y., Goldberg, Y., … Shwartz, V. (2023). Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models. https://doi.org/10.48550/arXiv.2305.14763 
    @misc{ShapiraLevyEtAl_2023_Clever_Hans_or_Neural_Theory_of_Mind_Stress_Testing_Social_Reasoning_in_Large_Language_Models,
  title = {Clever {{Hans}} or {{Neural Theory}} of {{Mind}}? {{Stress Testing Social Reasoning}} in {{Large Language Models}}},
  shorttitle = {Clever {{Hans}} or {{Neural Theory}} of {{Mind}}?},
  author = {Shapira, Natalie and Levy, Mosh and Alavi, Seyed Hossein and Zhou, Xuhui and Choi, Yejin and Goldberg, Yoav and Sap, Maarten and Shwartz, Vered},
  year = {2023},
  month = may,
  number = {arXiv:2305.14763},
  eprint = {2305.14763},
  primaryclass = {cs},
  publisher = {arXiv},
  doi = {10.48550/arXiv.2305.14763},
  url = {https://arxiv.org/abs/2305.14763},
  urldate = {2023-11-22},
  archiveprefix = {arXiv}
}
  16. Veres, C. (2022). Large Language Models Are Not Models of Natural Language: They Are Corpus Models. IEEE Access, 10, 61970–61979. https://doi.org/10.1109/ACCESS.2022.3182505 
    @article{Veres_2022_Large_Language_Models_are_Not_Models_of_Natural_Language_They_are_Corpus_Models,
  title = {Large {{Language Models}} Are {{Not Models}} of {{Natural Language}}: {{They}} Are {{Corpus Models}}},
  shorttitle = {Large {{Language Models}} Are {{Not Models}} of {{Natural Language}}},
  author = {Veres, Csaba},
  year = {2022},
  journal = {IEEE Access},
  volume = {10},
  pages = {61970--61979},
  issn = {2169-3536},
  url = {https://ieeexplore.ieee.org/abstract/document/9794684},
  doi = {10.1109/ACCESS.2022.3182505},
  keywords = {c/position,dl/llm,fw/formal,g/public,q/soc/hype}
}
  17. Wei, J. (2023). Common Arguments Regarding Emergent Abilities. 
    @misc{Wei_2023_Common_arguments_regarding_emergent_abilities,
  title = {Common Arguments Regarding Emergent Abilities},
  author = {Wei, Jason},
  year = {2023},
  month = may,
  journal = {Jason Wei's Blog},
  url = {https://www.jasonwei.net/blog/common-arguments-regarding-emergent-abilities},
  urldate = {2024-05-20},
  langid = {american}
}
  18. Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., … Fedus, W. (2022). Emergent Abilities of Large Language Models. Transactions on Machine Learning Research. 
    @article{WeiTayEtAl_2022_Emergent_Abilities_of_Large_Language_Models,
  title = {Emergent {{Abilities}} of {{Large Language Models}}},
  author = {Wei, Jason and Tay, Yi and Bommasani, Rishi and Raffel, Colin and Zoph, Barret and Borgeaud, Sebastian and Yogatama, Dani and Bosma, Maarten and Zhou, Denny and Metzler, Donald and Chi, Ed H. and Hashimoto, Tatsunori and Vinyals, Oriol and Liang, Percy and Dean, Jeff and Fedus, William},
  year = {2022},
  journal = {Transactions on Machine Learning Research},
  url = {https://openreview.net/pdf?id=yzkSU5zdwD},
  issn = {2835-8856}
}
  19. Zhao, Z., Wallace, E., Feng, S., Klein, D., & Singh, S. (2021). Calibrate Before Use: Improving Few-shot Performance of Language Models. Proceedings of the 38th International Conference on Machine Learning, 12697–12706. PMLR. 
    @inproceedings{ZhaoWallaceEtAl_2021_Calibrate_Before_Use_Improving_Few-shot_Performance_of_Language_Models,
  title = {Calibrate {{Before Use}}: {{Improving Few-shot Performance}} of {{Language Models}}},
  shorttitle = {Calibrate {{Before Use}}},
  booktitle = {Proceedings of the 38th {{International Conference}} on {{Machine Learning}}},
  author = {Zhao, Zihao and Wallace, Eric and Feng, Shi and Klein, Dan and Singh, Sameer},
  year = {2021},
  month = jul,
  pages = {12697--12706},
  publisher = {PMLR},
  issn = {2640-3498},
  url = {https://proceedings.mlr.press/v139/zhao21c.html},
  urldate = {2022-06-24},
  langid = {english},
  keywords = {!}
}

Share / cite / discuss this post

]]>Anna RogersI am joining ACL Rolling Review2024-03-25T13:00:00-04:002024-03-25T13:00:00-04:00https://hackingsemantics.xyz/2024/joining-arrIt’s official: I joined the ACL Rolling Review team as an editor-in-chief, and I’d like to share some brief thoughts on this.

When ACL Rolling Review was first launched, I wasn’t its biggest fan. The core motivation seemed to be that it would reduce the reviewer workload, and I am not convinced that that this goal is either achievable with ARR, or that it has in fact been achieved.

However, from bitter experience as a program chair of ACL’23, I am firmly convinced that we as a scientific community need a centralized and continually improving conference review system. If you have not been in that role, trust me: there are hundreds of things that you as a chair have to (a) know about, (b) remember about, (c) care enough about when it’s 2am, (d) know how to do well, (e) have support for in the system you’re using, (f) potentially argue with other chairs and the whole community about. This is why conference peer review is just way too complex to have a new set of chairs do it for the first time for every single conference - and too many people waste time and effort when anything goes wrong.

ARR is by no means perfect. I hope to help with fixing at least some of the issues while I’m there, but I know that even after I and everybody else do everything we can - it still won’t be perfect. Still, it’ll be better, and we as a community will have an iteratively improving system that gradually accumulates documentation, software support, and people familiar with it. Just this, by itself, is a huge step in the right direction. The service time that people invest in this is a precious resource, and it should be reused and iterated on as much as possible.

Here are some of the things that I am hoping to help the team to get done:

  • updating the Responsible NLP checklist (some questions are up for rethinking), and the way it is used
  • updating the ARR reviewer guidelines, in particular with respect to generative AI (in consultation with the new ACL committee on publication ethics)
  • implementing the structured author complaints to chairs, which were very well-received at ACL’23, and which would allow the authors to flag reviews for specific types of issues (see sec.5.3 of ACL’23 report)
  • figuring out ways to support the chairs in implementing the new ACL anonymity policy (according to which borderline anonymized papers have an advantage over preprinted papers)
  • analyzing the effects of the change in anonymity policy
]]>Anna Rogers[ACL 2023] Peer Review Report2023-07-10T13:00:00-04:002023-07-10T13:00:00-04:00https://hackingsemantics.xyz/2023/peer-review-reportThis post (at ACL conference website) summarizes the analysis of ACL’23 peer review process: https://2023.aclweb.org/blog/review-report/. The full analysis is available in this huge report that is up on ACL Anthology:

Anna Rogers, Marzena Karpinska, Jordan Boyd-Graber, and Naoaki Okazaki. 2023. Program Chairs’ Report on Peer Review at ACL 2023. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page xl–lxxv, Toronto, Canada. Association for Computational Linguistics. https://aclanthology.org/2023.acl-long.911/

]]>Anna RogersClosed AI Models Make Bad Baselines2023-04-03T13:00:00-04:002023-04-03T13:00:00-04:00https://hackingsemantics.xyz/2023/closed-baselines

This post was authored by Anna Rogers, with much invaluable help and feedback from Niranjan Balasubramanian, Leon Derczynski, Jesse Dodge, Alexander Koller, Sasha Luccioni, Maarten Sap, Roy Schwartz, Noah A. Smith, Emma Strubell (listed alphabetically)
Header image credit: Sasha Luccioni

What comes below is an attempt to bring together some discussions on the state of NLP research post-chatGPT.1 We are NLP researchers, and at the absolute minimum our job is to preserve the fundamentals of scientific methodology. This post is primarily addressed to junior NLP researchers, but is also relevant for other members of the community who are wondering how the existence of such models should change their next paper. We make the case that as far as research and scientific publications are concerned, the “closed” models (as defined below) cannot be meaningfully studied, and they should not become a “universal baseline”, the way BERT was for some time widely considered to be. The TLDR for this post is a simple proposed rule for reviewers and chairs (akin to the Bender rule that requires naming the studied languages):

That which is not open and reasonably reproducible cannot be considered a requisite baseline.

By “open” we mean here that the model is available for download, can be run offline (even if it takes non-trivial compute resources), and can be shared with other users even if the original provider no longer offers the model for download. “Open” models support versioning, and document for each model version what training data they used. A model is “closed” if it is not open.

By “reasonably reproducible” we mean that the creators released enough information publicly such that the model can be reproduced with the provided code, data, and specified compute resources, with some variation reasonably expected due to hardware/software variance, data attrition factors and non-determinism in neural networks. For instance, reproducing BLOOM would require a super-computer - but at least theoretically it is possible, given the measures to open-source the code, collect and document the data. So it is “reasonably reproducible” by our definition, even though not everybody could do it.

Relevance != popularity

Here’s a question many graduate students in NLP have been asking themselves recently:

This anxiety seems to be due partly to the fact that in our field, “relevance” has been extremely popularity-driven. For the last decade, there has always been a Thing-Everybody-Is-Talking-About: a model or approach that would become a yardstick, a baseline that everybody would be wise to have in their papers to show that what they’ve done is a meaningful improvement. This can be understood, since one of the driving values of the ML community is improving upon past work – otherwise, how would we know we are making progress, right? Post-2013 we had word2vec/GloVe, then there was a similar craze about BERT. Then GPT-3. And now – ChatGPT and GPT-4.

Why does this happen? There are two lines of reasoning behind this:

  1. The-Thing-Everybody-Is-Talking-About is either likely to be truly state-of-the-art for whatever I’m doing, or a reasonable baseline, so I better have it in my paper and beat it with my model.
  2. As an author, my chances of publication depend in part on the reviewers liking my work, and hence the safest bet for me is to talk about something that most people are likely to be interested in - a.k.a The-Thing-Everybody-Is-Talking-About.

(b) is actually a self-fulfilling prophecy: the more authors think this way, the more papers they write using The-Thing-Everybody-Is-Talking-About, which in turn reinforces the reviewers in the belief that that thing is really prerequisite. We see this cycle manifested as a mismatch between the beliefs of individual community members and their perception of others’ views on what research directions should be prioritized (e.g. focus on benchmarks or scale), as documented in the NLP Community Metasurvey. Though it takes effort, members of the research community can push back against that kind of cycle (and we will discuss specific strategies for that below). As for (a) - it made sense while The-Thing-Everybody-Is-Talking-About was actually something that one could meaningfully compare to.

The main point we would like to make is that this kind of reasoning simply no longer applies to closed models that do not disclose enough information about their architecture, training setup, data, and operations happening at inference time. It just doesn’t matter how many people say that they work well. Even without going into the dubious ethics of commercial LLMs, with copyright infringement lawsuits over code and art already underway, and unethically sourced labeled data – the basic research methodology demands it. Many, many people are bringing up the fact that as researchers, we are now in an impossible position:

  • We have very little idea what these models are trained on, or how:
  • The said black box is constantly changing:
  • Both our incoming prompts and outgoing answers may be undergoing unspecified edits via unspecified mechanisms. E.g. chatGPT “self-censors” with content filters which people have so much fun bypassing, and has proprietary prompt prefixes:

Yes, these models do seem impressive to many people in practice – but as researchers, our job is not to buy into hype. The companies training these models have the right to choose to be wholly commercial and therefore not open to independent scrutiny – that is expected of for-profit entities whose main purpose is to generate profits for their stakeholders. But this necessarily means that they relinquish the role of scientific researchers. As Gary Marcus put it,

I don’t expect Coca Cola to present its secret formula. But nor do I plan to give them scientific credibility for alleged advances that we know nothing about.

Why closed models as requisite baselines would break NLP research narratives

To make things more concrete, let us consider a few frequent “research narratives” in NLP papers, and how they would be affected by using such “closed” models as baselines. We will use GPT-4 as a running example of a “closed” model that was released with almost no technical details, despite being the subject of a 100-page report singing its praises, but the same points apply to other such models.

“We propose a machine learning model that improves on the state-of-the-art”:

  • To make the claim that our algorithm improves over whatever it is that a commercial model is doing, we need to at least know that we are doing something qualitatively different. If we are proposing some modification of a currently-popular approach (e.g., Transformers), without documentation, we simply cannot exclude that the “closed” model might be doing something similar.
  • Even if we believe that we are doing something qualitatively different, we still need to be able to claim that any improvements are due to our proposed modification and not model size, the type and amount of data, important hyperparameters, “lucky” random seed, etc. Since we don’t have any of this information for the closed “baseline”, we cannot meaningfully compare our model to it.
  • And even if we ignore all the above factors – to make a fair comparison with these models on some performance metric, we have to at least know that neither of our models has observed the test data. Which, for the “closed” model, we also don’t know. Even OpenAI itself was initially concerned about test data contamination with GPT-3, which could not possibly have improved - especially after the whole world has obligingly tested chatGPT for months. And it hasn’t improved.

The only thing that we as model developers can learn from the existence of GPT-4, is that this is the kind of performance that can be obtained with some unspecified combination of current methods and data. An upper bound or existence proof, which seems higher than existing alternatives. Upper bounds are important, and could serve as a source of motivation for our work, but they cannot be used as a point of comparison.

“We propose a new challenging task/benchmark/metric”:

Constructing good evaluation data is very hard and expensive work, and it makes sense to invest in it when we believe that it can be used as a public benchmark to measure progress in NLP models, at least for a few months. Examples of such benchmarks that have driven NLP research in the past include SQuAD, GLUE and BigBench. But public benchmarks can only work if the test data remains hidden (and even then eventually people evaluate too many times and start to implicitly overfit). This is obviously incompatible with the scenario where the developer of the popular “closed” models, only accessible via API, keeps our submitted data and may use it for training. And unless the models explicitly describe and share their training data, we have no way of auditing this.

This means that our efforts will be basically single-use as far as the models by that developer are concerned. The next iteration will likely “ace” it (but not for the right reasons).

Let us consider OpenAI policies in this respect:

  • ChatGPT by default keeps your data and may use it for training. It is said to provide an opt-out of data collection.
  • The OpenAI API policy was updated on March 1 2023, and currently states that by default data is not retained and not used for training. Whatever was submitted before this date, can be used, so we can safely assume that much if not all of existing public benchmark data has been submitted to GPT-3 since 2020, including the labels or “gold” answers - at least those that were used as few-shot prompts. Interestingly, OpenAI then uses the contamination as a reason to exclude some evaluations but not others: the GPT4 tech report says that they did not evaluate on BIG-bench because of data contamination (in v.3 of the report it’s footnote 5 on p.6), although they do present their results for 100% contaminated GRE writing exams (Table 9).

The overall problem is that opt-outs and even opt-ins are not sufficient in the case of something meant to be a public benchmark: as dataset creators, the future of our work might be affected not only by our own use of our data - but also by anybody else using it! It takes just one other researcher who wasn’t careful to opt-out, or wasn’t able to – and our data is “poisoned” with respect to future models by that developer. Even if only some few-shot examples are submitted, they might be used to somehow auto-augment similar user prompts. Last but not least, if we make our data public, the model developers themselves could also proactively add it to the training data, looking to improve their model. If the labels or the “gold” answers are not public for an important benchmark, it would be worthwhile for them to create some similar data.

It’s unclear yet how to solve this problem. Perhaps there will soon appear some special version of robots.txt that both prohibits use for AI training, and requires that any resharing of this data keeps the same flag. And, hopefully, the large companies will eventually be required to comply, and be subject to audits. In the short-term, it seems like the only option is to simply not trust or produce benchmark results for models where test-train overlap analysis cannot be performed.

“We show that model X does/doesn’t do Y: (model analysis and interpretability)

Since we only have access to GPT-4 via the API, we can only probe model outputs. If the plan is to use existing probing datasets or construct new ones, we have the same resource problem described above (the previously used probing datasets might have been trained on, the previously used techniques could have been optimized for, the new work will be single-use, and still have the train-test overlap problem to an unknown extent).

Furthermore, at least some of these models seem to intentionally not produce identical outputs when queried with the same probe and settings (perhaps via random seeds or different versions of the model being used in parallel). In this case, whatever results we get may already be different for someone else, which puts our basic conclusions at risk. This could include, for instance, the reviewers of the paper, who will rightfully conclude that what our report may not be true. Moreover, if the developer keeps tweaking the model as we go, then by the time we finish writing the paper, the model could change (perhaps even based on our own data). Which would also make our work not only obsolete before it is even reviewed, but also incorrect.

This issue might be addressed by “freezing” given versions of the model and committing to keep them available to researchers, but there is hardly any incentive2 for for-profit companies to do so. For instance, some popular models including Codex/code-davinci-002 have already been deprecated. We also have no public information about what changes lead or do not lead to a new version number (and it is likely that at least the filters are updated continually, as users are trying to break the model).

Last but not least, consider the effect of showing that model X does/doesn’t do Y:

  • “Model does Y”: without test-train overlap guarantees this is not necessarily a statement about the model. For example, chatGPT was reported to be able to play chess (badly). That looks unexpected of something that you consider a language model, but if you knew that it has seen a lot of chess data - it is hardly newsworthy that a language model can predict a plausible-looking sequence of moves. Basically, instead of discovering properties of a language model (which could be a research finding), we’re discovering that the internet dump it was trained on contained some chess data.
  • “Model doesn’t do Y”: by collecting cases where the model seems to fail, we implicitly help the commercial entity controlling that model to “fix” those specific cases, and further blur the line between “emergent” language model properties and test cases leaked in training. In fact, GPT-4 was already trained on user interactions gathered during the mass testing of ChatGPT, which provided Open AI with millions of free examples, including “corrected” responses to prompts submitted by users. In the long run, our work would make it harder for the next researcher to examine the next “closed” model. What’s even worse, it would decrease the number of easy-to-spot errors that might prevent ordinary users from falling for the Eliza effect, hence increasing their trust in these systems (even though they are still fundamentally unreliable).

In summary, by showing that a closed model X does/doesn’t do Y we would likely not contribute to the general understanding of such models, and/or exacerbate the evaluation issues.

“We show that model X is (un)fair/biased etc”: (AI ethics)

Let us say that we somehow showed that the closed model yields some specific type of misinformation or misrepresents a given identity group (as it was done e.g. for anti-Muslim bias in GPT-3). The most likely outcome for such work is that this specific kind of output will be quickly “patched”, perhaps before we even publish the paper. The result is that (a) our hard work is short-lived, which may matter for researcher careers, (b) we actively helped the company make their model seem more ethical, while their training data probably didn’t fundamentally change, and hence the model probably still encodes the harmful stereotypes that could manifest themselves in other ways. Consider how in Dall-E 2 the gender and identity terms were randomly added to make outputs seem more diverse, as opposed to showing the default identity groups (read: White Men).

So, should we just forgo studying “closed” models from the ethics angle? Of course not: independent analysis on commercial systems is strictly necessary. But we need to figure out ways to do this without providing companies with free data with which to mask the symptoms of the underlying problem. Here are some alternatives that may lean on skillsets that NLP researchers are still developing, and perhaps will be strengthened by collaborations with experts in HCI and social sciences:

  • User studies on whether people trust the over-simplified chatbot answers, how likely they are to verify information, whether students use it in ways that actually improves their learning outcomes, and interventions that promote safer use practices. This kind of work focuses on the potential effects of these models, given the known phenomenon of automation bias, and any negative findings can only be refuted with a public user study.
  • Discussing and documenting instances of real-world harms, where they can be traced to the model (akin to the Stochastic Parrots paper). Ideally, such cases would require not only a fix, but also public acknowledgment and hopefully compensation.
  • User studies of various demographic cohorts, to see if the system works equally well for them in different real-world tasks: something with qualitative evaluation, where a fix would require obtaining better training data for that cohort. But this kind of work would need to somehow avoid producing too much concrete evidence that could be used to simply “patch” the output.
  • Studies not just of these systems, but on their intended and real impact on society. We need a lot of research on system-level issues where a “fix” would require changes to the business model and/or the way these systems are presented and marketed. An obvious example is the jobs that are too risky to be automated with the unreliable, biased, hallucination-prone systems that we currently have. For instance, do policy-makers jump on the opportunity to hire fewer teachers, and what kinds of schools are more likely to be sent down that path?

“We develop a more efficient solution than model X”:

The reviewers would likely (and rightly) expect us to show that we improve efficiency while maintaining a similar level of performance, which means we inherit all the above evaluation issues. Also, we likely don’t even have enough details about the training of the “baseline”, including its computational costs, the amount and source of energy invested in it, etc.

We Do Have Options!

Dear members of the NLP community: the good news is that if you’d like to do… you know… actual research on language models, you do have open options, and more of them will probably be coming, as the cost of training goes down. Here are a few examples of models that come not only with reasonable descriptions of their training data, but even tools to query it:

Model Type Size Data sourcing Corpus Searchable training data
BLOOM multilingual LLM 560M-176B documentation efforts ROOTS Roots Search Tool
GPT-Neo models mostly-English LLMs 125M-2.7B Pile datasheet The Pile The Pile Data Portraits
T5 English LLM 60M-11B partial C4 documentation C4 C4 search


What about the reviewers who might say “but where’s GPT-4?” Here’s what you can do:

  • Preemptively discuss in your paper why you don’t provide e.g. chatGPT results as a baseline, before your paper is submitted. If necessary, use the arguments in this post in your rebuttals to reviewers.
  • Preemptively raise the issue with the chairs of the conference you plan to submit to, to ask if they have a policy against such superficial popularity-driven reviews. The ACL 2023 policy didn’t cover this, since the problem became apparent after the submission deadline, but it can be extended by future chairs. We will be following any policy discussions related to this in ACL conferences; if you have any comments, or if there are any major developments and if you’d like us to keep you in the loop - please use this form.
  • As a reviewer or chair, if you see someone insisting on closed baselines - side with the authors and push back.
  • Discuss these matters openly in your own community; as reviewers, we can continue to educate and influence each other to drive our norms to a better place.

Another question outside of the scope of this post, but that could be brought up for community discussion in the future, is whether the “closed” models should be accepted as regular conference submissions (in direct competition with “open” work for conference acceptance and best paper awards) – or perhaps it is time to reconsider the role of the “industry” track.

Our community is at a turning point, and you can help to direct the new community norms to follow science rather than hype – both as an author and as a reviewer. The more people cite and study the best available open solutions, the more we incentivize open and transparent research, and the more likely it is that the next open solution will be much better. After all, it is our tradition of open research that has made our community so successful.

Addendum: counter-arguments

Train-test overlap and uninspected training data has always been an issue, ever since we started doing transfer learning with word2vec and onwards. Why protest now?

People have in fact been raising that issue many times before. Again, even OpenAI itself devoted a big chunk of the GPT-3 paper to the issues with benchmark data contamination. The fact that an issue is old doesn’t make it a non-issue; it rather makes us a field with a decade of methodological debt, which doesn’t make sense to just keep accruing.

The-Closed-Model-Everybody-Is-Talking-About does seem to work better for this task than my model or open alternatives, how can I just ignore it and claim state-of-the-art?

Don’t. “State-of-the-art” claims expire in a few months anyway. Be more specific, and just show improvement over the best open solution. Let’s say that in your task ChatGPT is clearly, obviously better than open alternatives, based on your own small testing with your own examples. What you don’t know is whether this is mostly due to some clever model architecture, or some proprietary data. In the latter case, your scientific finding would be… that models work best on data similar to what they were trained on. Not exactly revolutionary.

Also, ask yourself: are you sure that the impressive behavior you are observing is the result of pure generalization? As mentioned above, there is no way to tell how similar your test examples are to the training data. And that training data could include examples submitted by other researchers working on this topic, examples that were not part of any public dataset.

The-Closed-Model-Everybody-Is-Talking-About does seem to work better for this task than my model or open alternatives, how can I just ignore it and not build on it?

That has indeed been the pathway to many, many NLP publications in the past: take an existing problem and the newest thing-that-everybody-is-talking-about, put them together, show improvement over previous approaches, publish. The problem is that with an API-access closed model you do not actually “build” on it; at best you formulate new prompts (and hope that they transfer across different model versions). If your goal is engineering, if you just need something that works - this might be sufficient. But if you are after a scientific contribution to machine learning theory or methods - this will necessarily reduce the perceived value of your work for the reviewers. And if the claim is that you found some new “behavior” that enables your solution, and hasn’t been noticed before - you will still need to show that this “behavior” cannot be explained by the training data.

Whatever we may say, The-Closed-Model-Everybody-Is-Talking-About is on everyone’s minds. People are interested in it. If I don’t publish on it, somebody else will and get more credit than me.

Well, that is a personal choice: what do you want the credit for and who do you want recognition from? Publishing on the “hottest” thing might work short-term, but, as shown above, if we simply follow the traditional NLP research narratives with these models as new requisite baselines in place of BERT, our work will be either fundamentally divorced from the basic principles of research methodology, or extremely short-lived, or both. Imagine looking at the list of your published papers 10 years from now: do you want it to be longer, or containing more things that you are proud of long-term?

Are there other ways to study these models that would not run into these issues? We discussed some such ways for ethics-oriented research, perhaps there are other options as well.

Can’t we just study very made-up examples that are unlikely to have been in training data?

First of all, if the point is to learn something about what that model does with real data - very artificial examples could be handled in some qualitatively different way.

Second, at this point you need to be sure that you are way more original than all those other folks who tested chatGPT for several months. Especially since the data used for RLHF comes from interactions with GPT3 - perhaps even your own!

Third, you would still need to know what part actually hasn’t been seen. For example, ChatGPT was reported to write a fable about a peanut butter sandwich stuck in a VCR, in King James Bible style, and that example got subsequently shared in dozens of media articles. This is a cool example, but what exactly is it that we believe to be impressive? The style transfer, the knowledge that things get stuck in VCRs, the plausible instructions? The degree of impressiveness of each one of these depends on what was in the training data. And even the impressiveness of the ability to tie these things together still depends on what combinations of “skills” were seen in training, and whether this is in fact a pure language model behavior, and not some combination of pipeline components.

We tried to reproduce that answer, but accidentally typed “CVR” instead of “VCR”. The result was very illuminating. We got generic instructions that could have come from something like WikiHow: how to wipe off something sticky off something electric. Which, of course, is no good here: a sandwich includes a large piece of bread, which you would need to remove by hand rather than by wiping it off. But the best part is that the model later “admitted” it had no idea what “CVR” was! (Indeed, large language models don’t inherently “know” anything about the world). And then, when prompted for “VCR”, apparently the directive to maintain consistency within the dialogue overruled whatever it could have said about ‘VCR”… so we got the same wrong instructions.

What did go without a hitch is the paraphrasing in the King James style. But it’s hard to imagine that paraphrasing was not an intended-and-trained-for “capability”, or that this style was not well represented in a large web-based corpus - o ye of little faith.

Does it work well? Yes. Is it a magical “emergent” property? No. Can we develop another paraphrasing system and meaningfully compare it to this one? Also no. And this is where it stops being relevant for NLP research. That which is not open and reasonably reproducible cannot be considered a requisite baseline.

Share / cite / discuss this post

Notes

  1. The work on this post started a while ago, and has nothing to do with either longtermism or the plea for democratization of GPU resources. ↩

  2. There are in fact incentives for such companies to close and deprecate previous versions of their models, for the sake of (a) reducing attack surface, (b) capping technical debt. These are legitimate concerns for commercial entities, but they are intrinsically in tension with their models being objects of scientific inquiry. ↩

]]>Anna Rogers[ACL 2023] Paper-Reviewer Matching2023-01-12T12:00:00-05:002023-01-12T12:00:00-05:00https://hackingsemantics.xyz/2023/reviewer-matchingAs a program chair of ACL’23, I was the lead author for this blog post on the conference website that summarized our approach to peer-review matching: https://2023.aclweb.org/blog/reviewer-assignment/ I was also the lead developer of this approach to matching. Post-mortem analysis of how it worked is available in this report:

Anna Rogers, Marzena Karpinska, Jordan Boyd-Graber, and Naoaki Okazaki. 2023. Program Chairs’ Report on Peer Review at ACL 2023. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page xl–lxxv, Toronto, Canada. Association for Computational Linguistics. https://aclanthology.org/2023.acl-long.911/

]]>Anna Rogers[ACL 2023] Generative AI Policy2023-01-10T12:00:00-05:002023-01-10T12:00:00-05:00https://hackingsemantics.xyz/2023/ai-policyThis blog post (on the conference website) summarized our approach to the use of generative AI in ACL conference submissions and reviewing: https://2023.aclweb.org/blog/reviewer-assignment/. I was its lead author.

]]>Anna RogersThe attribution problem with generative AI2022-11-01T04:00:47-04:002022-11-01T04:00:47-04:00https://hackingsemantics.xyz/2022/attribution

When the discussion about large pre-trained generative models hits the question of “what about all this work of artists, programmers and writers that is used in commercial products/models without their knowledge or consent?”, one of the arguments for why this is ok is the comparison of such models to latent search engines. It goes something like this:

As a human, you can and do search for inspiration in other people’s writing, code snippets and art. A generative model is similar, it just provides a convenient interface for a search over a huge amount of data as you go.

Side note: this is about the “latent search” or “synthesis” of the training data that the generative models perform in the process of their regular generation process. There is a related, but separate discussion about using models as a replacement for index-based search engines. For example, (Metzler, Tay, Bahri, & Najork, 2021) sets out a vision of models as “domain experts” generating authoritative answers to any questions that the user might have. (Shah & Bender, 2022) challenge this vision by discussing the many kinds of behavior that search users need to undertake which would simply not be supported by a “domain expert” model trying to generate one definitive answer (e.g. learning more before refining their question, considering the list of options, the incentives behind different sources, etc).

So what’s wrong with the “latent search engine” view of generative models?

It is obviously true that autoregressive language models do search for the most probable completion based on the prompt. And it is equally true that human writing and art is conditioned on the inputs encountered by the said humans in their lives, as well as relevant inputs that were deliberately sought out in response to a particular challenge. In literary studies and art there is the notion of intertextuality (Bakhtin, 1981; Kristeva, 1980; Barthes, 1977), covering a wide range of ways in which different texts/artworks are related (or perceived to be related by the reader), such as allusion, quotation, parody etc.

But there are a few important limitations to this analogy, including the fundamental differences in the mechanism behind the generative models and the human inspiration, the potential scale of societal impact for commercial models, and a very different set of stakeholders and benefactors. This post focuses on one particular point in which the search engine analogy breaks down: the attribution problem.

The attribution problem

When you use a search engine, you find a specific idea, artwork or code snippet for which you clearly see the source. There is a reference (even if the source is only known as stackoverflow user02348). Importantly, there is zero illusion that any thought/artwork/code is just there to be freely appropriated as your own work. If your search space is not the web but your own memory or life experience, you still usually know the sources for things that are citation-worthy (or you go on Twitter asking people “what was that movie/book/project/paper that had/did X?”)

If you are a researcher, you likely have something like Zotero to track references for you, and a huge database of books and papers. Even if your source by itself was sourced from elsewhere, and even if someone said the same thing without your knowing it – your credibility before your readers (and yourself) requires that you disclose the references that you do know. In the process of doing so, you will necessarily have to make sure that you actually believe the source to be reliable, and that you are aware of its role in your own reasoning.

Note that the attribution problem goes both ways: claiming full credit for the result of your work is possible if and only if you know and cite your sources. This is completely orthogonal to the degree of originality. Let’s say I publish this blog post and then I find that exactly the same text has already been published by someone else: I would still know that what I published was my own work. On the other hand, if instead of writing this blog post I asked GPT-3 to generate it, and even got exactly the same result - I could not claim any contribution at all. In publishing that text as my own, my role would be only to say “I interpret this as coherent text and agree with its content” (that’s what I think Jack Clark did when he used synthetic text as part of his Congress testimony). And what if I used GPT-3 to get “ideas” about what to write next - i.e. generating coherent sections of text that I would then edit - what exactly would I claim then? Not sure. But the ideas, the style, the amount of background knowledge etc. would all be only partially mine.

There was a recent Reddit discussion of how GPT-3 starts to get popular with students aiming to avoid doing their essays. Apart from the students’ completely misunderstanding the point of the exercise, and the waste of the teachers’ time, this discussion highlighted an idea that the AI-assisted writer actually gets the credit not for the writing, but for the “street smarts”: their ability to game the system and get high grades, even if their language skills are not so great. Some might be tempted to say that this is just like using a spellchecker or a service like Grammarly to improve one’s writing, but it seems clear that generating a full or partial draft is qualitatively different: you get help not only with the linguistic form, but also the content.

But aren’t people doing the same thing?

Yes, of course people build on other people’s work all the time. If you want to use something, you can do that - but society has worked out quite a few norms about how much of your own work has to go into the result. And because those norms have evolved over time, we are usually quite aware of our sources. Maybe not all of them, but the important ones for sure.

Any musician has listened to other music that influenced them. Art students go to galleries, and creative writing students read other people’s books. They and/or their teachers may even deliberately curate what they are exposed to, so as to get to a particular result. And all of them can give an interview with some account of their formative influences. That account will be incomplete and not coinciding with what the critics think, but that’s not the point: only that people do generally retain at least some memories of things that ended up very important for them.

Another key difference is that if they aim to be an original artist/musician/writer, while they build on prior work, the point is always to add enough of their own thinking that the next generation has something to similarly learn from them (and not only their ‘sources’). It is far from clear that we get that same degree of creativity from generative models.

With regard to AI art in particular: I’m not an artist at all, but it seems that it’s actually the style (shapes, color schemes etc) rather than just the particular images/artifacts that the artist spends a lifetime developing, and that also brings them professional recognition. They seem to very much disagree that it is ok to just appropriate that (Heikkilä, 2022). Spawning AI has built a tool for artists to detect when their work has been part of popular training datasets.

In conclusion: no, generative models are not doing the same kind of latent search over the possible things they could “say” as the humans do when they produce texts or art. A key difference is that for humans it is not only a cognitive activity driven by content considerations, but also a social activity. We are acutely aware of when attribution is needed, we provide that attribution, and we expect attribution in return. Granted, different people may have a different sense of when attribution is appropriate (based on their personal experience, familiarity with a subject, the social norms in their environment, etc.) - but that does not make the fundamental principle any less real.

Counter-arguments

In the spirit of discussion, here are some of the counter-arguments I have seen, and my responses to them.

Generative models are sufficiently creative

To claim that a generative model is sufficiently creative to not worry about attribution, we would first need to define “creativity”. Some bring up examples like DALL-E’s avocado chairs. To me, the creativity here is exhibited by the human who formulated the crazy prompt, while the model demonstrates compositionality in being able to recombine the visual “concepts” it had learned (in this case it had learned “chair”, “wood”, and “avocado”, as well as the visual schema “chair + material”). Most of us cannot draw well, so pretty much any execution would look impressive. But consider what this compositional skill would look like in the language domain, where we are all proficient enough: the model learned “John had a coffee” and “Mary had a tea”, and it was then able to produce “John had a tea”. Does that look as impressively creative as the avocado chair image?

I also wouldn’t interpret creativity as randomness (e.g. as controlled by the temperature setting of GPT-3). If I were to get the writer’s block and had to resort to random writing prompts to get me unstuck, that would rather be a symptom of a lack of creativity, wouldn’t it? Furthermore, with the current models increasing the randomness of generation is likely to sacrifice the factual correctness of the generated data, as it necessarily moves further away from the training distribution - and there are no mechanisms for conceptual consistency. Creativity/nonsense is not an acceptable trade-off in most scenarios where the generated text is anchored to the real world or some long-form narrative.

Finally, “creativity” may be discussed in publications on AI art or AI-human collaborations as some external characteristic of the generated text/artwork, scored by critics on some aesthetic dimensions, as in (Hitsuwari, Ueda, Yun, & Nomura, 2022). I would argue that this is also not the relevant notion of creativity for this discussion. Since we are talking about a machine learning system, evaluation of any aesthetic properties of its output has the same problem as any other benchmarks: unless we know what the system saw in training, we cannot tell whether it actually acquired some ability or just parrots the seen examples. So far I have seen no studies of very large models given all their training data (given that this data is typically not made fully publicly available in a query-able form). Since fundamentally the current models are optimized to produce a statistically likely completion of the current prompt, the burden of proof is on the side that claims creativity.

What we do know from the studies with smaller models is that they can and do reproduce passages of training data verbatim (Carlini et al., 2021; Carlini et al., 2022), inter alia. The capacity for memorization and, hence, plagiarism would increase with size (Lee, Le, Chen, & Lee, 2022). Given that, I would argue that even if we had some general proof of capacity for generative models to synthesize meaningful and original content, it would not be enough: after all, humans also can be creative, but teachers still suspect student plagiarism on a case by case basis. For a statistical learner, how creative (or trustworthy) a given generation is would likely depend on how much evidence it had, and how similar the different datapoints were. So unless the company selling the model provides some guarantees of originality in specific cases, it simply passes the responsibility for the potential plagiarism on to its unwitting customers.

Can we just add references?

When presenting their vision of a “domain expert” end-to-end information retrieval, (Metzler, Tay, Bahri, & Najork, 2021) argue for a model that does add some references, and moreover strives to present “both sides” for controversial topics. Perhaps it would be an easy fix for the attribution problem - if we just added pointers to the training data examples that were the most similar to the generated output?

Let’s say you’re writing a deep learning blog post about self-attention in Transformers. Let’s say that your “writing aid” model would give you the following combination of sentences:

The self-attention mechanism in Transformers is able to compute pair-wise relations between patches globally, consequently achieving feature interactions across a long range. It is… regarded as a mapping of query and key/value pairs to an output, each of which being represented by a vector. A well-known concern with self-attention… is the quadratic time and memory complexity, which can hinder model scalability in many settings.

All of these sentences actually come from different research papers. Augmented with links to those papers, the same paragraph would look like this:

The self-attention mechanism in Transformers is able to compute pair-wise relations between patches globally, consequently achieving feature interactions across a long range. [https://arxiv.org/pdf/2201.00462v2.pdf] It is… regarded as a mapping of query and key/value pairs to an output, each of which being represented by a vector [https://arxiv.org/pdf/1807.03052.pdf]. A well-known concern with self-attention… is the quadratic time and memory complexity, which can hinder model scalability in many settings. [https://arxiv.org/pdf/2009.06732.pdf].

The key difference is that the first paragraph looks like something actually “done” by the model, and you might be tempted to actually use it. The references destroy the illusion of the attribution-free text: unless you are comfortable simply copying phrases from other people’s work, the “writing aid” illusion falls apart.

Admittedly, this example is exaggerated: perhaps only some part of the generated text would be so clearly plagiarized. Perhaps it would only happen occasionally. But without these references the attribution norms in scientific community would still be broken. And with them, it would mean that you’re relying on GPT-3 for the high-level thinking behind your research. Which would make sense depending on (a) on the degree to which you believe it capable of such thinking, (b) if so - the degree to which you are comfortable taking credit for thinking that is not your own.

(Shah & Bender, 2022) make the case that the references approach is insufficient even for the “domain expert” QA model envisaged by (Metzler, Tay, Bahri, & Najork, 2021): the model may end up being the arbiter of truth for cases that are far from resolved, may present “both sides” on topics like the flat earth theory, and may obscure the real sources of information behind the citation (e.g. something called “XYZ clinic” may actually be a homeopathy provider with no medical credentials). Of course, there are cases in which the answer is straightforward enough to trust the current models with - but unfortunately we can’t easily tell which cases are “safe”.

If you go deep enough, everything has a reference.

If you go deep enough, everything has a reference. Nobody expects attribution for basic algebra or the English alphabet. Nobody has ethical qualms about writing with Grammarly or spell checkers. Why demand attribution for abstract ideas or artistic styles?

True, when we write academic articles nowadays, nobody expects you to provide the trail of references all the way down to Aristotle. But few people would say that taking someone’s recent NeurIPS paper and republishing it would be ok. Yes, it is a continuum, but it’s still real.

What exactly is common knowledge and what deserves a reference at a given point in time varies by person, depending on their domain knowledge and principles. Still, everybody has a fairly clear idea of what their own boundaries are. Would you personally be comfortable with changing some variable names in a StackOverflow snippet and passing it as your own work? Would you tell your child it’s ok to copy-paste essay passages from public domain sources - after all, it’s not illegal? How about if you hear an apt metaphor in someone’s keynote that you haven’t heard anywhere else - would you say that it’s “just English” and use it as your own? Whatever your answers are to these questions - you have these answers, which means that you have your own attribution norms. They matter to you. And part of the reason you have this specific set of norms is that you know that this is what the people around you expect.

“Fair use”

This is just luddism. The printing press put the calligraphers out of a job, and the world is better off this way. The notions of copyright and intellectual property are obsolete and will soon dissolve in “fair use”. If that puts artists/writers/programmers out of work - so what, society just needs to adapt.

The printing press wasn’t literally powered by the work of all the calligraphers in the world, taken and used commercially without their knowledge or consent - especially at a time when at least some protection against that already exists in contemporaneous laws. “Fair use” may sound like a reasonable approach for academic research or for individual creators producing AI-assisted content (with proper source attribution), but that’s not what is under discussion - it’s the AI companies’ right to use any data they can get hold of to train commercial models, without sharing any proceeds with the original creators or even letting them know their work was used. That fight is far from over, and the few available court decisions (such as the ongoing LinkedIn case) are on a case-by-case basis rather than something that the companies can already use as a blanket permission. An investigation for an actual lawsuit is underway with respect to GitHub CoPilot (Joseph Saveri Law Firm & Butterick, 2022).

I am not sure what kind of adaptations on the part of society are being envisaged. Let us imagine one possible scenario: you are a programmer in a world dominated by a future CoPilot-like system which everybody uses, and which is trained on all public code. Any new public code of yours is fed to that system, and everybody else is instantly able to use it. Since there is no attribution, your public work can no longer help you to build up reputation, community and a professional profile that would be well known outside your current company, which would make it harder to change jobs should anything go wrong. Your employer knows this, and tweaks a few HR policies.

Maybe the future CoPilot owner works out some licensing scheme which gives you some royalties when your code snippets are used? This is where the platform power comes in, and we wish we hadn’t been so enthusiastic about “fair use” for commerce. Fun fact: only 0.96% of the 7 million artists on Spotify made even $5K in 2020 (Smith, 2021). Only 0.19% (13,400 artists) out of 7 million artists were popular enough to make $50K a year.

Acknowledgements

Many thanks to amazing folks from HuggingFace for feedback & suggestions! In particular, Christopher Akiki, Gérard Dupont, Sasha Luccioni, and Aleksandra Piktus (in alphabetical order).

Updates

The text of the post was clarified thanks to feedback in the Twitter thread.

References

  1. Bakhtin, M. M. (1981). Discourse in the Novel. In M. Holquist (Ed.), The Dialogic Imagination: Four Essays. University of Texas Press. 
    @incollection{Bakhtin_1981_Discourse_in_novel,
  title = {Discourse in the Novel},
  booktitle = {The {{Dialogic}} Imagination: Four Essays},
  author = {Bakhtin, M. M.},
  editor = {Holquist, M.},
  year = {1981},
  publisher = {{University of Texas Press}},
  annotation = {Open Library ID: OL20720399M}
}
  2. Barthes, R. (1977). The Death of the Author. In S. Heath (Tran.), Image, Music, Text: Essays (Thirteenth, pp. 142–148). London: fontana. 
    @incollection{Barthes_1977_Death_of_Author,
  title = {The {{Death}} of the {{Author}}},
  booktitle = {Image, Music, Text: Essays},
  author = {Barthes, Roland},
  translator = {Heath, Stephen},
  year = {1977},
  edition = {Thirteenth},
  pages = {142--148},
  publisher = fontana,
  address = {{London}},
  url = {https://sites.tufts.edu/english292b/files/2012/01/Barthes-The-Death-of-the-Author.pdf},
  isbn = {978-0-00-686135-5},
  langid = {english}
}
  3. Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F., & Zhang, C. (2022). Quantifying Memorization Across Neural Language Models. https://doi.org/10.48550/arXiv.2202.07646 
    @misc{CarliniIppolitoEtAl_2022_Quantifying_Memorization_Across_Neural_Language_Models,
  title = {Quantifying {{Memorization Across Neural Language Models}}},
  author = {Carlini, Nicholas and Ippolito, Daphne and Jagielski, Matthew and Lee, Katherine and Tramer, Florian and Zhang, Chiyuan},
  year = {2022},
  month = feb,
  number = {arXiv:2202.07646},
  eprint = {2202.07646},
  eprinttype = {arxiv},
  primaryclass = {cs},
  publisher = arxiv,
  doi = {10.48550/arXiv.2202.07646},
  archiveprefix = {arXiv}
}
  4. Carlini, N., Tramèr, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., … Raffel, C. (2021). Extracting Training Data from Large Language Models. 30th USENIX Security Symposium (USENIX Security 21), 2633–2650. 
    @inproceedings{CarliniTramerEtAl_2021_Extracting_Training_Data_from_Large_Language_Models,
  title = {Extracting {{Training Data}} from {{Large Language Models}}},
  booktitle = {30th {{USENIX Security Symposium}} ({{USENIX Security}} 21)},
  author = {Carlini, Nicholas and Tram{\`e}r, Florian and Wallace, Eric and Jagielski, Matthew and {Herbert-Voss}, Ariel and Lee, Katherine and Roberts, Adam and Brown, Tom and Song, Dawn and Erlingsson, {\'U}lfar and Oprea, Alina and Raffel, Colin},
  year = {2021},
  pages = {2633--2650},
  url = {https://www.usenix.org/conference/usenixsecurity21/presentation/carlini-extracting},
  urldate = {2022-10-19},
  isbn = {978-1-939133-24-3},
  langid = {english}
}
  5. Heikkilä, M. (2022). This Artist Is Dominating AI-generated Art. And He’s Not Happy about It. 
    @misc{Heikkila_2022_This_artist_is_dominating_AI-generated_art_And_hes_not_happy_about_it,
  title = {This Artist Is Dominating {{AI-generated}} Art. {{And}} He's Not Happy about It.},
  author = {Heikkil{\"a}, Melissa},
  year = {2022},
  month = sep,
  journal = {MIT Technology Review},
  url = {https://www.technologyreview.com/2022/09/16/1059598/this-artist-is-dominating-ai-generated-art-and-hes-not-happy-about-it/},
  urldate = {2022-10-18},
  langid = {english}
}
  6. Hitsuwari, J., Ueda, Y., Yun, W., & Nomura, M. (2022). Does Human–AI Collaboration Lead to More Creative Art? Aesthetic Evaluation of Human-Made and AI-generated Haiku Poetry. Computers in Human Behavior, 107502. https://doi.org/10.1016/j.chb.2022.107502 
    @article{HitsuwariUedaEtAl_2022_Does_human-AI_collaboration_lead_to_more_creative_art,
  title = {Does Human\textendash{{AI}} Collaboration Lead to More Creative Art? {{Aesthetic}} Evaluation of Human-Made and {{AI-generated}} Haiku Poetry},
  shorttitle = {Does Human\textendash{{AI}} Collaboration Lead to More Creative Art?},
  author = {Hitsuwari, Jimpei and Ueda, Yoshiyuki and Yun, Woojin and Nomura, Michio},
  year = {2022},
  month = oct,
  journal = {Computers in Human Behavior},
  pages = {107502},
  issn = {0747-5632},
  doi = {10.1016/j.chb.2022.107502},
  langid = {english}
}
  7. Kristeva, J. (1980). Desire in language: A semiotic approach to literature and art. Columbia University Press. 
    @book{Kristeva_1980_Desire_in_language_semiotic_approach_to_literature_and_art,
  title = {Desire in language: A semiotic approach to literature and art},
  author = {Kristeva, Julia},
  year = {1980},
  publisher = {Columbia University Press}
}
  8. Lee, J., Le, T., Chen, J., & Lee, D. (2022). Do Language Models Plagiarize? https://doi.org/10.48550/arXiv.2203.07618 
    @misc{LeeLeEtAl_2022_Do_Language_Models_Plagiarize,
  title = {Do {{Language Models Plagiarize}}?},
  author = {Lee, Jooyoung and Le, Thai and Chen, Jinghui and Lee, Dongwon},
  year = {2022},
  month = mar,
  number = {arXiv:2203.07618},
  eprint = {2203.07618},
  eprinttype = {arxiv},
  primaryclass = {cs},
  publisher = arxiv,
  doi = {10.48550/arXiv.2203.07618},
  archiveprefix = {arXiv}
}
  9. Metzler, D., Tay, Y., Bahri, D., & Najork, M. (2021). Rethinking Search: Making Domain Experts out of Dilettantes. ACM SIGIR Forum, 55(1), 13:1–13:27. https://doi.org/10.1145/3476415.3476428
  10. Shah, C., & Bender, E. M. (2022). Situating Search. ACM SIGIR Conference on Human Information Interaction and Retrieval, 221–232. https://doi.org/10.1145/3498366.3505816
  11. Smith, D. (2021). 13,400 Artists (Out of 7 Million) Earn $50k or More From Spotify Yearly. 
    @misc{Smith_2021_13400_Artists_Out_of_7_Million_Earn_$50k_or_More_From_Spotify_Yearly,
  title = {13,400 {{Artists}} ({{Out}} of 7 {{Million}}) {{Earn}} \$50k or {{More From Spotify Yearly}}},
  author = {Smith, Dylan},
  year = {2021},
  month = mar,
  journal = {Digital Music News},
  url = {https://www.digitalmusicnews.com/2021/03/18/spotify-artist-earnings-figures/},
  urldate = {2022-10-19},
  langid = {american}
}
  12. Joseph Saveri Law Firm, & Butterick, M. (2022). GitHub Copilot Investigation. 
    @misc{JosephSaveriLawFirmButterick_2022_GitHub_Copilot_investigation,
  title = {{{GitHub Copilot}} Investigation},
  author = {{Joseph Saveri Law Firm} and Butterick, Matthew},
  year = {2022},
  url = {https://www.saverilawfirm.com/our-cases/github-copilot-intellectual-property-litigation},
  urldate = {2022-10-18}
}

Share / cite / discuss this post

]]>Anna RogersField Notes on Hybrid Conferences (EMNLP 2021)2021-11-17T12:00:47-05:002021-11-17T12:00:47-05:00https://hackingsemantics.xyz/2021/hybridThis is a quick summary of my field notes on the hybrid conferences from EMNLP2021 🌴, as an on-site attendee. I was able to attend thanks to WiNLP travel award, for their panel on the role of peer review in diversifying NLP. This was the first ever *ACL hybrid conference, and the chairs deserve applause for all their hard work.

This post is meant not as a criticism, but rather as a post-mortem that would hopefully be useful for organizers of future events. I can only share my own experience, and would love to hear from others. This post does not offer a comprehensive solution for to how to do this better - only some thoughts and comments.

Other shared impressions that I know of:

(let me know if I’m missing any other posts)

Segregation between on-site and virtual events

The fundamental issue is that the on-site conference experience is complete enough that people who are on-site have more than enough things to do without checking in on the virtual part. There were fewer people on-site than usual, but I think even 50 people would probably just keep chatting to each other full-time (as they did in the early days of ACL). Probably this time it was worse than average because this is the first on-site meeting after a year of lockdowns, and thus it was too much joy to see human faces again to exchange that for zoom. But I don’t think that this factor would ever go away.

Furthermore, if we are on-site it means we are tired from traveling and likely also jetlagged. I knew very well that virtual part was also going on, but I even missed a big chunk of the on-site program, because I just didn’t have the energy physically (and also had non-conference urgent stuff to do for ARR). The result was that I made it to the grand total of 1 virtual poster in the whole week.

A part of the problem is that our conferences are just generally too big. In a regular pre-pandemic on-site conference there were already too many parallel sessions going on, and thus it was already hard enough to pick and choose, and get to the right rooms at the right time. If the hybrid format offers the on-site attendees a subset of that program that is “live”, and the rest of the talks are recorded anyway, I think simply following the on-site part will always be a too-tempting option. If some of the parallel sessions are virtual, the topical division imo wouldn’t offer sufficient incentive to attend them just for their topic, because most of us seem to have many research interests, and will likely always find some exciting work that happens to be presented on-site.

This is why parallel on-site / virtual sessions are problematic. Unfortunately, they are also problematic when they are consecutive, because of the limited working hours in the on-site location. In day 1 there was a 9am invited talk, 4 oral sessions until 18:15, 45 min for dinner break, and then another 2-hour virtual poster session 19:00-21:00. Speaking for myself, I just cannot absorb this much of a conference in a day. I do see that evening/night virtual events make sense to accommodate the presenters who are inevitably going to be in some other time zone than the conference location, but this also inevitably contributes to the segregation.

And this segregation is obviously not great. The people who can afford to travel to conferences are the ones who have money, visas, health, AND time - and EACH of these criteria cuts off a LOT of people. Money-wise, the student & diversity travel awards are better than nothing, but not a solution, as there will never be enough awards for all who deserves to come. Personally, I had zero travel support during my PhD, and without a visit to NAACL sponsored by ACL student research workshop I would probably not be here today. I am acutely aware that my getting that funding likely meant that someone else didn’t, and maybe they were/would be a better researcher.

For all of these reasons, not to mention the environment, I have previously advocated for fully switching to virtual conferences. I do see that this would be taking a lot of joy out of science (for people with money, visas, health and time), and so there is little appetite for such extreme measures - especially in that demographic. I am also aware that students need to network with that demographic to find jobs, and those students who do get to conferences get an important competitive advantage, which they will not want to lose. But if the community decides that these two factors outweigh inclusivity and the solution is hybrid conferences - I really don’t think we have a recipe that works for both sides yet.

Technical aspects of organization: notes on different types of events from an on-site attendee perspective

Keynotes & invited talks

Conference keynotes work great in the hybrid format, because only one thing is going on, it’s generally well attended, and everybody is there from both channels.

The same was generally true of invited talks for workshops, even though there were several workshops in parallel, and so more competition for the audience.

On-site talks

In the on-site talks the presenter being virtual or on-site didn’t make a lot of difference for me. But with only 5 minutes for questions, having to locate & open the chat on-site often seemed like too much effort. So I ended up mostly not doing it, unless I already had laptop open.

Questions to papers work better as asynchronous chat, but it’d be nice to have some dedicated slots in the conference program to do that. And authors should get notified when there are questions for their papers. I had 2 papers, jetlag, and a workshop to organize, and so definitely did not have the presence of mind to keep checking on those chats.

Even without the hybrid thing, I think poster format for conferences is just inherently better than 15 min talks, and I heard many people say the same. Way more interactivity, can go in-depth and/or chat & brainstorm as needed.

Posters

Hybrid poster sessions are a challenge. To have virtual attendees in the live poster sessions we’d have to have some kind of conference robots, which is just too expensive. Having separate on-site & virtual sessions deprives the virtual crowd of the former. If we have the on-site presenters also present virtually a second time, they get double exposure, which seems unfair to virtual-only participants.

Shall we just give up on physical posters and switch to gather-town in perpetuity (provided that its infrastructure scales to be fast enough?) Yes, it’s amazing to be able to talk to people live, but see above: it doesn’t seem to be possible to do it in an inclusive way. Plus we have the lovely task, cost and eco footprint of printing & carrying those posters. In EMNLP 2019 I nearly lost my poster in Hong Kong airport, because I was so jetlagged after the flight!

While live interaction with people at the poster feels better (if you’re among the lucky ones to be on-site), I do think it is strictly inferior to gather-town in terms of interaction with the content: it’s a lot easier to take notes, check up any papers mentioned in the discussion, tweet interesting stuff, look up people you ‘run’ into. Have you ever come back from an on-site poster session with a phone full of poster photos that you never touched since? I certainly have.

Panels

I was an on-site panelist with two virtual panelists at WiNLP. The panel was great, but it presented a challenge I’ve never thought about: camera positioning. The room had the standard setup of a large screen with the projected speaker view for the online participants, and in front of that screen was a chair for the on-site speaker facing the on-site audience.

Since the screen projecting the speaker view was behind me, I couldn’t see who I was talking to, and so had to have a laptop with zoom on a table in front of me. The end result was that the on-site people saw me stare at laptop in front of them, and virtual people saw a side view of me staring at the laptop. I honestly don’t know how this could be resolved. I hope this gets read by an academic whose hobby happens to be videography.

Underline grievances

Separate from the hybrid format is the heap of trouble with Underline.io, which EMNLP used for the hybrid part. This time the virtual attendance cost more, but the platform experience did not improve to justify that. As in ACL and NAACL 2021, it was slow, and the linking between papers, videos, live zooms and associated chats added a ton of friction. A one-click feature “add this to my schedule in my time zone” should NOT be so hard. On-site, we had to look for things both on underline, in whova, and in the printed handbook, as they sometimes had different information.

I did not expect that in the closing remarks the speakers had to say “next” for someone to press the button to advance the slides. Definitely didn’t seem like we’re reaching for AGI yet…

But this was the main conference. It was a lot worse for the workshops and tutorials, which did not even have any schedules on the platform - except for their own websites, which would be harder for the attendees to cross-compare and make a composite schedule of.

The on-site problems were so numerous that it’d be funny if it was not borderline disastrous. I heard that the crowdwourcing tutorial was assigned to an empty room without the set-up gear, and they had to run between rooms!

In the Insights from Negative Results workshop, we started by losing 20 scheduled minutes because they gave us the same zoom link as another workshop. Then in another talk our on-site mike died and we couldn’t get through to the speaker who went overtime. And then they re-logged in for some reason, and that kicked the organizers out of the zoom altogether. For dessert, my own pre-submitted poster was simply missing in gather.town. They don’t even let people know if there are any problems with uploaded pdfs or videos. The underline team was on-site in sufficient numbers, but I couldn’t help wishing that I did not have to fetch them all the time.

I was part of the team for EMNLP 2020, which seems to have so far have delivered the best virtual conference experience with a combination of miniconf, gather.town and rocketchat. This was a ton of work, and I totally see why subsequent conferences went with underline because it just seems like a one-stop infrastructure solution. But underline truly makes the virtual part way worse than it has to be, and they clearly haven’t adapted their platform based on everything that was said after NAACL and ACL. I doubt they will now. Given that this is a computer science-ish field with tons of money, do we really have to inflict this on ourselves?

Location

A completely orthogonal dimension to the hybrid format and Underline is the location. Which in this case was Punta Cana, Dominican republic. Which looks like this:

Don’t get me wrong: the Caribbean is magical. I would probably never have made it without this conference, I’ll remember it forever, and I’m happy I was able to go. I wish everybody else could have seen the palm trees, the sunrise on the beach, the parrots, and everything else.

At the same time, if I were to design a special hell for an academic - I’d give them a limited time in a tropical beach with turtles to snorkel with, a buffet with infinite supply of mango smoothies… and a deadline to watch a bunch of talks, or write a grant application, or something like that. In this setup you get tortured by FOMO no matter what you choose to do.

You can also try to compromise, which will probably result in doing a bad job of both options. I honestly tried the student solution of just not sleeping. I lived on 6 hours of sleep for a week to go for a swim before the conference - and I normally need 8-9 hours. Result: still foggy and exhausted, 3 days after I’m back. I’m sure the quality of my thinking suffered, and I rambled incoherently to people who deserved better. It also feels profoundly wrong to be at a resort, where everybody relaxes, but you are running between things like it’s the start of the term.

Kudos for an ingenious solution to Marzena Karpinska, who risked her phone and headphones and literally watched the crowdsourcing tutorial in the pool. Wish someone made waterproof laptops.

So… if we do have any more conferences in tropical resorts, I’d suggest to make them at least 2 weeks long, with half a day dedicated to snorkeling, birding, kayaking and everything else that has blissfully nothing to do with research, but is a sin to miss. I can even nominate special chairs for all that!

Another thing I didn’t expect, but totally should have: there were almost no power outlets, and wifi was patchy and unreliable (probably depending on how many tourists were streaming movies at a given time). Kind of duh, this place is emphatically not meant for work!

Final thoughts

Once again: EMNLP 2021 was certainly unforgettable, and I’m very happy and priviledged to be able to attend it. And the organizers put an insane amount of volunteer work into getting the first hybrid conference to run as smoothly as possible. There is certainly a lot of valuable experience here for future events, as well as plenty of food for thought.

]]>Anna Rogers