I am joining ACL Rolling Review

It’s official: I joined the ACL Rolling Review team as an editor-in-chief, and I’d like to share some brief thoughts on this.

When ACL Rolling Review was first launched, I wasn’t its biggest fan. The core motivation seemed to be that it would reduce the reviewer workload, and I am not convinced that that this goal is either achievable with ARR, or that it has in fact been achieved.

However, from bitter experience as a program chair of ACL’23, I am firmly convinced that we as a scientific community need a centralized and continually improving conference review system. If you have not been in that role, trust me: there are hundreds of things that you as a chair have to (a) know about, (b) remember about, (c) care enough about when it’s 2am, (d) know how to do well, (e) have support for in the system you’re using, (f) potentially argue with other chairs and the whole community about. This is why conference peer review is just way too complex to have a new set of chairs do it for the first time for every single conference - and too many people waste time and effort when anything goes wrong.

ARR is by no means perfect. I hope to help with fixing at least some of the issues while I’m there, but I know that even after I and everybody else do everything we can - it still won’t be perfect. Still, it’ll be better, and we as a community will have an iteratively improving system that gradually accumulates documentation, software support, and people familiar with it. Just this, by itself, is a huge step in the right direction. The service time that people invest in this is a precious resource, and it should be reused and iterated on as much as possible.

Here are some of the things that I am hoping to help the team to get done:

• updating the Responsible NLP checklist (some questions are up for rethinking), and the way it is used
• updating the ARR reviewer guidelines, in particular with respect to generative AI (in consultation with the new ACL committee on publication ethics)
• implementing the structured author complaints to chairs, which were very well-received at ACL’23, and which would allow the authors to flag reviews for specific types of issues (see sec.5.3 of ACL’23 report)
• figuring out ways to support the chairs in implementing the new ACL anonymity policy (according to which borderline anonymized papers have an advantage over preprinted papers)
• analyzing the effects of the change in anonymity policy