Field Notes on Hybrid Conferences (EMNLP 2021)
This is a quick summary of my field notes on the hybrid conferences from EMNLP2021 š“, as an on-site attendee. I was able to attend thanks to WiNLP travel award, for their panel on the role of peer review in diversifying NLP. This was the first ever *ACL hybrid conference, and the chairs deserve applause for all their hard work.
This post is meant not as a criticism, but rather as a post-mortem that would hopefully be useful for organizers of future events. I can only share my own experience, and would love to hear from others. This post does not offer a comprehensive solution for to how to do this better - only some thoughts and comments.
Other shared impressions that I know of:
- Jordan Boyd-Graber (as a virtual attendee): Video, Text
- Sam Bowman (as an on-site attendee): Twitter thread
(let me know if Iām missing any other posts)
Segregation between on-site and virtual events
The fundamental issue is that the on-site conference experience is complete enough that people who are on-site have more than enough things to do without checking in on the virtual part. There were fewer people on-site than usual, but I think even 50 people would probably just keep chatting to each other full-time (as they did in the early days of ACL). Probably this time it was worse than average because this is the first on-site meeting after a year of lockdowns, and thus it was too much joy to see human faces again to exchange that for zoom. But I donāt think that this factor would ever go away.
Furthermore, if we are on-site it means we are tired from traveling and likely also jetlagged. I knew very well that virtual part was also going on, but I even missed a big chunk of the on-site program, because I just didnāt have the energy physically (and also had non-conference urgent stuff to do for ARR). The result was that I made it to the grand total of 1 virtual poster in the whole week.
A part of the problem is that our conferences are just generally too big. In a regular pre-pandemic on-site conference there were already too many parallel sessions going on, and thus it was already hard enough to pick and choose, and get to the right rooms at the right time. If the hybrid format offers the on-site attendees a subset of that program that is āliveā, and the rest of the talks are recorded anyway, I think simply following the on-site part will always be a too-tempting option. If some of the parallel sessions are virtual, the topical division imo wouldnāt offer sufficient incentive to attend them just for their topic, because most of us seem to have many research interests, and will likely always find some exciting work that happens to be presented on-site.
This is why parallel on-site / virtual sessions are problematic. Unfortunately, they are also problematic when they are consecutive, because of the limited working hours in the on-site location. In day 1 there was a 9am invited talk, 4 oral sessions until 18:15, 45 min for dinner break, and then another 2-hour virtual poster session 19:00-21:00. Speaking for myself, I just cannot absorb this much of a conference in a day. I do see that evening/night virtual events make sense to accommodate the presenters who are inevitably going to be in some other time zone than the conference location, but this also inevitably contributes to the segregation.
And this segregation is obviously not great. The people who can afford to travel to conferences are the ones who have money, visas, health, AND time - and EACH of these criteria cuts off a LOT of people. Money-wise, the student & diversity travel awards are better than nothing, but not a solution, as there will never be enough awards for all who deserves to come. Personally, I had zero travel support during my PhD, and without a visit to NAACL sponsored by ACL student research workshop I would probably not be here today. I am acutely aware that my getting that funding likely meant that someone else didnāt, and maybe they were/would be a better researcher.
For all of these reasons, not to mention the environment, I have previously advocated for fully switching to virtual conferences. I do see that this would be taking a lot of joy out of science (for people with money, visas, health and time), and so there is little appetite for such extreme measures - especially in that demographic. I am also aware that students need to network with that demographic to find jobs, and those students who do get to conferences get an important competitive advantage, which they will not want to lose. But if the community decides that these two factors outweigh inclusivity and the solution is hybrid conferences - I really donāt think we have a recipe that works for both sides yet.
Technical aspects of organization: notes on different types of events from an on-site attendee perspective
Keynotes & invited talks
Conference keynotes work great in the hybrid format, because only one thing is going on, itās generally well attended, and everybody is there from both channels.
The same was generally true of invited talks for workshops, even though there were several workshops in parallel, and so more competition for the audience.
On-site talks
In the on-site talks the presenter being virtual or on-site didnāt make a lot of difference for me. But with only 5 minutes for questions, having to locate & open the chat on-site often seemed like too much effort. So I ended up mostly not doing it, unless I already had laptop open.
Questions to papers work better as asynchronous chat, but itād be nice to have some dedicated slots in the conference program to do that. And authors should get notified when there are questions for their papers. I had 2 papers, jetlag, and a workshop to organize, and so definitely did not have the presence of mind to keep checking on those chats.
Even without the hybrid thing, I think poster format for conferences is just inherently better than 15 min talks, and I heard many people say the same. Way more interactivity, can go in-depth and/or chat & brainstorm as needed.
Posters
Hybrid poster sessions are a challenge. To have virtual attendees in the live poster sessions weād have to have some kind of conference robots, which is just too expensive. Having separate on-site & virtual sessions deprives the virtual crowd of the former. If we have the on-site presenters also present virtually a second time, they get double exposure, which seems unfair to virtual-only participants.
Shall we just give up on physical posters and switch to gather-town in perpetuity (provided that its infrastructure scales to be fast enough?) Yes, itās amazing to be able to talk to people live, but see above: it doesnāt seem to be possible to do it in an inclusive way. Plus we have the lovely task, cost and eco footprint of printing & carrying those posters. In EMNLP 2019 I nearly lost my poster in Hong Kong airport, because I was so jetlagged after the flight!
While live interaction with people at the poster feels better (if youāre among the lucky ones to be on-site), I do think it is strictly inferior to gather-town in terms of interaction with the content: itās a lot easier to take notes, check up any papers mentioned in the discussion, tweet interesting stuff, look up people you ārunā into. Have you ever come back from an on-site poster session with a phone full of poster photos that you never touched since? I certainly have.
Panels
I was an on-site panelist with two virtual panelists at WiNLP. The panel was great, but it presented a challenge Iāve never thought about: camera positioning. The room had the standard setup of a large screen with the projected speaker view for the online participants, and in front of that screen was a chair for the on-site speaker facing the on-site audience.
Since the screen projecting the speaker view was behind me, I couldnāt see who I was talking to, and so had to have a laptop with zoom on a table in front of me. The end result was that the on-site people saw me stare at laptop in front of them, and virtual people saw a side view of me staring at the laptop. I honestly donāt know how this could be resolved. I hope this gets read by an academic whose hobby happens to be videography.
Underline grievances
Separate from the hybrid format is the heap of trouble with Underline.io, which EMNLP used for the hybrid part. This time the virtual attendance cost more, but the platform experience did not improve to justify that. As in ACL and NAACL 2021, it was slow, and the linking between papers, videos, live zooms and associated chats added a ton of friction. A one-click feature āadd this to my schedule in my time zoneā should NOT be so hard. On-site, we had to look for things both on underline, in whova, and in the printed handbook, as they sometimes had different information.
I did not expect that in the closing remarks the speakers had to say ānextā for someone to press the button to advance the slides. Definitely didnāt seem like weāre reaching for AGI yetā¦
But this was the main conference. It was a lot worse for the workshops and tutorials, which did not even have any schedules on the platform - except for their own websites, which would be harder for the attendees to cross-compare and make a composite schedule of.
The on-site problems were so numerous that itād be funny if it was not borderline disastrous. I heard that the crowdwourcing tutorial was assigned to an empty room without the set-up gear, and they had to run between rooms!
In the Insights from Negative Results workshop, we started by losing 20 scheduled minutes because they gave us the same zoom link as another workshop. Then in another talk our on-site mike died and we couldnāt get through to the speaker who went overtime. And then they re-logged in for some reason, and that kicked the organizers out of the zoom altogether. For dessert, my own pre-submitted poster was simply missing in gather.town. They donāt even let people know if there are any problems with uploaded pdfs or videos. The underline team was on-site in sufficient numbers, but I couldnāt help wishing that I did not have to fetch them all the time.
I was part of the team for EMNLP 2020, which seems to have so far have delivered the best virtual conference experience with a combination of miniconf, gather.town and rocketchat. This was a ton of work, and I totally see why subsequent conferences went with underline because it just seems like a one-stop infrastructure solution. But underline truly makes the virtual part way worse than it has to be, and they clearly havenāt adapted their platform based on everything that was said after NAACL and ACL. I doubt they will now. Given that this is a computer science-ish field with tons of money, do we really have to inflict this on ourselves?
Location
A completely orthogonal dimension to the hybrid format and Underline is the location. Which in this case was Punta Cana, Dominican republic. Which looks like this:
Donāt get me wrong: the Caribbean is magical. I would probably never have made it without this conference, Iāll remember it forever, and Iām happy I was able to go. I wish everybody else could have seen the palm trees, the sunrise on the beach, the parrots, and everything else.
At the same time, if I were to design a special hell for an academic - Iād give them a limited time in a tropical beach with turtles to snorkel with, a buffet with infinite supply of mango smoothiesā¦ and a deadline to watch a bunch of talks, or write a grant application, or something like that. In this setup you get tortured by FOMO no matter what you choose to do.
You can also try to compromise, which will probably result in doing a bad job of both options. I honestly tried the student solution of just not sleeping. I lived on 6 hours of sleep for a week to go for a swim before the conference - and I normally need 8-9 hours. Result: still foggy and exhausted, 3 days after Iām back. Iām sure the quality of my thinking suffered, and I rambled incoherently to people who deserved better. It also feels profoundly wrong to be at a resort, where everybody relaxes, but you are running between things like itās the start of the term.
Kudos for an ingenious solution to Marzena Karpinska, who risked her phone and headphones and literally watched the crowdsourcing tutorial in the pool. Wish someone made waterproof laptops.
Soā¦ if we do have any more conferences in tropical resorts, Iād suggest to make them at least 2 weeks long, with half a day dedicated to snorkeling, birding, kayaking and everything else that has blissfully nothing to do with research, but is a sin to miss. I can even nominate special chairs for all that!
Another thing I didnāt expect, but totally should have: there were almost no power outlets, and wifi was patchy and unreliable (probably depending on how many tourists were streaming movies at a given time). Kind of duh, this place is emphatically not meant for work!
Final thoughts
Once again: EMNLP 2021 was certainly unforgettable, and Iām very happy and priviledged to be able to attend it. And the organizers put an insane amount of volunteer work into getting the first hybrid conference to run as smoothly as possible. There is certainly a lot of valuable experience here for future events, as well as plenty of food for thought.