Jeff’s Rebuttal Guide
Jeff’s Rebuttal Guide
@jeffbigham
11/21/2019
It’s rebuttal day, it’s rebuttal day! Some people hate rebuttals; I really enjoy rebuttals. I enjoy them in the way that I enjoy being able to cut off half a page from a paper written in Latex without removing any content to fit within the page limit. Yes, that much!
Rebuttals are weird. They are one-sided conversations. They favor the “in” group who has experience writing them.
People in the community have written some guides to writing rebuttals. I have seen a lot of people sharing Merrie Morris’s rebuttal writing guide, which is great. I’ll try not to be redundant.
This post contains three sections -- (i) a rebuttal template that I loosely follow, (ii) a set of small tips that I’ve accumulated while helping with rebuttals this time around, and (iii) three examples of rebuttals that I wrote that actually worked! (2 of which on pretty low-scoring papers after the first round).
Before actually getting to the template, it’s worth thinking about the rebuttal as something you are designing for your reviewers. You want them to easily see how you have addressed their problems, you want to convince them that you’ll be able to effectively address their concerns, and you want to remind them about why they should care about your paper.
<thanks reviewers!>
<quick summary of contribution, your most compelling one or two sentences>
<possibly supported by reviewer comments -- e.g., “R1 said this is the best paper ever”>
<address major points, as part of a coherent story, esp put anything the ACs called out here… cite Rs who mentioned it (R1,R2)>
<address minor points, as many (all) that you can fit in the character limit>
- be polite but firm!
- short sentences, every word counts
- first draft should address every single point, revise and revise, work toward the character limit without removing content… if you’ve been tweeting endless for years like me, as i advise everyone to do, then you’ve learned how to work within tight character limits
- cite reviewers in your points to make it easy for them to find where you responded (R3)
* reviewers who made some point that is important to them, which they think you didn’t address, will not want to accept your paper
- accepting limitations can establish credibility, don’t just refute everything
- frame things positively!
* revisions will make the paper better, not correct some terrible flaw
* try especially to end paragraphs, and the whole rebuttal on a positive note
- don’t say you “will edit the paper” when you can say, “we have edited the paper”
* e.g., “We edited Figure 2 to include a new column that separates out Word Error Rate”
* e.g., include specific details, e.g., “the effect size was 30.2” not “we will include the effect size”
* e.g., if you say you’re going to add 3 paragraphs about X, be specific about how you’ll make room for those three paragraphs
- if people questioned the “type” of contribution you made, refer what you did back to Jake Wobbrock’s “Contributions in HCI” and describe how you fulfilled what is written there about how each is validated
- in rebuttals, you can bring in more context
* one common criticism we get in papers is related to whether or not we engaged with our user group sufficiently (you can see it below in our VizWiz rebuttal) -- i have generally used the rebuttal to relay true experiences about our substantial engagement with people with disabilities that establishes credibility but doesn’t necessarily fit well in a research paper
- throw a “Hail Mary” (mostly for when your paper is basically doomed…mostly kidding?)
* find a detail the reviewers got wrong and get a tiny bit indignant about it
* cite one of Landay’s rants, especially one of the old ones, “THIS IS EXACTLY WHAT LANDAY WAS TALKING ABOUT!”
* take some popular fad critique of CHI, twist it around so your work is the exemplar of all that this critique believes is wrong with CHI… probably someone on your subcommittee is a champion of that critique and will listen in if they hear it mentioned… for this tip, it helps to be on Twitter & CHI Meta *all* *the* *time* so you know what is popular this year… for a while you got a lot of credit for appealing to “systems work is underrated”, then it was “notes are judged so much more harshly than full papers”, then it was “replication is good!”, then it was “qualitative work is cool too!”… idk what this year’s winning thing is!
- if someone tells you your paper is better suited for ASSETS than CHI, as people told me for like years (until recently), flip the f’ out. Warning: this does not help you get your paper accepted
I’ve written a lot of rebuttals over the years. The usual outcome is that the scores don’t change very much. Sometimes they change a little. These three rebuttals I think are pretty reasonable, and each led to some pretty big swings. I did not include the actual reviews from these papers, so they’re mostly just useful to get a sense for overall style.
Making the Web Easier to See with Opportunistic Accessibility Improvement, UIST 2014. (average score= ~2.4)
Crowd Storage, CHI 2014. (average score = ~2.5)
VizWiz: Nearly Real-time Answers to Visual Questions, UIST 2010. (average score = ~3.5, but it got Best Paper eventually)
I am more and more convinced that we should do away with the conference model, and thus rebuttals. Journals like IMWUT and TOCHI have convinced me that a journal review process can work as effectively and as expeditiously. In that model, you don’t have to have a completely one-sided conversation. You can return to the reviewers a draft prepared to address their comments. You return with a response not bounded in length, daring reviewers to reject your work despite the fact that you addressed every point that they made.
Until all of our venues move to that model, I hope this helps folks newer to the process write more successful rebuttals!
This page and contents are copyright Jeffrey P. Bigham except where noted.
Blog posts are not intended to be final products, but rather a reflection of current thinking and/or catalysts for discussion, like tweets but longer.
