This document aims to help authors, reviewers, and technical papers program chairs understand the new double-blind review process that SC16 is adopting. Please direct your questions and comments to papers@info.supercomputing.org.

Rationale

Technical papers submitted to SC16, SC17, and SC18 will undergo a double-blind review process. In this process, authors do not know who reviews their papers and reviewers do not see author names.

The primary rationale for double-blind review is to mitigate implicit or explicit bias, as there is ample evidence that double-blind policies can reduce such bias. There is also evidence that papers subjected to double-blind review receive more citations than those that undergo single-blind review; thus double-blind review may also be associated with higher quality papers.

Please see the references at the end of this document for additional discussions and editorials on double-blind review. Those of McKinley (2008; updated in 2015) and Snodgrass (2007) are particularly lucid. From the former comes the helpful reminder that “Double-blind is not perfect, just better;” and from the latter that, “… uncertainty as to the authors’ identity is often sufficient to realize most or all of the benefits of masking.” In other words, the goal is not “perfect” blinding but rather enough uncertainty about authorship in the process to achieve a goal of reducing actual or perceived bias.

Guidance to Authors

If you are an author, you should write your paper so as not to disclose your identity or the identities of your co-authors. The following guidelines are best practices for “blinding” a submission in a way that should not weaken it or the presentation of its ideas. These guidelines are broken up into the major submission and review phases: while writing (before submitting), at submission time, and during the rebuttal process.

These practices were distilled from McKinley (2015) and Snodgrass (2007).

If you have additional questions or comments, please send them to the SC Technical Papers Chairs at papers@info.supercomputing.org.

While Writing

• Do not use your name or your co-authors’ names, affiliations, funding sources, or acknowledgments in the heading or body of the document.
• Do not eliminate self-references to your published work that are relevant and essential to a proper review of your paper solely in an attempt to anonymize your submission. Instead, write self-references in the third person. Recall that the goal and spirit of double-blind review is to create uncertainty about authorship, which is sufficient to realize most of its benefits.
• To reference your unpublished work, use anonymous citations. From Snodgrass (2007): “The authors developed … [1]” where the reference [1] appears as, “[1] Anonymous (omitted due to double-blind review).” You will have a way to explain these references to the non-conflicted Technical Papers Chairs or their designee(s); see “At Submission Time” below.

At Submission Time

• At submission time, you will be asked to declare conflicts of interest you may have with program committee members. You will also have the option to upload a list of conflicts. Reviewers will be asked separately to verify declared conflicts.
• Suppose you feel that there is supplemental material essential to reviewing your submission but which would also reveal your identity, e.g., an earlier technical report, published software. In this case, you will have the option to upload or link to those materials and include an explanation at the time of your submission. By default, reviewers will not see this material; instead, the non-conflicted Technical Papers Chairs or their designee(s) will review and use their discretion to decide whether to include such materials during the review.
• Because of the double-blind process, there will no longer be a limit on the number of submissions by Reviewers. (Area Chairs are subject to limits.) However, there will be a limit of four (4) on the number of accepted papers for Reviewers.

During the Reviewing Period

• You are not forbidden from disseminating your work via talks or technical reports. However, you should not try to directly or otherwise unduly influence program committee members who may be reviewing your paper.

During the Rebuttal Period

• During the rebuttal period, authors should still assume double-blind review. Therefore, authors should not disclose their identities in their rebuttal to the reviewers. However, as with the original submission, authors will have the option of entering identity-revealing information in a separate part of the rebuttal form that will, by default, be visible only to a non-conflicted Chairs, or their designee(s) in the case of conflicts.

Guidance to Technical Papers Chairs and Reviewers

If you are a Technical Papers Chair or Area Chair (hereafter, “Chair”) or if you are a Reviewer (i.e., Technical Papers Committee member), the following is a set of guidelines you should follow. Generally speaking, the procedures draw inspiration from the three principles Snodgrass suggests in his 2007 editorial on ACM SIGMOD’s move to double-blind reviews:

“The first is that authors should not be required to go to great lengths to blind their submissions. The second is that comprehensiveness of the review trumps blinding efficacy. The final principle … is that [editors and chairs] retain flexibility and authority in managing the reviewing process.”

Before the Submission Deadline

• Correctly identifying conflicts of interest (COIs) is one of the most important procedural aspects of double-blind review. Therefore, before the paper submission deadline, Chairs and Reviewers should log into the review system, Linklings, to verify and upload their conflicts of interest. [http://submissions.supercomputing.org] This process can be a little time-consuming, so please plan accordingly.
• During paper bidding, Reviewers should let their Chair know if they suspect a conflict with a submission and what they believe is the nature of the conflict.

During the Reviewing Period

• A Reviewer may accidentally discover the identities of the authors during the review. (For instance, he or she might be checking references to determine the novelty of the submission and discover a technical report with the same content.) In this case, the Reviewer should disclose this discovery to a Chair. Such incidents do not necessarily “violate” the double-blind policy, and the Reviewer may continue to review the paper. The spirit of double-blind reviewing is that Reviewers should not actively try to discover who the authors of a submission are.
• A Reviewer who thinks he or she knows the identity of the authors should not reveal his or her suspicion in his or her review or during discussions with other reviewers (whether online or in-person).
• SC16 is adopting the “Double-Blind until Accept” procedure. That is, author identities will remain hidden until the program committee has determined all of the accepted papers. For rejected papers, author identities remain hidden even after rejection.
• Reviewers who feel that knowing the author names or affiliations is necessary to review a submission can make their case to a Chair at any time during the review process.
• Reviewers who wish to ask colleagues to help with reviews must clear these requests with a Chair first and take steps to ensure that the colleague understands the double-blind policy. In any case, per prior SC reviewing guidelines, a Reviewer is responsible for representing his or her reviews fully.

During the Program Committee Meeting

• Chairs should still observe and manage conflicts as they would in a single-blind review. For instance, they should avoid a discussion of a paper until all of the paper’s conflicted reviewers have left the room.

Conflicts of Interest

The following language also appears in the SC16 Frequently Asked Questions document:

http://sc16.supercomputing.org/submitters/technical-papers/papers-faq/

A potential conflict of interest (COI) occurs when a person makes a decision that

1. could result in that person, a close associate of that person, or that person’s company or institution receiving significant financial gain, such as a contract or grant; or
2. could result in that person, or a close associate of that person, receiving significant professional recognition, such as an award or the selection of a paper, work, exhibit, or another type of submitted presentation.

Technical Papers Committee will have a chance to disclose potential conflicts during the review process. Chairs will make every effort to avoid assignments that have a potential COI.

For SC16, you have a COI with

• your Ph.D. advisors, postdoctoral advisors, Ph.D. students, and postdoctoral advisees forever;
• your family relations by blood or marriage, or equivalent (e.g., a partner), forever;
• people with whom you collaborated in the past five years, including:
  • co-authors on an accepted/rejected/pending research paper;
  • co-PIs on an accepted or pending grant;
  • those who fund your research, researchers whom you fund, or researchers with whom you are actively collaborating;
  • people who were employees or students at your primary institution(s) in the past five years, or people who are active candidates for employment at your institution(s); and
  • close personal friends or others with whom you believe a COI exists.

Note that “service” collaborations, such as writing a DOE, NSF, or DARPA report, or serving on a program committee, do not inherently create a COI.

Other situations can create COIs, and you should contact the Technical Papers Chairs for questions or clarification on any of these issues.

Frequently Asked Questions

Q: How strictly will the new double-blind policy be enforced?

A: Authors should consider the guidance above as a set of guidelines and not necessarily as a set of draconian rules the violation of which results in immediate rejection. The Technical Papers Chairs and their Area Chairs will have enough discretion to apply the double-blind policies in a reasonable way.

Q: My work builds on a system that I previously developed. How can I cite this work?

A: Follow the guidelines on the use of third-person self-references as described in the subsection, “While Writing.” Following the example of Snodgrass (2007), it would not be untrue even to write in the text of your paper that the developers of the system upon which you are building made this system available to you. Recall the principle is to introduce uncertainty about your identity.

Q: I strongly object to double-blind review. What should I do?

A: Read the references below. If you remain unconvinced, you may vent at papers@info.supercomputing.org with the subject line tag, “#ConscientiousObjector”.

Q: Doesn’t double-blind hurt prolific or famous authors?

A: The empirical evidence that double-blind affects the acceptances of such authors is inconclusive; see the references below for details.

References

• SC16 Frequently Asked Questions. http://sc16.supercomputing.org/submitters/technical-papers/papers-faq/
• S. McKinley (2015). “More on Improving Reviewing Quality with Double-Blind Reviewing, External Review Committees, Author Response, and in Person Program Committee Meetings.” http://www.cs.utexas.edu/users/mckinley/notes/blind-revised-2015.html
• T. Snodgrass (2007). “Editorial: Single- versus double-blind reviewing.”http://history.acm.org/pictures/tods/tods.acm.org/editorial.pdf
• N. Laband and M. J. Piette (1994). “A citation analysis of blinded peer review.” The Journal of the American Medical Association (JAMA): The Second International Congress on Peer Review in Biomedical Publication, 272(2):147–149.http://www.ncbi.nlm.nih.gov/pubmed/8015128

Acknowledgements. Rich Vuduc and Lois Curfman McInnes prepared the initial version of this document at the request of the SC Steering Committee. They thank John West, Lori Diachin, Jeffrey Vetter, Mattan Erez, Michela Taufer, Rajeev Thakur, Wu Feng, and D.K. Panda, and the SC16 Area Chairs, among others, for their helpful comments, feedback, and pointers.