CHI '97 Design Briefings:
Overview and Guidelines for Reviewers and Associate Chairs
Introduction
Hello,
Thank you for joining the CHI '97 Design Briefings committee. As a reviewer
or an Associate Chair (also known as a meta-reviewer), you play an essential
role in ensuring the high quality of design briefings. This note provides
an overview of what will happen during the review process, and includes
the guidelines for reviewers and Associate Chairs. Please look over the
appropriate sections; you may also wish to look at the review
form or meta-review form. Contact
us at chi97-design-br@acm.org
if you have any questions.
Tom Erickson,
Discourse Architecture Laboratory,
Apple Research Labs,
Apple Computer, 301-4UE
E-Mail: thomas@apple.com
Cupertino office: voice: (408) 974-3767; fax: (408) 974-5505
Minneapolis office: voice: (612) 823-3663; fax: (612) 823-1576
or
Ian McClelland
Manager Applied Ergonomics
Philips Corporate Design
Building OAN, PO Box 218,
5600 MD, Eindhoven,
Netherlands
E-Mail: C834997@NLccMAIL.snads.ph
ilips.nl
Tel: +31 40 2733311
Fax: +31 40 2734959
Thanks for your help,
Tom Erickson and Ian McClelland
CHI 97 Design Briefing Co-Chairs
Overview of the Process
1. THE CHI 97 DESIGN BRIEFINGS REVIEW PROCESS
The design briefings review process is changing a bit, to bring it more
into line with the review process for papers and other archival materials.
Here's a brief sketch of the process so you understand where you fit in:
1.1. SUBMISSION AND DISTRIBUTION
Papers are submitted to the Co-Chair by September 20th, as described in
the call for participation. Each paper is mailed to five reviewers and one
Associate Chair responsible for managing its review. (Each reviewer will
receive approximately five papers; each Associate Chair will be responsible
for 8 to 12 papers.)
1.2. REVIEWS
The completed reviews are due by 5pm (17:00), Wednesday, October 17th. They
should be emailed to the Associate Chair responsible for that paper (and
copied to the Design Briefing co-chairs). See the guidelines below.
1.3. META-REVIEWS
The Associate Chair uses the reviews to produce a meta-review that summarizes
the individual responses, and provides a single set of ratings for the paper.
This may involve asking the reviewers for more information, seeking outside
reviews in the case of controversial papers, and trying to compensate for
variations in expertise and perspective among the reviewers. The completed
meta-reviews are due by 5pm (17:00), Wednesday, October 30, and should be
emailed to the co-chairs.
1.4. FINAL DECISIONS AND AFTERWARDS
The meta-reviews, along with the reviews, are used to determine the final
list of accepted submissions at the CHI 97 Technical Program meeting. In
difficult or unusual cases, the Associate Chair for a design briefing may
be consulted via phone or email. Once the accept/reject decision is made,
reviews and meta-reviews will be passed on to the design briefing authors.
Reviewer Guidelines
2. CHI 97 DESIGN BRIEFINGS REVIEWER GUIDELINES
2.1 BE POLITE AND TACTFUL
Authors put a great deal of effort into their work. Please respect this
by wording your comments carefully and seriously. In the context of the
rejection of a submission, an innocently intended offhand remark or joke
can hurt or offend, so an extra five minutes doing a "tact check"
of your review is well spent. This does *not* mean, however, that you should
avoid criticizing or pointing out weaknesses in a submission--simply express
your criticisms tactfully and professionally.
2.2 BE CONSTRUCTIVE
Note that the role of reviews is not simply to make an accept/reject decision
for CHI. Reviews are important in helping the author improve the quality
of the paper, whether it is accepted for CHI, or whether it is rejected
and revised for another conference or a journal. So please spend some time
giving comments about how the work could be improved, even if you're certain
that this is a "definitely reject" paper.
2.3 BE THOROUGH
Please be explicit about the strong and weak points of the paper. Different
reviewers often disagree about the merits of a paper, and having explanations
for why a paper was rated as it was is enormously helpful to the person
who must make a decision on the basis of widely differing reviews.
2.4 REMEMBER YOUR AUDIENCE
Remember that many design briefings are coming from people who have not
traditionally been part of the CHI community, and so they may not be aware
of conventions -- of style, or content -- that you take for granted. Also
remember that not everyone is a native speaker of English (regardless of
how polished their submission may appear). Do your best to avoid jargon
and idiomatic language in your review.
2.5 DESIGN BRIEFINGS ARE NOT PAPERS
A design briefing is a presentation of a notable user interface design.
Briefings typically focus on the evolution of the design, discussing its
rationale, users, and the studies, design techniques, and evaluation methods
that informed the development process. Design briefings differ from papers
in two important ways:
1) The aim of a design briefing is to reveal the particularities of the
design and the details of the context and practices which shaped it; extensive
references to the literature are not necessary.
2) Practices employed by designers for creating, evaluating, and modifying
user interfaces vary greatly in their degree of formality: rigorously collected,
statistically significant data, for example, is not required; however, clear
explanations of evaluation methods, decision criteria, resolution of trade-offs,
etc., are vital.
Design briefings are *not* intended to serve marketing or public relations
purposes; promotion of products and companies should be avoided as much
as possible.
See the Call for Participation for more details (https://chi1997.acm.org/call/design-br/).
2.6 CONFIDENTIALITY
You should treat the design briefings and their reviews as confidential
materials, not to be discussed outside the CHI reviewing process.
Associate Chair (Meta-Review) Guidelines
3. GUIDELINES FOR DESIGN BRIEFINGS ASSOCIATE CHAIRS
3.1 A META-REVIEW IS NOT A REVIEW
Your role is primarily to integrate the comments of reviewers, and, if
necessary,
to differentially weight the ratings of reviewers to compensate for differences
in expertise, experience, and assessment criteria, etc. You should not be
reviewing the paper itself (although it may be helpful to refer to the papers),
but producing a single document that provides a fair and coherent summary
of the design briefing, its strengths, and its weaknesses. In exceptional
cases -- if there is great disagreement among the reviewers, if there are
an insufficient number of reviews, or if there are other reasons that lead
you to be concerned about the quality or fairness of the review -- you may
optionally write your own review, and include that in your deliberations.
3.2. QUALITY ASSURANCE
Another role of the Associate Chair is to ensure that the reviews you are
working with are informative and useful both to you, and the design briefing
authors. Normally the reviews you receive will be sufficient. Occasionally
it may be necessary to ask a reviewer to provide more information, to word
a review more tactfully, or to make sure that the reviewer used appropriate
criteria.
3.3 WRITING GUIDELINES, ETC.
Of course, the Reviewers Guidelines also apply to you.
===================================================
END: CHI '97 DESIGN BRIEFING OVERVIEW AND
GUIDELINES FOR REVIEWERS AND ASSOCIATE CHAIRS
===================================================