April 30, 2025
Mara Solen
I've faced a lot of rejections lately, with three of my last four authored and co-authored papers as well as one grant and one fellowship all being being rejected. Two of the rejected papers are supposed to be in my thesis, I still have two additional in-progress papers that must be in my thesis, and failing to get the fellowship means that I have to finish my PhD sooner rather than later. All of this together means that I need to finish four full papers by the end of the summer, which is stressful!
Because I've been thinking about these rejections and trying to stay motivated despite them, I decided to take a look at something that I have thought was funny in the past. When you submit a research paper, it is peer-reviewed by a set of reviewers. In my field, the systems that we use to review force us to select a rating for the paper overall, reviewer expertise level, and often a few specific attributes about the paper like novelty, utility, and rigor. These ratings are typically given on likert scales. One thing I've noticed in the past is that I've gotten reviews where two reviewers choose options on opposite ends of the rating scale for one of these attributes! For example, I submitted a paper for which one reviewer put a 5/5 (excellent) for the research rigor, and another put 1/5 (fundamentally flawed).
This observation led to my interest in exploring how consistently my past reviewers have rated my papers. In the visualizations below, each loop represents one paper version that was reviewed. Loops that overlap are part of the same submission cycle, as some venues like the CHI conference or TVCG journal include multiple review rounds where all reviewers assess the submission. If the loops are very circular, it means that reviewers largely agreed, while loops that are very misshapen indicate that reviewers did not agree. The bigger the loop is, the higher the ratings. The colour represents the results of the reviewing round, with green meaning revisions requested, brown meaning reject, and pink meaning accept. I chose this colour scheme to be like a flower that is growing (green) and then either blooms (pink, accepted) or dies (brown, rejected), because many of these papers needed multiple seasons before finally properly blooming.
I think it's interesting how many of my submissions have strong reviewer disagreement. At the same time, it's interesting to me that the ones that do eventually get published definitely become rounder and bigger over time, which makes sense.