Great article. One of the papers it cites is https://arxiv.org/abs/2403.07183, which is also great and looks at the issue of LLM usage to write peer reviews.
It’s an issue I’ve noticed personally, as I’m seeing an increasing number of reviews that lack substance and are almost entirely made of filler content. Here’s an excerpt from a particularly egregious recent example I ran into, which had this to say on the subject of meaningful comparison to recent work:
> Additionally, while the bibliography appears to be comprehensive, there could be some minor improvements, such as including more recent or relevant references if applicable.
The whole review was written like this, with no specific suggestions for improvement, just vague “if applicable” filler. Infuriating.
Funny. It used to be if you received that sort of response, you might imagine the author being pressed for time and giving a sort of prewritten/canned copy response.
I guess LLMs have removed some of the tedium from the process while making it more tedious for the recipient. That's annoying.
It’s an issue I’ve noticed personally, as I’m seeing an increasing number of reviews that lack substance and are almost entirely made of filler content. Here’s an excerpt from a particularly egregious recent example I ran into, which had this to say on the subject of meaningful comparison to recent work:
> Additionally, while the bibliography appears to be comprehensive, there could be some minor improvements, such as including more recent or relevant references if applicable.
The whole review was written like this, with no specific suggestions for improvement, just vague “if applicable” filler. Infuriating.