A striking example of how not to present data. If the Cognizant AI team is here: please can you fix it in the next version of the paper?

Obviously, not the best plot to use according to Data Visualization theory and common practice, but I think it candidly conveys the point anyway.

As someone else points, the data is the worrying aspect, as it points towards state-of-the-art models not being able of making more than 0 consecutive steps without errors.

I think it's a brilliant example of how to use data to make a point.

https://xkcd.com/1162/

Except on figure 1 they're all at 0, making it look like the authors didn't know how to use the models or deliberately made them do nothing.

I think it just looks that way because they used a linear x axis for comedic effect.

I was just thinking "these guys will talk about this graph for the rest of their lives", it's the best graph you could ever hope to put into a paper. Loved it.

In case you want to know what’s going on in the left side of that chart, they gave a log scale in appendix a. I was thinking it was silly to not just use that version on the top, but I guess log scales make big differences ’feel’ smaller.

A log scale is actually appropriate in this context from a first-principles perspective. Per scaling laws (and also general behavior of epsilon-probability of failure multiplied N times), you would generally expect more vs. less effective techniques to have multiplicatively greater or fewer steps until failure, not additively greater/fewer. Figure 1 is comical, but the appendix figure is the more scientifically appropriate one.

At that rate, they might as well have gone one step further and made the x axis exponential scale to make it feel even bigger.

The dashed lines on top of the data points and labels is making me wince

Really seems like the reason logarithmic scales were invented..