I built this yesterday to help understand papers I'm interested in. It's using the gemini 2.5 flash lite model, but you can run it yourself[1] and switch to 2.5 pro for better results.
Happy to answer any questions or take suggestions on how I can improve it!
1. https://github.com/montanaflynn/asxiv
My site, https://www.emergentmind.com, is similar, though I'm two years in :)
I've found Gemini 2.5 Flash is the best model in terms of speed/cost/quality. Pro is great as well, but probably not necessary for most chat-with-paper functionality.
I'll add too that building an AI layer on top of arXiv is a deep, deep rabbit hole depending on how far you want to take the project. Drop me a note if you want to chat more about my experience with it.
Regardless, thanks for sharing this!