I wrote zoekt. From what I understand valtown does, I would try to use brute force first (ie. something equivalent to ripgrep). Once that starts breaking down, you could use last-updated-timestamps to reduce the brute force:
* make a trigram index using Zoekt or Hound for JS snippets older than X
* do brute force on snippets newer than X
* advance X as you're indexing newer data.
If the snippets are small, you can probably use a (trigram => snippets) index for space savings relative to a (trigram => offset) index.
I wrote zoekt. From what I understand valtown does, I would try to use brute force first (ie. something equivalent to ripgrep). Once that starts breaking down, you could use last-updated-timestamps to reduce the brute force:
* make a trigram index using Zoekt or Hound for JS snippets older than X * do brute force on snippets newer than X * advance X as you're indexing newer data.
If the snippets are small, you can probably use a (trigram => snippets) index for space savings relative to a (trigram => offset) index.