> If I could change one thing about S3's API I would like an option to read the metadata with the listings. Rclone stores modification times of files as metadata on the object and there isn't a bulk way of reading these, you have to HEAD the object. Or alternatively a way of setting the Last-Modified on an object when you upload it would do too.
I wonder if you couldn't hack this in by storing the metadata in the key name itself? Obviously with the key length limit of 1024 you would be limited in how much metadata you could store, but it's still quite a lot of space, even taking into account the file path. You could use a deliminator that would be invalid in a normalized path, like '//', for example: /path/to/file.txt//mtime=1710066090
You would still be able to fetch "directories" via prefixes and direct files by using '<filename>//' as the prefix.
This kind of formatting would probably make it pretty incompatible with other software though.
I think that is a nice idea - maybe something we could implement in an overlay backend. However people really like the fact that the object they upload with rclone arrive with the filenames they had originally on s3, so I think the incompatible with other software downside would make it unattractive for most users.
I wonder if you couldn't hack this in by storing the metadata in the key name itself? Obviously with the key length limit of 1024 you would be limited in how much metadata you could store, but it's still quite a lot of space, even taking into account the file path. You could use a deliminator that would be invalid in a normalized path, like '//', for example: /path/to/file.txt//mtime=1710066090
You would still be able to fetch "directories" via prefixes and direct files by using '<filename>//' as the prefix.
This kind of formatting would probably make it pretty incompatible with other software though.