We had this at Clemson University about 10 years ago. I think the whole design came out of an MIT lab though. There's video of it still up here: https://www.clemsongis.org/clemson-sandbox
it's not the exact science museum experience mentioned, but a quick Google search for "sand table video augment" pulled up a few, and this was the first one I found:
https://share.google/89A6x4yfaw5a4hOuh
When the first Xbox were getting long in the tooth, I believe people were repurposing the motion tracking bar as the mechanism to measure the topography of the sand table. That, coupled with a video projector mounted over the top of the sand table provides the additional colors and elevation lines. (And of course a bit of cool software to process and produce the image.)
This is one of those things that's really not that hard, nor expensive, with one decent hacker who wants to set it up. Maybe $1500 of parts? Feels like the kind of donation lots of people here could make to a local lower budget kids science centre. And I bet would be the kind of donation these centres would love to have.
As a bonus: you can likely make it out of "100% recycled e-waste" and "100% recycled lumber" (if you're building the table, too), giving it an extra educational theme. not only is this cool, fun, and educational, but it's a demonstration of doing something good with a used depthsense, projector, and computer.
Like a regular 2D fluid sim, the projection is not the bottleneck. The sim can be slow or fast depending on the quality you want. This one runs fine on integrated graphics card on my laptop.
The Phaeno is a place I think would be perfect. They are well funded and have interest in everything remotely automotive related. Also, they like to blend the border between science and art. Make sure to add some silhouettes of modern and old cars when pitching to them.
That would be hard to source / use, as It's a bit dated. But the technic behind it could be replicated. It uses IR LED on the bezel, around a glass. With total internal reflection the light only shows up where you touch the glass. Then film that with a camera. The advantage to my method, you don't get interference of objects hovering above the screen. But getting the camera & display behind the screen, could be hard.
I've just done something similar with a Object tracking + IR Floodlights and one of VA Imaging's MER2 cameras. We have had some fantastic results with objects sat on these "Interactive Dance Floors" from Alibaba. Didn't even need the IR Absorbing film to get decent tracking results using OpenCV @ 300fps or so.
This sound intriguing. Is the camera placed above the scene? How do those interactive Dance Floors come into play, did you use them for the light? Do they not have triggers, when somebody stands on it?
Yep, we've got the camera mounted vertically above the screen. We use the dance floors to render an interactive Unity game. The pixel density is pretty good. These panels have an array of IR LEDs embedded in them for detection but the density isn't good enough for small objects we want to track. Great for detecting anything the size of a foot or bigger.
Isn't that what VCs in general are doing? Hiring for more money, with more expected gains from you, with a different kind of legal arrangement, but still hiring nevertheless.
Funding people means you trust the people are so good they will push any idea to success. Funding an idea means you trust the idea is so good it will push any people to success.
Funding people means having a lot of trust in them. What's unsaid is if the investor believes coloring outside of the lines to make everyone more money is a breach of that trust, or just the normal cost/risk of business.
How do you do the rendering? Is it sorted (radix?) instances? Do you amortize the sorting over a couple frames? Or use some bin sorting? Are you happy with the performance?
Yes, Spark does instanced rendering of quads, one covering each Gaussian splat. The sorting is done by 1) calculating sort distance for every splat on the GPU, 2) reading it back to the CPU as float16s, 3) doing a 1-pass bucket sort to get an ordering of all the splats from back to front.
On most newer devices the sorting can happen pretty much every frame with approx 1 frame latency, and runs in parallel on a Web Worker. So the sorting itself has minimal performance impact, and because of that Spark can do fully dynamic 3DGS where every splat can move independently each frame!
On some older Android devices it can be a few frames worth of latency, and in that case you could say it's amortized over a few frames. But since it all happens in parallel there's no real impact to the overall rendering performance. I expect for most devices the sorting in Spark is mostly a solved problem, especially with increasing memory bandwidth and shared CPU-GPU memory.
If you say 1 pass bucket sorting.. I assume you do sort the buckets as well?
I've implemented a radix sort on GPU to sort the splats (every frame).. and I'm not quite happy with performance yet. A radix sort (+ prefix scan) is quite involved with lot's of dedicated hierarchical compute shaders.. I might have to get back to tune it.
I might switch to float16s as well, I'm a bit hesitant, as 1 million+ splats, may exceed the precision of halfs.
We are purposefully trading off some sorting precision for speed with float16, and for scenes with large Z extents you'd probably get more Z-fighting, so I'm not sure if I'd recommend it for you if your goal is max reconstruction accuracy! But we'll likely add a 2-pass sort (i.e. radix sort with a large base / #buckets) in the future for higher precision (user selectable so you can decide what's more important for you). But I will say that implementing a sort on the CPU is much simpler than on the GPU, so it opens up possibilities if you're willing to do a readback from GPU to CPU and tolerate at least 1 frame of latency (usually not perceivable).
You might want to consider using words (16 bit integer) instead of halfs? Then you can use all the 65k value precision in a range you choose (by remapping 32bit floats to words), potentially adjust it every frame, or with a delay.
Yeah you're right, using float16 gets us 0x7C00 buckets of resolution only. We could explicitly turn it into a log encoding and spread it over 2^16 buckets and get 2x the range there! Other renderers do this dynamic per-frame range adjustment, we could do that too.
Do you have a picture of the installation at the science museum?
The color just rotates through the oklab space (change hue only).