Oldie but goodie is Nodebox[0] and for data-driven/dynamic compositing see Cables.gl[1]. Kirell Benzi's work uses the latter and is nothing short of breathtaking[2].

[0]: https://www.nodebox.net/

[1]: https://cables.gl/

[2]: https://youtu.be/CltYdTVH7_A