At CodeSandbox we use Firecracker for hosting development environments, and I agree with the points. Though I don't think that means you should not use Firecracker for running long-lived workloads.
We reclaim memory with a memory balloon device, for the disk trimming we discard (& compress) the disk, and for i/o speed we use io_uring (which we only use for scratch disks, the project disks are network disks).
It's a tradeoff. It's more work and does require custom implementations. For us that made sense, because in return we get a lightweight VMM that we can more easily extend with functionality like memory snapshotting and live VM cloning [1][2].
I don't know if this is relevant, but I've been intrigued by DragonflyBSD's "vkernel" [0] feature which (supposedly) allows for cloning the entire runtime state of the machine (established TCP connections, etc.) into a completely new userland memory space. I think they use it mostly for kernel debugging right now, but it's interesting to think about the possibilities of being able to just clone an entire running operating system to a new computer without interrupting even a single instruction.
At CodeSandbox (https://codesandbox.io) we're also working on this! Main focus of us is that we're running the environment in Firecracker microVMs, which allows us to snapshot and clone environments very quickly. This enables us to create a VM for every branch, which comes with the added advantage that every branch automatically has a snapshotted preview environment that can resume in ~2 seconds.
> what version of the kernel do you use (the github page says 5.10 but isn't that quite old?)
Right, they have tested with 5.10, but it also works with higher kernel versions. Our host currently runs 5.19 and we're planning to upgrade to 6.1 soon. The guest runs 5.15.63, we use a config very similar to the recommended config by FC team (it's in the FC repo). It's important to mention that we had to disable async pagefaulting (a KVM feature) with more modern kernel versions, as VMs could get stuck waiting for an PF resolve.
> What do you use to build the 'micro' images
We created a CLI that creates a rootfs from a Docker image. It pulls the image, creates a container and then extracts the fs from it to an ext4 disk. For the init, we forked the open sourced init from the Fly team (https://github.com/superfly/init-snapshot) and changed/added some functionality.
> How do you keep timesync of you're not using a timesync daemon?
IIRC we expose the time as a PTP device (handled by kvm) and run phc2sys to sync the time in an interval. Firecracker has some documentation on this, where it recommends chrony. It can also be done with vsock, but it would be more manual.
> Handle kernel and app logs without adding an log daemon, and same through vsocks, etc?
The init forwards stdout/stderr of the command it runs to its own stdout, which Firecracker then logs out by itself. A supervisor reads these and writes the logs to files.
> We created a CLI that creates a rootfs from a Docker image. It pulls the image, creates a container and then extracts the fs from it to an ext4 disk.
You guys don't happen to have a public writeup about how this works, do you? Maybe it's as simple as it sounds, but Fly and CodeSandbox both have some magic to turn Docker images into VM disks that I'd like to know how to build :)
Fly is doing fancy stuff to avoid using docker entirely, but with docker you can just run "docker export" to dump an image to a .tar file that contains the whole filesystem. Built-in feature. I use this as a convenient way to grab a foreign platform sysroot for clang cross-compilation; just pick a Docker image and rip the filesystem out.
Oh thanks a lot. PTP! I need to try this out. And thanks for the init from fly.io too! And... I think I know now why my vms would get stuck thanks for the tip!
Hey all! I'm one of the co-founders of CodeSandbox. We just launched Python support on CodeSandbox, which allows you to quickly create Python prototypes or do full development of Python in CodeSandbox (together with our VSCode extension).
If you have any questions, either about the implementation or the product, don't hesitate to leave a message!
I'm using a web based IDE for my dev, but I'm biased in that sense. The main advantage for me is that I can easily switch branches, as every branch has its own VM. + I can easily share in-progress work. I do use the VSCode integration, because I'm very used to VSCode.
That said, I've also spoken with people who use a Web IDE next to their local environment. E.g. they use a Web IDE for reviewing PRs or making smaller changes, and they use their local editor for feature development.
If this is just for being able to easily switch branches, have you considered using git worktrees? Essentially you have have multiple branches checked out into different directories at the same time. The key point though is that all git refs are still maintained in a single place so you are not actually having full clones for each worktree that you have.
I personally use Emacs and magit which has great support for worktrees. I do not know if the VSCode git integration supports it.
Ah right, the main advantage I have here is that the dev server also stays running when I go to another branch (since it's a different VM). So I can work on one thing, share it with the team and in the meantime continue on another branch. They can see the devserver / running code, while I am working on something else.
It's especially useful for things like migrations or dependency management. In one branch I could work on something that has some database migrations, and then I can still switch to other branches without having to roll back the migrations.
I didn't know about git worktrees though! I'm going to read up on this.
> In one branch I could work on something that has some database migrations, and then I can still switch to other branches without having to roll back the migrations.
This is a good use case of git worktree.
> I didn't know about git worktrees though! I'm going to read up on this.
Indeed, you will like it.
The only limitation is that you cannot checkout the same branch in multiple worktree.
> In one branch I could work on something that has some database migrations, and then I can still switch to other branches without having to roll back the migrations.
I use it with VS Code since the project I work on takes 5-10 minutes to build if a major header is modified. It shows up as a separate directory, but VS Code will still realize that the directory is a git worktree.
The only thing kind of clunky is that you have to keep track of which branch is open in which directory since you can't checkout the same branch in two different directories.
> The only thing kind of clunky is that you have to keep track of which branch is open in which directory since you can't checkout the same branch in two different directories.
Hey all! I'm one of the co-founders of CodeSandbox and a big fan of Rust. Rust support is something that we've been working on for a while, but the recent addition of Docker support in CodeSandbox really enabled it. Happy to answer any questions!
I recently tried to work on CodeSandbox. Simple examples didn’t work or I couldn’t figure out how to make them work. Using repl.it now. I highly recommend to go through basic examples of frontend projects and evaluate them.
I'd say that CodeSandbox has a strong focus on extending the existing workflow for developers. That's why we have a VSCode integration, a GitHub integration with a GitHub App that creates a running dev env for every branch/PR, and we make sure that generally all editor features you expect (autocomplete, go to definition, hover info, etc) are available for the languages that we support.
No, we've built our own web editor & iOS code editor. In our v1 editor we did run VSCode in the browser, but that was before code-server was released (in 2018). Even if we wanted to run code-server, that would be impossible as we allow for multiple users to open the same sandbox/branch, which wouldn't fit the model of VSCode server (which is single user per server).
We reclaim memory with a memory balloon device, for the disk trimming we discard (& compress) the disk, and for i/o speed we use io_uring (which we only use for scratch disks, the project disks are network disks).
It's a tradeoff. It's more work and does require custom implementations. For us that made sense, because in return we get a lightweight VMM that we can more easily extend with functionality like memory snapshotting and live VM cloning [1][2].
[1]: https://codesandbox.io/blog/how-we-clone-a-running-vm-in-2-s...
[2]: https://codesandbox.io/blog/cloning-microvms-using-userfault...