Since launching the Suborbital Extension Engine (SE2), support for Python has been one of our top feature requests. Today, partnering with Wasm Labs at VMware, we're excited to announce the debut of Python in SE2! With access to Python's extensive standard library, users working with data can do so in the language they know and love for its readability and versatility.
Okay, “hello world” is not the most exciting screenshot. And there's some boilerplate we need to clean up. But it runs! And what's behind the scenes is the culmination of an enormous amount of work from the core Python team, VMware, Suborbital, and the broader WebAssembly community. Let's dive in!
The Path to WebAssembly
SE2 is a complete plugin system for SaaS applications: users submit code and SE2 handles the full lifecycle of safely running that code to customize and extend the host application. Goodbye, webhooks!
We do this by packaging the user-provided code into standalone WebAssembly modules, leveraging WebAssembly's exceptional sandboxing, performance, and portability.
Turning source code into a WebAssembly module is relatively straightforward for compiled languages — wasm32 is “just” another target architecture, like Intel's x86 or ARM's aarch64. But because Python is an interpreted language, our builder has to do additional work to put everything together.
Inside the Builder
SE2 is built around storing and executing standalone WebAssembly modules. To support Python, we must first compile the CPython interpreter to WebAssembly, then combine it with the user's Python script, and finally package the whole thing up into a self-contained module.
We also need a bidirectional foreign function interface, so that the SE2 runtime can invoke the user's code, and the user's code can import and access SE2's developer APIs.
This takes a lot of glue code, but fortunately the Wasm Labs team at VMware have been working on bringing popular language runtimes—including Python, Ruby, and PHP—to WebAssembly in a way that supports exactly this use case. All of their pre-built modules are available in the Wasm Language Runtimes project.
We worked together with VMware to get Python running end-to-end in SE2, and ta-da, it works! The python.wasm we use in our builder is the product of this collaboration.
Of Filesystems and Pre-initialization
There's a lot of nuance to getting this right—which we'll save for a future article—but one facet worth digging into is filesystem access.
The WebAssembly System Interface (WASI) is designed to be secure by default. For example, WASI does not allow filesystem access except to paths that are explicitly pre-opened by the host runtime. And that’s exactly the behavior we want: we designed SE2 to run stateless, ephemeral instances of WebAssembly modules, and denying access to the underlying disk simplifies our operations at scale.
...except Python really wants to read its standard library from somewhere.
So what can we do?
For SE2, the right solution is to use wasi-vfs to synthesize a virtual filesystem right inside the WebAssembly module. And that's on our roadmap.
But for this first release, we're doing something different. WebAssembly (with WASI) is capable of safe, sandboxed access to the host filesystem, but as mentioned before, we don’t want to allow that at runtime in order to keep SE2’s architecture as simple and robust as possible.
Instead, we grant limited filesystem access only during the build process. By using Wizer to pre-initialize the Python interpreter, we create an opportunity for Python to read the relevant bits of its standard library into memory, where they can be accessed without reaching back out to the disk. Then, we take a snapshot right before jumping into the user-provided code.
We'd want to pre-initialize for performance reasons regardless—it's way faster to start from a snapshot than from scratch—but the side effect of not needing further access to the filesystem at runtime is great!
There are corner cases where this fails, so we’ll need to incorporate wasi-vfs into a future release of SE2, but naïve pre-initialization works surprisingly well for scripts which only use normal imports from the Python standard library.
Third Party Modules
Lastly, while the Python standard library is great ("batteries included!"), it's not complete. What makes Python so versatile is its robust ecosystem of third-party modules.
We don't support those, yet.
Two reasons:
Many popular Python libraries include native Rust, C, or Fortran code which we also need to somehow reliably compile to WebAssembly.
The SE2 user experience is designed around small code snippets—individual functions—so we don’t yet have an obvious place for users to specify their third-party dependencies.
Both of these are eminently solvable; they’ll just take time. And until then, our users have access to all of the wonderful features (and clean syntax!) that Python itself brings to the table.
If you want to keep updated on our progress, make sure to subscribe to our newsletter or follow us on Twitter! You can also sign up for the SE2 beta and we’ll let you know when we’re ready for you to take it for a spin.
A sincere thank you to Laura Langdon, Oscar Spencer, and the entire Wasm Labs team for assistance in preparing this article.