February 3 2021
A few months ago, I spent some time exploring Rust and Elixir interoperability by toying with the Rustler library. Not being very familiar with Rust at the time, I fumbled my way through the basics. Some of the lessons learned were featured in my last blog post, a suitable introduction, I hope, to the content I’d like to present below.
I will assume some familiarity with Genservers and Rust syntax. I will also attempt to draw parallels with the Erlang C NIF library, though these are mostly remarks intended as complementary information. I will not provide a complete example in this writing, though I will point you to an exploratory project where these concepts have been put together and applied.
With this in mind, I will be discussing the translation of complex types and techniques to persist them over NIF calls. I will also bring forth the subject of calling Elixir functions from Rust, the inverse of the trivial case.
One of the first hurdles in building an Elixir & Rust application is the translation of data. Thankfully, Rustler provides powerful conveniences for encoding and decoding structured data, in the form of macros. Non-exhaustively:
NifTuple allows the annotation of Rust structs to be translated directly into Elixir tuples.NifMap: idem into Elixir maps.NifStruct: idem into Elixir structs.As an example, let’s create a struct we’d like to access both in Elixir and Rust code. The Elixir definition is straight forwards: name it, define the keys and optionally enforce them with @enfore_keys.
defmodule MyApp.SharedStruct do
@enforce_keys [:ref, :is_cool]
defstruct [:ref, :is_cool]
end
Although not strictly necessary for the translation from Rust to be successful, working with a nicely defined struct rather than a bare map, albeit with a defined __struct__ key, is likely preferable and allows for the specificity of enforced keys.
Defining the struct in Rust leverages the macros mentioned above, in this case we derive NifStruct to preserve the structure during translation. It’s worth mentioning, should the names of the Elixir and Rust structs differ, the #[module = "MyApp.OtherStructName"] annotation will help in creating the desired correspondence.
#[derive(NifStruct)]
pub struct SharedStruct {
ref: ResourceArc<Mutex<Private>>,
is_cool: bool
}
I’ve defined two fields in both struct definitions, ref and is_cool. The latter is simply a boolean. The former makes use of the rustler::resource::ResourceArc type, a thread-safe, reference-counted storage for Rust data. It is analogous to ErlNifResourceType in Erlang’s erl_nif C library and allows Rust structs to be persisted across NIF calls as Erlang terms.
Now, you might be thinking: “Cool but why?”. In short, to keep track of structs private to the Rust code. Typically these might be defined in dependencies we can’t annotate with e.g. #[derive(NifStruct)] or simply ones we don’t need access to in Elixir. In practice, this amounts to passing references in and out of the NIFs; and in this case, the in-memory location of the Private struct so that it can be accessed in a subsequent NIF call. Conveniently, Erlang’s Garbage Collection mechanism will automatically drop the struct when there are no more references to the resource. This, once again, is akin to the behaviour of enif_release_resource in C.
Lastly, data stored in a ResourceArc is immutable by default. As with std::sync::Arc, the solution, is to introduce a Mutex or a RwLock (std or parking_lot).
Calling Rust functions from Elixir is trivial, calling Elixir functions from Rust requires a few more tools. Rustler provides the handy rustler::env::OwnedEnv::send_and_clear to send a message from a Rust thread to an Erlang process.
pub fn send_and_clear<F>(&mut self, recipient: &LocalPid, closure: F) where
F: for<'a> FnOnce(Env<'a>) -> Term<'a>,
In other words, we can leverage message passing to trigger an Elixir function call, provided we know the pid of the process we’re passing the message to. Conveniently, the pid can be retrieved from the NIF’s environment with env.pid(), if the caller process is to be the recipient, or simply passed in if it’s another.
In order to construct the message which includes the SharedStruct, we create an OwnedEnv, a process-independent environment used for creating Erlang terms outside the NIF. Once the message is sent, the method frees all the terms in the environment.
let msg_env = OwnedEnv::new();
msg_env.send_and_clear(&pid, |env| (atoms::hi(), shared_struct).encode(env));
The message could then be, for instance, received on the Elixir end by a Genserver.
def handle_info({:hi, %SharedStruct{} = shared_struct}, state) do
# Handle message...
{:noreply, state}
end
As far as I am aware, directly calling Erlang functions from NIFs, be they written in C or Rust, isn’t possible other than indirectly through message passing.
Formatting Elixir-Rust projects can be done by creating an alias to run both formatters:
# mix.exs
def project do
[
# ...
aliases: aliases()
]
end
defp aliases do
[
fmt: ["format", "cmd cargo fmt --manifest-path native/tinybeam/Cargo.toml"]
]
end
Formatting Elixir-C projects is done similarly (assuming you have a ClangFormat set up):
# mix.exs
defp aliases do
[
fmt: ["format", "cmd clang-format -i c_src/*.[ch]"]
]
end
In both cases, running mix fmt will leave the codebase nicely formatted.
rustler::resource::ResourceArc. This amounts to passing references to the memory locations of the data in and out
of the NIFs.ResourceArc is immutable by default. Leverage Mutex or RwLock if the data needs to be mutated.@enforce_keys.send_and_clear with the target pid to do so.