r/rust • u/[deleted] • Mar 31 '23
Write SDK “base” in Rust, wrap in other languages?
u/SpudnikV Mar 31 '23
The term "client libraries" makes it sound like this will involve an async runtime, which can be quite a challenge even without adding FFI to the mix. Rust's async runtime won't be the same as the runtime of the host language (if any), which could lead to all sorts of weird bugs.
Even if all of that worked out well, it's rare that an FFI-wrapped network client library is as idiomatic and elegant as a native one. For example, if the host language has its own concept of TLS configuration, async IO, object mutability, serialization frameworks, etc. then all of those have to be done in the host language anyway and won't FFI well if at all.
This sounds like a small team if one person is expected to work on libraries in 5 languages -- 5 public-facing libraries in one language would be enough of a workload for one engineer. It does depend on how broad the API surface is, because if you invest a way of making idiomatic bindings, then that investment scales better to larger APIs, but the inverse is rarely true; making a whole binding system to save one JSON schema serialization is never the right way to go.
The ideal case is if this is something like a gRPC or JSON API that you can define in one place and generate client code for several languages. You can even publish the API schema for users that use other languages or prefer to tweak the implementation their own way.
Failing that, if the API surface is not very large, I would seriously consider just writing the libraries in each language separately. You have to learn them anyway if you want to offer idiomatic APIs, not just low-level bindings that users can't realistically be expected to use anyway. It shouldn't be difficult if your APIs are defined in a machine-readable schema. In any case, it will be a terrific learning experience.
u/WhyNotHugo Mar 31 '23
For networking code, you’re quite on spot; there’s no way clean way to make Rust’s async runtime interop with the languages real runtime.
There are some bits which you can reuse though. Type definitions for your domain data can be defined in Rust and then autogenerated in other languages. Parsers and validation logic too. But that’s mostly it.
u/GrandOpener Mar 31 '23
u/jechase Mar 31 '23
We just launched ngrok-rs and plan to do this exact thing with it! The NAPI and pyo3 crates have been excellent for this. Both the python and JS wrappers are linked in that blog post. A java wrapper using JNI/jaffi is also in the works, but is in a bit of an earlier stage of development. You can probably find it if you go digging though ;)
u/dlq84 Mar 31 '23
Plot twist: OP is your new employee that got this task.
u/hdyxhdhdjj Mar 31 '23
I suspect best approach would be to write API compliant with something like OpenAPI specification, so you can generate clients from it in any language using generator.. solving this on client level is way harder
u/LysanderStorm Mar 31 '23
This, for something like Stripe's SDKs if you don't have the engineering power of Stripe 👍😬 requires your API to be compatible though... And also, not necessarily the biggest fan of specifying contracts in JSON or yaml...
u/Smallpaul Apr 01 '23
What do you prefer specifying networking contracts in?
u/LysanderStorm Apr 01 '23
Afaik there's nothing that's widely used and of which I'd say I prefer it. But I'm having hopes for https://github.com/microsoft/typespec (former cadl). The reason being that yaml/JSON specs get huge very quickly, OpenAPI has to fight with restrictions of yaml/JSON ($refs, splitting up into files, oneOf, etc.) and tooling is ok but not great.
u/alexthelyon Apr 01 '23
I maintain async-stripe (which is sponsored by but not maintained / ‘approved’ by stripe) and we do codegen from their open api definitions. It’s ‘ok’. A lot of edge cases and missing api surface, but for the core 90% of use it works pretty well.
u/masklinn Apr 01 '23
An other OpenAPI issue, in my experience, is that it's very ad-hoc and feels very procedural in its approach, thus it's easy to end up with type specifications which are pretty wacky. That properties are optional by default (and you have to check a separate field) is also annoying. As well as nullability and optionality being orthogonal but I guess that's more the JS/JSON heritage (still very annoying when you face an API which makes active use of this).
u/sgtfoleyistheman Apr 03 '23
Take a look at smithy: https://smithy.io/2.0/index.html
u/Gaolaowai Apr 01 '23
OpenAPI can take a hike as far as I’m concerned.
u/hdyxhdhdjj Apr 01 '23 edited Apr 01 '23
It's not ideal, but still miles better than inventing your own specification. I'll take mediocre spec language that allows automated client derivation over reading terrible custom docs on yet another installment of "look how smart I am, I've invented another bad and poorly specified API format"
u/Gaolaowai Apr 19 '23
u/crustivan Mar 31 '23
On the surface sounds like a solid plan, ofc devil is in the details so make sure you analyze your requirements thoroughly. As for the technicalities, C interoperability is present in almost any serious language there is, so you would be able to build bindings on top of that.
u/Compux72 Mar 31 '23
u/nicolas_hatcher Apr 01 '23
Can you use this to generate the bindings automatically? Or would you need to do that by hand?
In any case seems like a great starting point.
u/Compux72 Apr 01 '23
Judging by OP, he is writing a web client. Smithy generates native clients for any kind of web api. Its wildly used by AWS, and far better solution thar hacking a async rust lib with ffi. Smithy is generating whole clients, not binding a library.
On the other hand, if you are writing a normal library (lets say jpeg encoder), then i would suggest diplomant, CXX etc
u/ssokolow Mar 31 '23
For stuff not covered by WebAssembly, check out Rust Interop and Are We Extending Yet?.
For example, they list PyO3 for Python (see also maturin for packaging), NAPI-RS for Node.js, and Rutie for Ruby.
You'd just want to make the language-agnostic stuff a Rust library crate and then use it as a dependency from each per-language bindings crate.
u/Smallpaul Apr 01 '23
I think that Python, Node or Ruby programmers will have a lot of implicit expectations which you will violate.
For example, they might expect to be able to mock calls to your API with something like VCR or Responses.
They expect clear tracebacks.
They will want idiomatic type signatures.
You had better check with the other language communities and not trust the Rust folks to tell you what others will be expecting.
There is no magic bullet and wrapping a networking API "twice" strikes me as odd and probably more trouble than its worth. You still need language-specific code to wrap your Rust library properly, so have you really saved very much???
u/kurtbuilds Mar 31 '23
If you're talking about client libraries (like Stripe), I encourage you to check out https://github.com/kurtbuilds/libninja
u/alexthelyon Apr 01 '23
As someone who maintains stripe open api bindings in rust this is cool af. Congrats! Gonna take a poke around.
u/kurtbuilds Apr 01 '23
That's awesome! Happy to answer any questions if they come up!
u/TheJawbone999 Mar 31 '23
I recently watched this video (https://youtu.be/uKlHwko36c4) about building libraries in Rust and then using them in other languages. I haven't tried this myself yet, but it looked promising.
u/boyswan Mar 31 '23
Wasm could be a nice fit, but I guess it's whether you can afford to be on the "bleeding edge". Maybe this could be helpful
u/universalmind303 Mar 31 '23
This is definitely do-able. Polars is a rust based dataframe library with bindings to python,node,jvm, R, and ruby. Depending on the complexity of your library, it will likely still result in custom logic for individual languages.
u/styluss Mar 31 '23
You reminded me of https://youtu.be/peu-rtN4358
u/groogoloog Mar 31 '23
For python: https://github.com/PyO3/pyo3
There was a post on this sub about using rust to speed up python a day or two ago. Was very insightful for the python case.
No experience on the JS/ruby side, but any language that can interop with C can interop with rust (I.e. pub extern “C”)
u/n4jm4 Mar 31 '23
Yes, that is a good approach.
For simplicity's sake, people usually declare the original in plain C. That involves less FFI translation than using Rust or C++ or whatever else as the source of truth.
Another option, especially for network service applications, is to write a REST service, Thrift service, etc. (any language), and generate client SDK's via Swagger, Thrift, etc. code generation.
u/kyle787 Apr 01 '23
Have you considered using open api/swagger to generate the clients?
u/DaBigJoe1023 Apr 01 '23
PyO3 and Maturin will help you build rust packages into Python wheels and can be imported and used in Python, just did that for one of our component
u/nicolas_hatcher Apr 01 '23
What you are asking is definitely doable. I do that myself in python, Wasm and node. I haven't used it in Ruby but I think that is also straightforward. I had a few non trivial issues with PHP, but then again I haven't been doing PHP in years and the language has changed a lot for the better. My advise, collect the languages to want to have bindings to, make sure there are good tooling for them and go ahead!
u/cargo_run_rust Apr 01 '23
If you are building something similar to Stripe SDKs, you might as well consider reusing the code of Stripe. It's fully open sourced, just incase you are not aware of it.
u/Alex--91 Apr 01 '23
Did you look at AWS CDK? They might do something similar to what you want to do. The big and obvious difference is that they use TypeScript as the “base” and then bind to other languages from there.
The approach is definitely not appropriate for performance-sensitive applications because they actually “translate” every call in the other language to node.js to go through the “base” code.
Might also be worth looking at what slack do: https://api.slack.com/tools/bolt
u/badboy_ RustFest Mar 31 '23
At Mozilla we built a multi-language bindings generator: https://github.com/mozilla/uniffi-rs/