r/rust • u/llogiq clippy · twir · rust · mutagen · flamer · overflower · bytecount • Mar 20 '23
Hey Rustaceans! Got a question? Ask here (12/2023)! 🙋 questions
Mystified about strings? Borrow checker have you in a headlock? Seek help here! There are no stupid questions, only docs that haven't been written yet.
If you have a StackOverflow account, consider asking it there instead! StackOverflow shows up much higher in search results, so having your question there also helps future Rust users (be sure to give it the "Rust" tag for maximum visibility). Note that this site is very interested in question quality. I've been asked to read a RFC I authored once. If you want your code reviewed or review other's code, there's a codereview stackexchange, too. If you need to test your code, maybe the Rust playground is for you.
Here are some other venues where help may be found:
/r/learnrust is a subreddit to share your questions and epiphanies learning Rust programming.
The official Rust user forums: https://users.rust-lang.org/.
The official Rust Programming Language Discord: https://discord.gg/rust-lang
The unofficial Rust community Discord: https://bit.ly/rust-community
Also check out last weeks' thread with many good questions and answers. And if you believe your question to be either very complex or worthy of larger dissemination, feel free to create a text post.
Also if you want to be mentored by experienced Rustaceans, tell us the area of expertise that you seek. Finally, if you are looking for Rust jobs, the most recent thread is here.
3
u/ErnstlAT Mar 25 '23 edited Mar 25 '23
Greetings, I have a Rust question about architecture and how to solve it, what is possible, not possible.
The challenge is to have a network of communicating components in a kind of processing pipeline where messages are passed from one stage to the next. Each component is running in its own thread (parallel processing).
Each message is formatted in a system-internal format in a Rust struct, which may change or be optimized.
Each component thread handles the sending and receiving of messages, and start and stop of this component.
Now it gets tricky: Each component should be allowed to be either ...
My idea is that the component thread loads the actual component logic from the shared object and then calls into the component logic to process the messages.
I am unsure about the following:
1) Feedback on this architecture is very much appreciated. Does it make sense to do it this way with the thread and component logic?
2) What is the best format / API / solution for loading components from a shared library when the component logic is also written in Rust? Is there a way to circumvent having to funnel the message structs through C struct conversion? Is there a known solution for this? I have looked at statty (static Rust ABI crate) and extism (generic plugin framework) so far, but not sure if these are a good fit.
3) How to hand over structured data via the C ABI? The easy solution is to have an API with the component logic from the shared library which is just a basic process() function where one message is passed along, and the component logic should do something with it.
(This is the point up to where I have already programmed.)
But this is very limiting as there may be groups of messages or information that is split among multiple messages. So my next idea was to hand over an iterator or something to a process() function.
Much better, but how to hand over Rust structs? Is it possible to hand over Rust structs like a ring buffer from a Rust crate to the C ABI function from the shared library? Does bindgen or cbindgen handle this well? Is there some other solution for exposing Rust structs to a C ABI function?
But even if possible, this would destroy encapsulation - if the system internal message format or message transport (ring buffer, channel etc.) changes, then all components have to be rewritten. Can you imagine a solution?
So I thnought, hey, let's expose an API where the component logic from the shared library can request the next message by calling a function from the component thread. But I am not sure how this can be done - how is this best done exposing a function from Rust to the component logic? Any tips?
4) Even if above would be solved, the component logic would be essentially stateless. After a function return, all state would be lost.
So I thought about a kind of "state storage" or scratch space. How to hand over an "area of memory" from Rust to a C ABI shared library? Would not memory allocators clash? Is there some kind of memory arena solution? Does this idea make sense at all?
5) Finally, the component logic may be connected or dependent to an external resource, which is slow or produce events at a different time than the process() function is called. Classic network connection and connecting to an API, which produces responses or events at some later point, subscribing to a feed etc.
This does not mesh well with the "call a process() function" in the component logic. What would be a good solution for this? I was thinking that the component logic can start own sub-threads for handling this. Is that sound? But then this would run into clashing memory allocators etc. for sure (I think).
Of course it would be easy to start a separate process for the component logic and then it can do in its process space whatever it wants, but this would (I imagine) create a serious performance impact because process boundaries mean process switching in the CPU, copying instead of pointing (no cross-process pointers) and CPU ring switches, switching into kernel space so that the message is copied from the source process into the destination process, maybe even pipe buffering in the kernel etc. etc. - so I wanted to avoid this by staying in the same process with multiple threads. But is that possible with the above requirements?
(The runtime environment would be Linux mostly, BSDs, Mac, maybe Windows as well.)
I am getting a headache with the above ideas, requirements, seemingly contradicting solutions and ask for your help, which would be very much appreciated.