Linux people doing Linux things, it seems.
“It’s herding cats: introducing Rust effectively is one part coding work and ninety-nine parts political work…”
All software development in a team is. More like 20/80 or 40/60 if you’re lucky.
Except in this case, it was a bunch of old C devs who aren’t just resistant but openly hostile to change, and they’d rather bully people into silence than try to progress.
Several downvotes with zero comments to refute or discuss your point. Some devs don’t like you calling them out
In a twist of delicious fate, my instance doesn’t have downvotes. They get dropped before they even hit the database. So I’ll never know or “feel ashamed” if they don’t bother to take time to refute it. 🤣
If you want to talk about bullying you ought to include all the rust zealots who show up to shit on C every chance they get.
Okay, but this was but an example of that, so it’s not really a relevant grievance, is it?
bunch of old C devs
I knew this ageist bullshit would pop up. I know we lost our mentors and are kinda feeling in the dark, but the moment people pop out the ageist slurs I know they’ve got nothing to say.
“Old” doesn’t have to mean biologically old. In this case, it means people who have been doing it for a long time—long enough that they’re set in their ways.
So while I can understand the confusion, it doesn’t apply here.
The C developers are the ones with the ageist mindset.
The Rust developers certainly are not the ones raising the point “C has always worked, so why should we use another language?” which ignores the objective advantages of Rust and is solely leaning on C being the older language.
Here’s the thing: you’re not going to force all of us to learn Rust.
That’s precious coming from Linux developers.
I am a heavy Linux user. I run multiple microservices on multiple headless devices all Linux.
This sounds like every fucking Windows user you’ll ever encounter.
“Here’s the thing: you’re not going to force all of us to learn to use Linux.”
So, yeah…
Switching everything from C to Rust because it has better memory safety is more akin to changing languages from English to Esperanto because it has gender neutral pronouns and other cool features. Maybe it’s a good idea, but it’s understandable that some people are reluctant.
Maybe it’s a good idea, but it’s understandable that some people are reluctant.
I understand that position. I also understand how the words and phrases that the C community has used to communicate with the Rust community seems to be completely dismissive, not just reluctant.
I quoted what I did explicitly because of how a statement like that comes off to the person it’s aimed at. It doesn’t make them feel like they’re on an even footing working on the same project with the overall goal of it becoming better.
memory safety is more akin to changing languages from English to Esperanto because it has gender neutral pronouns.
I mean… not at all? Memory safety is huge for cybersecurity, buffer overflows and the like are common attack surfaces. C requires you to have deep knowledge of safe memory management practices and even then you can end up with memory issues. Rust was developed to avoid such issues entirely. I understand the reluctance but it feels to me like arguing “we should just stick with COBOL because it works.”
Gender neutral pronouns are pretty huge too. Sure you can do them in English without too many problems usually, just as it’s also possible to code safely in C. It requires everyone to change their old habits, but it’s much less of a change than is involved in adopting a whole new language.
Anyway, I do like Rust better personally.
I would still say that getting people to the point where they can write safe C code every time is harder than learning Rust, as it’s equivalent to being able to write rust code that compiles without any safety issues (compiler errors) every single time, which is very difficult to do.
Seems like it’ll be the year of RedoxOS desktop! /s
Adding rust to a massive mature C project that targets lots of architectures and has many contributors is a difficult process. If it succeeds it is going to take a lot more time and patience.
Especially when Rust has limited support for less common architectures. This has been forcing distros like gentoo to drop support for more niche arches since many common packages like python-cryptography are now pulling in rust as a mandatory dep.
Is there an email thread where c and rust Linux kernel devs actually discuss what’s about? Because so far I see some Linux drama and I have no slightest clue what’s about
Rust guys want to make the kernel safer, more expressive, and easier to maintain. To do that they need to know how the kenrnel talks between its parts to ensure they are creating matching behavior. The C guys don’t really care about the Rust guys and say that they can’t be bothered to guarantee interoperability because they like to change how things work on the C side to make things better in the C code.
Here’s what I think. Both opinions are correct.
Rust is sufficiently different that you cannot expect C developers to learn rust to the level they have mastered C in order to be working at the kernel level. It’s not going to happen.
I don’t really know too much about rust. Maybe one day I’ll actually mess around with it. But the one time I looked at a rust git repo I couldn’t even find where the code to do a thing was. It’s just different enough to be problematic that way.
So I think probably, the best way IS to go the way linus did. Just go ahead and write a very basic working kernel in rust. If the project is popular it will gain momentum.
Trying to slowly adapt parts of the kernel to rust and then complain when long term C developers don’t want to learn a new language in order to help isn’t going to make many friends on that team.
Good news there’s a project that aims to implement Unix in Rust called Redox and it’s already a good enough project for studying microkernel design
Bad license though
Good license to capitalists.
Not even, it will suffocate on its own by having the capitalists keeping their changes from each other. Like a bucket of crabs; where if one crab is about to get free the others grab onto it and pull it down.
Kernels really benefit from being “forced” to share the code changes as the GPL license, they are too tied to HW, and HW needs a lot of capital when iterating.
Permissive licenses mean faster and more widespread adoption, it’s up to project maintainers if the tradeoff is worth it. Ideally a company would realize that an open source part of their project probably isn’t radically going to affect their revenue stream, but you don’t just have to convince devs, you have to convince the suits and lawyers, and they will tell you to just build your own rather than give up any precious IP.
But that’s the thing where you are wrong. They clearly state they don’t want C developers to learn Rust. In the particular video posted he was saying “I want you to explain to me how this particular API works so that I can do it”
The concerns about who fixes what on a merge when the C code breaks Rust code are valid, but that’s easily fixed by gathering with the Rust developers, explaining the changes and letting them fix it.
You could alternatively phrase “But that’s the thing where you are wrong” as “But here’s the crux of why I disagree”, it’s a bit more personable
This isn’t a disagreement. One person is stating something incorrect. You can disagree on opinion, but facts are facts. The person being referred to here isn’t asking others to learn Rust, they are just asking for more information about the already existing C code so that they can write their Rust code to interoperate with it. This misunderstanding is exactly why that developer was getting heckled on stage, and is the reason why now one has left the project. I would appreciate it if you didn’t make a misunderstanding sound like a valid opinion. Enough damage has already been done.
It doesn’t matter if you know it’s a fact, and i agree with you for the record. It’s about bringing people along with you - you catch more flies with honey than vinegar - and creating good vibes in the softwaresphere
Have you ever tried catching flies? Vinegar works better than honey, after all, flies eat shit.
That to me sounds like exactly the reason why developers like the above have left. They are having to take on the burden of gently letting down other devs who are angry over a simple misunderstanding. A misunderstanding that wouldn’t have happened if they had been listening or bothered to ask first before jumping to conclusions. Imagine someone heckles you on stage and you have to respond kindly. I certainly wouldn’t. If someone had listened to my talk, misinterpreted it, then heckled me over it you can bet I would be angry and would respond in kind. To then see this misinformation being spread again would drive me nuts. I can see why they left.
The bottom line for me is that Rust devs who work on this stuff for free shouldn’t be getting hounded by C devs just for asking for proper documentation that frankly they should have provided in the first place. I say this as someone who is skeptical of Rust for various reasons.
I’ve inserted myself into your C project because only idiots write C. Rust is the one true god, mUh MeMoRy sAfteY! Now please explain to me how C works.
LMAO
Not an expert in both the languages but I heard that C developers are trained to use memory smartly, sometimes even reuse a range of allocated memory for completely different purpose to save cycles freeing and reallocating. But for Rust developers, everything is about making sure when one should get the hand away from the memory, and whose memory is allowed to be touched.
Sounds to me like sharing rides that maximise economically but we may have some oops moments sitting on someone’s laps vs absolute private rides to make sure no one in your family will be harmed but we have to make sure everyone gets a car only when needed.
It is quite interesting to see how it will work out eventually…
Unfortunately there are a lot of problems created by using C in the kernel, and having all of this done manually. Many kernel vulnerabilities including several severe ones have been due to issues with memory management. Even the whitehouse has spoken on these issues related to C. Rust has been proven to be comparable to C in terms of performance, sometimes even faster. So it doesn’t make a great deal of sense to keep using C for new projects.
That all being said Rust has had its own issues. There was a recent vulnerability in older versions of cargo the Rust package manager for instance. It’s a somewhat new language so obviously teething issues are to be expected, and it might be too soon to use Rust for mission critical systems. It’s also a harder language to learn and understand, so that makes adopting it more difficult especially for very experienced C developers like those who work on the Linux kernel. It might be better to wait and see what other languages like Zig and Carbon manage to do, but those are even newer and will take more time to actually be production ready.
Expecting C programmers to like a compiler-based approach to memory safety is like expecting petrolheads to like a car purely because it’s electric. They have always viewed compiler based memory safety techniques as guard rails for novices. In their view, good bowlers don’t need guard rails at the bowling alley. It’s a massive massive clash of cultures and the rust folks come into the discussion with an assumption that C devs would leap with joy at the chance to automate memory management. Rust and C are complete opposites, but rust programmers seem to assume that just because rust is fast C programmers will love it.
That might be true but it’s not what happened at that specific conference. I beg you watch the clip to see what happened. Also fuck programmers with the attitude you describe. It’s been proven wrong over and over again with so many C memory safety vulnerabilities.
I saw the clip previously. The rust guys are absolutely assuming that the C guys would go for something because (a) the compiler guarantees it’s memory safe (b) the semantics would be encoded in the type system. They demonstrate this using rust terminology and algebraic data types. Algebraic data types are the bees knees, (but not with that syntax and clumsiness), and compiler guarantees are the bees knees, but that’s not how a C programmer who’s middle aged sees the world, it just isn’t. Your typical middle aged C programmer grew up telling pascal programmers that automatic array bounds checking is for wimps and real men use pointer arithmetic and their programs run five times as fast. They were always right because their programs did really run significantly faster, but now rust comes along and its fast and safe. Why wouldn’t C programmers like it? Because the speed was the excuse and the lack of guardrails was the real reason they liked C.
I said it’s a massive culture clash that the rust folks didn’t realise they were having because they just assume that “memory safe” wins people round, whereas C folks value their freedom from automatic compiler-based safety, and here you are, sounding like a rust person, saying it isn’t a culture clash at all and that the rust folks are right about memory safety and the C folks are just being irresponsible.
They aren’t asking C devs to write Rust code, which is what the guy being a heckler was claiming. Why don’t they want to right Rust? For exactly the reasons you describe. The thing is though that’s not currently being asked of them, all they actually want is the documentation to create that code themselves.
You really don’t have to explain any of the culture clash to me lol. I’ve written both C/C++ and Rust. My C and C++ coding skills are demonstrably better (or at least used to be, it’s been a while) than my Rust skills. Why? Because of how complex those guardrails are. The difference is I have the self awareness to know that my lack of Rust skills doesn’t mean that the language is bad, or that C is a safe language to use. Rust tutorials could be improved. Perhaps an easier to use language like Zig might be more useful for some people. I feel like it’s a good compromise between safety and ease of use. Rust though is still incredibly progressive for the industry, and will improve systems security, maintainability and reliability going forward if only people would stop getting in the way.
TL;DR: Vast culture clash that rust guys didn’t perceive and C guys hated and false assertion that “you don’t need to learn rust” based on inexplicably naive lack of understanding that maintenance might be necessary.
If someone builds a rust api on top of your C code inside your project, you have exactly five choices: (1) preserve the assumptions the rust code is making (2) only change your code if you have a rust expert to collaborate with handy (3) edit the rust code yourself (4) break the rust assumptions leading to hard to find bugs (5) break the build. The C guys hated all five of those options, and the rust guys told them they didn’t need to worry their pretty little heads about it. ON, they weren’t as dismissive as that, but they either didn’t understand those as issues or didn’t care about them or dismissed them.
The rust guys were asking the C guys to tell them the semantics so that they could fix the type signatures for their rust functions and the C guys were reluctant to do that because they wanted to be able to change the semantics of that turned out to be useful to them. They didn’t want to commit so something that was documented in a way they weren’t familiar with because they felt that even if they wanted to, they couldn’t ensure their code was compliant with this specification going forward because they didn’t understand the rust type signature fully. (They got hung up on the self argument and launched a rant against OOP.)
The rust guys knew instinctively that the Result return type meant that the operation could fail and could tell from the two arguments to that both in what ways it could fail and every kind of answer it could produce if it succeeded, but the C guys found almost none of that obvious. This was for just one function in the rust API, but it also radically changed the way of doing it. This one rust call replaced the whole algorithms of ask, check answer, if none, check this and that, otherwise do this blah blah blah. The C guys are used to keeping everything lean and simple with a single purpose and were being asked to think of a while collection of procedural knowledge and edge cases with a handle everything monolith. But they were audibly reluctant to commit to that being all the edge cases because they don’t think of all of those tests as one thing and instinctively wouldn’t write something that checks for all of the edge cases because (a) in a lot of circumstances the code they’re writing only needs to know that there was a problem and will give up quickly and move on and (b) they want to be able to freely choose to add other edge cases in the future like they normally do without having to worry about the rust code breaking.
They weren’t complaining that they were being asked to write rust, they were complaining that they didn’t want to learn rust, and they were complaining this because they could see that to preserve all the rust API type signatures they would have to understand them, the expectations around them and memory safety principles, so that a rust programmer in the future wouldn’t have to change the rust type signature.
The rust guys would have gained a lot more traction by just asking the C guys to keep a bunch of comments up to date detailing the semantics and error checking procedures, and promising to edit their rust API if the C code changes, but I suspect they didn’t ask for that because they know that no guarantees come from a comment and they want to be sure that the rust code works across all the possible scenarios and in rust culture, that is always documented in the type system where it can be enforced.
The rust guys spoke like it was self evident that having a monolithic API with a bunch of stuff guaranteed by the rust compiler was best, but seem not to have realised that this is a massive culture clash because the C guys come from a culture of rejecting the idea of compiler guarantees anyway (because they have long had confidence in their ability to hand optimize their code to be faster than some prescriptive compiler’s output and look down on people who choose to have the guardrails up).
They felt like they were being asked to help write an interface definition in a monolithic style that they have always rejected, to achieve goals that they have long resisted, in a language that they find alien, with no guarantees for them that the rust guys were going to stick around to agree and implement the rust changes necessary if they changed the C code, and with no confidence that they understood what would count as a breaking change at the rust level.
This perceived straightjacket made them particularly cross. They complained about the inability to change their C code and its semantics and the need to learn enough rust to understand quickly what not to change, but they didn’t want to not change things and would need to edit the rust API at the same time as editing the C code if they didn’t want the rust build to break, and then there would be even more downstream changes from that, so realistically they would need not only to be able to understand the rust type signatures, they would need to be able to edit both the type signatures and the functions themselves, and basically maintain all the downstream rust, and they would want to be sure they were writing efficient rust, well aware that it took them decades to get to the level of extreme efficiency they write in pure C, a much simpler language.
The rust guys said “Just tell us what your code means so we can write our type signatures”, but the C guys didn’t want to help create for themselves a prison whose walls were of a strange and intricate design they found hard to perceive, made out of materials they didn’t have experience working with. They felt like the first guys were asking them how all the doors, windows, chimneys, air vents etc of the house that they built by hand would ever be used, so they could encase it in a stainless steel shell and make it part of a giant steel city. The C guys said “but I might want to build an extension or a wider garage!” They claimed that the C guys didn’t have to learn how to weld or manufacture steel sheets, and that their house would be much safer, but for some reason this didn’t win the C guys round to the plan, and there’s a bunch of people online calling the C guys tech luddites for not liking the whole thing and saying that they were incorrect that they needed to learn rust just because the rust guys made that claim, but that claim is actually completely incorrect unless you think that it’s OK to stop the project compiling with your pull request or you think that changes to the C code should be banned wherever a rust API is built on top of it.
The rust guys would have gained a lot more traction by just asking the C guys to keep a bunch of comments up to date detailing the semantics and error checking procedures, and promising to edit their rust API if the C code changes, but I suspect they didn’t ask for that because they know that no guarantees come from a comment and they want to be sure that the rust code works across all the possible scenarios and in rust culture, that is always documented in the type system where it can be enforced.
I could be being daft but I thought this is more or less what the Rust guys were asking for. Tell us the current symantics of the system, and if it changes in future let us know what the new semantics are and we will fix the Rust code accordingly.
I do understand what you mean though about enforcing restrictions on what the C guys can do without breaking the Rust code. I think you run into situations wherever two languages meet. The way most projects handle this is the upstream releases a new version, or a release candidate of a new version with their breaking changes documented and then downstream updates their stuff accordingly when they get time. Obviously this is one project, but I imagine it’s possible for the C guys to update stuff in a pull request and then drop an email in LKML to the Rust guys so they know stuff needs fixing. None of this seems that hard to me.
Ultimately though everything here is Linus decision. Either your in or your out. If Linus says yes to Rust doing whatever then that’s what’s going to happen. Likewise if he says no, then it’s not going to happen that way. Until he weighs in no one can really say how this will end.
Personally though I disagree with the C guys. Safety features are important and should be used where it is practical to do so. Until now C has had the justification that it’s still the fastest language and by a significant margin. Now a somewhat safe language like Rust exists with the same speed and capabilities I don’t think we can afford to continue ignoring safety for the sake of a few bruised egos. If this was a proper industry like aviation safety would always come first, and if that means adopting new technologies and forcing people to adapt. I can understand if C devs have a hard time adapting, I don’t expect it to happen overnight. The expectation though should be they should learn some Rust eventually, even if it’s just enough to know the type signatures and what not that they might break with their changes to C code. Kernel devs are supposed to be some of the smartest computer people out there. If they can’t learn even that small amount of another language then should they really still be kernel developers?
One detail about Rust in the kernel that often gets overlooked: the Linux kernel supports arches to which Rust has never been ported. Most of these are marginal (hppa, alpha, m68k—itanium was also on this list), but there are people out there who still use them and may be concerned about their future. As long as Rust remains in device drivers only this isn’t a major issue, but if it penetrates further into the kernel, these arches will have to be desupported.
(Gentoo has a special profile “feature” called “wd40” for these arches, which is how I was aware of their lack of Rust support. It’s interesting to look at the number and types of packages it masks. Lotta python there, and it looks like gnome is effectively a no-go.)
It seems like gcc rust would pretty much fix that issue, since soon gcc will be able to compile rust for any architecture gcc supports.
Assuming that it works out, yes, this might fix the problem. On the other hand, I remember gcj, which kind of quietly vanished after a while, so I prefer to reserve judgement until gcc’s Rust implementation is ready for production use.
The rust community is its own worst enemy. The political infighting and constant compulsion to shit on other languages is a turn off to many, and there are plenty of applications where memory safety is not the highest priority.
In this case, the issue is really the C kernel devs, not the Rust devs. Some are not open to new ideas at all. Take a look at the conference video he linked to for example: https://youtu.be/WiPp9YEBV0Q?t=1529. He clearly states that he’s not trying to make the C devs learn Rust.
My comment was speaking in a general context.
I’ve seen the video and I agree with Ted. Anyone with experience understands creep, and although Wedson denies it it’s exactly where they’re headed. Ted and others are right to voice these concerns and attempt to set very clear expectations for the rust developers.
They took on the task knowing it was experimental, would be difficult, and that they would be second class citizens - you don’t get to agree to the terms and then complain about them later.
