I'm back working full time in C, I'll provide two examples (links in my profile so this comment doesn't become too overly promotional).

SoloKeys is an open security key. The 1st version of its firmware was in C, and while building the v2 hardware we decided to also rebuild the firmware in Rust.

- Could have we started in Rust? I'm not sure because embedded Rust was very immature when we started SoloKeys.

- Was it good to move to Rust? Probably, some of the bugs we had in C were the classical overflows that in theory shouldn't happen in Rust. The future will tell.

In summary: SoloKeys is my personal "classical example" of starting in C for various reasons and upgrading to Rust to improve safety. Let's look at the opposite example. :)

Firedancer is a reimplementation of Solana validator in C. The original validator, now called Agave, is in Rust. You can think of: 1) a Solana validator as a single-machine database, and 2) the Solana blockchain as a distributed system where all nodes run the db and independently (try to) run the same transactions on some initial state to (hopefully) produce the same resulting state. By this definition our goal is to make a second implementation of a db node in a different language, we could have chosen any language.

C was chosen for performance, but it's NOT the superficial "C is faster than rust", in fact the goal is to make the Rust implementation reach comparable performance. The difference is in the mental model, where we look at how data flows in the (single machine) system and try to optimize it at all levels. We had C code already for other projects that we repurposed it for Firedancer.

The experience of porting Rust to C has its own complexities, I'm going to list a few. Note that these are not critics to the language nor to the developers. If Solana started in C and we were porting it to Rust we would find some of the same issues and probably more. I'll try to highlight my learnings in case someone else has to port a project from language A to B -- granted our case might be a bit special because the goal is to run both in parallel and achieve consensus while often times B is going to replace A.

- In C we don't use malloc. Pretty much everything is pre-allocated and thus has known bounds. In Rust we see a lot of Vec or other dynamic structures whose bounds are not clear. In some cases these led to discovering DoS vulnerabilities. Note that this is not C, this is our mental model that translates to using C without mallocs.

Learning 1: always know your (memory) bounds.

- Rust `?` operator is very nice for Rust, very bad for someone who's trying to read the code, especially if they (i.e. me) have to match all error cases. It's very easy to miss some. Together with dyn err, it can easily make error propagation kind of random. We're spending a considerable amount of dev time and fuzzing resources not just to avoid issues with C, but to make sure we always return the same value/error in C as in Rust.

Learning 2: specify error behavior (e.g. expected errors, precedence of errors), not just the correct flow.

- Tests. This is my pet peeve, I apologize in advance. Many unit tests generate their input and then test a specific behavior. I work primarily in cryptography so let me make an example in this area: we want to test a signature verification. Bad test IMO: create a signature, then check that verify is successful. Good test IMO: there's a bunch of bytes (that we can create once by running the signature code, that's totally fine), check that the verify is successful. Think to what happens if you change the signature scheme. The first test is still passing, the second will highlight that there was a change and requires new bytes. I have countless examples of breaking changes in the Rust project that required major rewrites in C, that no one in the "Rust team" even thought they could cause any issue because unit tests are not highlighting any problem.

Learning 3: always have at least 1 test that's just a bunch of constant bytes in input.