Maintenance of core parts of the embedded / Cortex-M ecosystem

[japaric]

A.K.A. @japaric is a bottleneck for development :-)

There are critical components of the embedded / Cortex-M ecosystem that are currently being
maintained by me. Unfortunately I don't have the bandwidth to keep up with everything so issues and
PRs have been piling up in some repos.

This issue is a call for maintainers (if you want to help leave a comment!) and a place to discuss
how to organize maintainership duties.

First, these are the crates that should be maintained by more people:

• embedded-hal

Abstractions for things like Serial, SPI, I2C, etc. to build generic drivers that work with AVR, ARM
Cortex-M, MSP430, RISCV, etc. microcontrollers.

There's a bunch to do here as some abstractions are missing and async support is currently
nonexistent. Most of the work to do here is ensuring that the established process for adding new
traits is followed and that discussions move towards consensus.

Quite a few people have volunteered to help and are now collaborators so I think this one is well
covered for now.

• svd2rust

Tool used to transform CMSIS SVD files (XML files that describe the hardware registers of a
device) into code for low level manipulation of hardware registers. Although the SVD spec was
created by ARM svd2rust is being used for both ARM Cortex-M and MSP430 devices. In the later case
TI provides DSLite files (their version of SVD files) and those are converted to SVD files using the
dslite2svd tool.

It's expected that svd2rust will also grow AVR support -- Atmel has its own description file
format: atdf -- using the atdf2svd approach. And there are XSVD (JSON version of SVD files) files
for RISCV -- I don't think they are official though? -- so svd2rust could also grow support for
RISCV via a xsvd2svd tool.

Unfortunately svd2rust has grown into an unmaintainable mess. The test suite takes a lot of time
to run (it takes longer than the CI limit) and it only checks if the generated code still compiles
-- it provides no feedback about whether the generated code exposes a certain interface or not. The
underlying parser lacks error handling and simply panics on errors. Also, svd2rust doesn't give
you helpful error messages when your SVD file has some problem -- this is an important feature to
have when manually editing SVD files.

So both svd2rust and the underlying svd parser need to be refactor (read: rewritten) to make
them more modular and testable. I think the approach to take here is a refactor like the one
proposed in rust-embedded/svd#37. Basically we should be able to test code generation on parts of a SVD
file. For example that <enumeratedValues>..</enumeratedValues> produces this enum Foo { .. }
and this API impl Foo { fn is_variant(&self) -> bool { .. } }, etc. This will let us write unit
tests that can replace the huge existing tests that only check for compilation errors. And while you
are at it you can add error handling to the parser and nice error messages to svd2rust.

This is a huge task but it will make it easier to continue the development of svd2rust. There a
bunch of open issues related to SVD files not being supported; the refactor should make it easier to
fix those issues.

If you want to tackle this let me know in the comments!

---

Then we have core cortex-m crates.

• cortex-m

This repo provides access to Cortex-M specific features like access to registers, safe wrappers around
assembly instructions and access to core peripherals.

One old issue in this repo is providing a svd2rust generated API to access the core peripherals.
The current API has been manually written and is not as type safe as a svd2rust one would be.
Fixing this requires writing or putting together an SVD file that contains only core peripherals,
running svd2rust over it and committing the code to the cortex-m repo.

• cortex-m-rt

This crate takes care of communicating the memory layout of a program to the linker and providing
implementations of language items that are required to build a no_std program.

There are a few missing features in this crate: support for placing functions / interrupt handlers /
the whole program in RAM, relocation of the vector table, etc. that need a design. It may be useful
to coordinate with the representatives of the other embedded targets to see if we can provide a
similar user interface for these features.

• cortex-m-quickstart

This crate is used as a Cargo project template for writing Cortex-M applications.

Every now and then a user figures a new way to misfollow the instructions and encounters a new error
message. These errors should be documented in the troubleshooting guide.

---

Regarding organization I think it may make sense to move these core components under the
rust-embedded org and then create, say, a cortex-m team in charge of maintaining the cortex-m
crates. As the AVR, MSP430 and RISCV communities grow they may also want to move core components
into the rust-embedded org and create their own maintainer teams. Thoughts on this?

cc @ryankurte @Emilgardis might be interested in picking up svd2rust development
cc @dylanmckay (AVR) @pftbest (MSP430) @dvc94ch (RISCV) see last paragraph, also svd2rust support

[ryankurte]

I like the idea of moving components into rust-embedded (and increasing the bus factor with that org 😂).

I'd like to see a code of conduct (something line the contributor covenant and the contacts and policy to support it, as well as contribution guidelines (ie. how to pr, policy on breaking changes and versioning, adoption of projects by rust-embedded etc.) at the org level.

Also willing to pick up some of rust2svd/svd-rs(/embedded-hal) development/support/maintenance, and to update japaric/svd#37 to get it merged as a first step in the refactor if that's useful.

[dylanmckay]

@japaric

It's expected that svd2rust will also grow AVR support -- Atmel has its own description file
format: atdf -- using the atdf2svd approach. And there are XSVD (JSON version of SVD files) files
for RISCV -- I don't think they are official though? -- so svd2rust could also grow support for
RISCV via a xsvd2svd tool.

I have already written a library to parse ATDF files into data structures, which should be fairly easy to plug in.

It is [almost being] used in the ruduino library to implement a trait-based AVR-specific HAL. It is being used in another library that defines all registers/IOs for every AVR device.

[dylanmckay]

I like the idea of moving components into rust-embedded (and increasing the bus factor with that org 😂).

Agree strongly. I have mixed feelings on whether this should be done for AVR before it actually gets merged into upstream Rust though.

I've tried to be quite careful to manage expectations with the AVR fork - there's been quite a bit of interest, but the experience is still quite buggy. I think that moving some of the AVR stuff to the rust-embedded org might give off the impression that the AVR experience is well supported, where in fact things such as having optimisations disabled on a nontrivial project make it quite likely to have an LLVM assertion error.

I think there's a lot of value in bringing the embedded projects into one organisation. It'd definitely help the embedded experience become a lot more seamless, especially with strong collaboration on things such as HALs.

[hannobraun]

@japaric I'm happy to help out here and there with the cortex-m crates, but I don't have the bandwidth to drive any major initiatives.

Fixing this requires writing or putting together an SVD file that contains only core peripherals,
running svd2rust over it and committing the code to the cortex-m repo.

I did some digging. I already knew about these incomplete SVD files, so at least there must have been some interest by ARM at some point.

I found this issue, specifically this comment, which states that core SVD files exist now, referring to some Keil pack. I found this download page, and I believe the pack that the comment refers to is "ARM V2M-MPS2 Board Support PACK for Cortex-M System Design Kit" under the "Keil" heading.

I downloaded the file (despite the name it's just a zip file; add ".zip" to the name and you should be able to open it), and yes, it contains SVD files for all the Cortex-Ms. I don't know if they're any good, and I don't know what the EULA says about using them, but I guess at the very least it makes sense to check them out.

Regarding organization I think it may make sense to move these core components under the
rust-embedded org and then create, say, a cortex-m team in charge of maintaining the cortex-m
crates.

👍

[docbrown]

@hannobraun It looks like the SVDs in that pack are for some kind of abstract Cortex-M device. They include peripherals (timer, GPIO, serial engine, etc) that aren't common to all Cortex-M designs. Maybe they're just copied directly from the Cortex-M System Design Kit?

@japaric If a SVD needs to be written from scratch, I can try to put one together directly from the Technical Reference Manuals so that there's no problem with licensing. Unless someone is already doing that.

[hannobraun]

@docbrown Ah, too bad. And I'm guessing we're not allowed to modify it (though I haven't read the EULA).

[dvc94ch]

So moved all riscv crates to a rust-riscv org. Should help find riscv crates, contributing etc. If someone wants to collaborate I can add you to the org.
There is also a riscv-crates repo that creates a cargo workspace to make hacking on riscv crates easier - rust and llvm forks are updated.
Some of these crates could be moved to rust-embedded once they're more mature.

[docbrown]

@hannobraun Yeah, it's kind of a minefield. There's the EULA (which appears to only reference example code and drivers, but IANAL), a 3-clause BSD(?) license from ARM on the SVDs themselves, and then another license from ARM applied to the code generated from the SVD (the licenseText element). And if the SVDs were just copied from the reference design in the Cortex-M System Design Kit, then there's likely a EULA for that, too (which we can assume Keil is in compliance with, but the kit doesn't appear to be publicly available to verify that). Writing an SVD from scratch seems like the safest option, IMO.

[hannobraun]

@docbrown

Writing an SVD from scratch seems like the safest option, IMO.

I'd try to contact ARM first, to try and clarify the license. That might not come to much, but it probably won't hurt either. But if you're willing to write an SVD from scratch, I'm not trying to stop you! Just offering suggestions.

[Emilgardis]

I'd be happy to help maintaining svd2rust. I'm also glad that were taking this direction because the pressure and responsibility has been something I've noticed to be far to big for a one-man team.

[japaric]

@Emilgardis @ryankurte

Thanks for volunteering!

I have just sent you collaborator invites for both japaric/svd and japaric/svd2rust. Those two
are in a worse state than the cortex-m repos and I don't want to block development on those until we
figure out an org structure.

@ryankurte

I'd like to see a code of conduct (..) as well as contribution guidelines (..) at the org level.

Great idea! Any suggestion on how to make it visible at the org level? I don't think you can have
pinned repos at the org level. Though it seems you can have some description text at
https://github.com/$ORG

@dylanmckay

I have already written a library to parse ATDF files into data structures

Awesome! I know that @kunerd and ecstaticmorse_ (IRC nickname) are interested in svd2rust support
for AVR and/or a atdf2svd tool. You should all synchronize efforts.

[scowcron]

@dylanmckay @japaric re: ATDF

atdf2rust really would be nice to have, even for micros in the ATSAM families (Cortex-M). Atmel's SVD files are non-compliant and don't play nicely with svd2rust. I'd love to be able to generate code for those without have to hack svd2rust to ignore problems in the SVDs.

The other significant pain point I had working with svd2rust was lack of register group support, but I see that there's progress on pulling in svd v0.6.0, so I won't harp too much on that.

[kunerd]

@dylanmckay @japaric: ATDF

@ecstatic-morse created this atdf parser repo, which also can parse .atdf files into svd2rust structs, if I remember correctly. From my point of view it would make sense to separate the code generation part from the description file parsing part, so that for example svd2rust and atdf2rust tools can be created by using the same code generation base, either as a lib or by adding different executables to svd2rust. I think I can offer some time to support on this.

@dylanmckay: btw - thanks for the great work on the avr-rust fork

[jamesmunns]

@japaric I would be interested in also helping to maintain at least svd2rust. I have now spend 1/2 a week with it (due to the cluster issue), and feel more comfortable with it (but I am still far from being an expert).

[japaric]

@dylanmckay @dvc94ch Since you have org for core AVR / RISCV crates I would suggest you add a link to your orgs to the README of this repo. Also if you tag issues with the help-wanted label then when someone visits the org page they'll see how many help-wanted issues each repo under the org has.

@jamesmunns Awesome! Thanks. Sent you an invite.

@kunerd if the ATDF parser produces a svd2rust::Device struct then I think it would make sense to have an atdf2rust tool in the svd2rust repo. Could you open an issue in the svd2rust repo to discuss this?

[ryankurte]

@japaric org level code of conduct / contributor guidelines / how to contact are probably good candidates for rust-embedded.org? Then the org description can just be a link there and the main page already has a link back.

[hannobraun]

I don't think you can have pinned repos at the org level.

You can have pinned repositories for orgs! Might require some specific permission though, I don't know. See screenshot:

[japaric]

Sorry this has taken so long.

This my initial proposal for re-organizing the existing embedded repos under the rust-embedded org.
Feedback is of course welcome!

Teams, collaborators and repos

Collaborators are split in two groups:

• Repository specific collaborators or just "collaborators". They have access to a specific repo,
  and they help triaging issues, and reviewing and merging PRs.

• Team members. Teams oversee the coherent development of the crates / tools under their purview.
  They share the same set of tasks with collaborators but have a few more executive / administrative
  tasks: they set the overall development direction by approving RFCs, and they are also in charge
  of managing crates published on crates.io.

This is the tentative distribution of teams and repos. The initial membership reflects more or less
things as they are right now.

The names after the tilde (~) indicate collaborators to that repo.

• embedded-linux team

  • members: @posborne, @nastevens
  • gpio-utils
  • i2cdev
  • linux-embedded-hal ~ @japaric
  • rust-sysfs-gpio
  • rust-sysfs-pwm
  • spidev

• tools team

  • members: @japaric
  • cargo-binutils?
    • or equivalent if/when Ship more LLVM tools with the Rust toolchain rust-lang/rust#49584 gets approved
  • itm?
  • svd2rust ~ @ryankurte @Emilgardis @jamesmunns
  • svd ~ @ryankurte @Emilgardis

• cortex-m team

  • members: @japaric
  • immediate goal: make everything work on stable
  • cortex-m-quickstart
  • cortex-m-rt
  • cortex-m-semihosting
  • cortex-m

• resources team

  • members: @japaric
  • awesome-embedded-rust ~ @RandomInsano @berkus @therealprof @rahul-thakoor
  • discovery?
  • Should we move the book and embedonomicon into the rust-embedded org?

• hal / common dependencies team

  • members: @japaric
  • bare-metal
  • embedded-hal ~ @hannobraun @ilya-epifanov @thejpster @therealprof
  • nb

Code of conduct

All participants in rust-embedded projects will adhere to the Contributor Covenant Code of Conduct
v1.4.

The code of conduct (CoC) will be linked from the rust-embedded org landing
page as a pinned repository. A copy of the CoC will be included
in each repository under the rust-embedded org (CODE-CONDUCT.md), and a note about the CoC will be
included in the repository README (see failure for an example).

Teams and collaborators will enforce the code of conduct in the repositories they oversee and on
the #rust-embedded IRC channel.

---

Thoughts?

Also, we need more people on the teams so this is your chance to volunteer to a specific team!

Finally, we need an e-mail address, or some other private channel, to receive reports related to
violations of the CoC. @posborne @nastevens would be possible to set a [email protected]
e-mail alias?

[rahul-thakoor]

Hello! I would like to help more. However, I am new to the Rust programming language and would definitely love to learn and contribute more, if I get assistance from this wonderful community!
I am very interested in contributing to embedded-linux ecosystem.

Some thoughts:

1. Are there plans to implement the serial interface for linux-embedded-hal? Could be interesting to interact with modules such as ESP8266 WiFi SoC or WISOL Sigfox module via UART.

2. As of Linux 4.8 the new gpiochip char devices provides a new API and sysfs gpio is supposedly deprecated. There is already a C library, libgpiod, for interacting with the linux GPIO
   character device with C++ and Python bindings in the works. It would be great to have a Rust implementation.

3. I can also help with French and Chinese(Simplified)() translation of awesome-embedded-rust or other resources if needed.

4. Regarding maintaining the several projects, perhaps derek can help?

Thanks

[therealprof]

@japaric I'd like to get involved in the tools team and would like to add two important topics to the list which we should investigate: flashing and debugging, potentially also non-ITM logging (i.e. serial monitor support). I'd be definitely interested in lending a hand with cargo-binutils.

Potentially I'd also love to help with cortex-m though I currently have no idea what to contribute there.

[ghost]

Hello together.
I'm also new to Rust but in software engineer business since 20 years now. Currently I'm not so happy with the organization of the different SVD files for the STM family of MCUs. @adamgreig is working on a promising approach to keep the SVDs from STM in one place and collect the needed patches there.
Another point I'll like to help out are the currently spread implementations of the HAL driver implementations all across over different crates.
Best, Ingo

[musitdev]

Hello,
I can also give a hand and I've proposed to help in the refactoring of SVD. I develop with Rust since end 2012 and I have a few experience in embedded ARM Cortex (I work on a STM32f7 card for audio applications). I can help with french translation too.

[posborne]

As of Linux 4.8 the new gpiochip char devices provides a new API and sysfs gpio is supposedly deprecated. There is already a C library, libgpiod, for interacting with the linux GPIO
character device with C++ and Python bindings in the works. It would be great to have a Rust implementation.

Hi @rahul-thakoor. I'm the author of the sysfs gpio crate. I have most of the core bits of a new crate based around the GPIO character device here: https://github.com/posborne/rust-gpio-cdev/tree/first-pass. I need to make some final decisions on APIs and document but an initial version should be on crates.io in a few weeks at the latest.

Once stabilized, this crate will likely move over to the rust-embedded org (which I created awhile back). I considered wrapping libgpiod but decided it made more sense to just interface with the kernel directly -- the API is very similar where it makes sense, but I have also strayed some in order to provide better safety/ergonomics using Rust's ownership system.

Both will need to be supported for some period of time, but I think it makes sense to start pushing people toward the cdev implementation if they know they will only be targeting 4.4+.

[andre-richter]

I would be happy to help on the resources team. Most probably I can contribute by reviewing material.
Regarding resources, I am also in the process of porting tutorials for bare metal programming on RPi3 (aarch64, cortex-a) to Rust.
Currently, the code is more or less literally translated from C, with some minor abstractions here and there. But it is definitely lacking a layered approach like there is with cortex-m, cortex-m-rt, cortex-m-quickstart, with all its safety abstractions etc.
Therefore, I wonder if there are any plans for cortex-a in the not too distant future, or is that not on the radar?

[berkus]

Of these projects I'm probably able to help with embedded-hal somewhat, so will be happy to do it when i can, but alas won't take maintainership.

[pftbest]

Hello, @japaric! Can you please give me access to https://github.com/japaric/msp430-quickstart so I can merge japaric/msp430-quickstart#3

[rudihorn]

I'd be happy to help manage PR's etc. with the https://github.com/japaric/stm32f103xx-hal crate.

[korken89]

Hi @japaric, I have been working for a long time with the core Cortex-M parts for the C++ version of RTFM, where I am now (~slowly) transitioning to Rust.
If you want help with the core cortex-m* crates, give me a ping.

[japaric]

I finally (!) wrote an RFC for this: #136. We'll discuss this RFC in the next embedded WG meeting (#130) which is in ~12 hours. If you can't make it to the IRC meeting but can comment on the RFC before the meeting please do! Even if everyone who attends the meeting is on board with the RFC we'll leave the RFC open for at least a week before we officially approve it. So leave comments over there if you have suggestions or concerns.