Improve support for Zephyr RTOS applications #516

kartben · 2023-02-08T14:18:39Z

The zephyr-esp32 builder is already a great start in allowing to run Zephyr apps from Wokwi.
It would be great to look into generalizing/improving support by allowing the users to configure things such as:

providing a prj.conf
providing DT overlays
supporting more than just basic esp32, but also potentially rp2040, esp32-s2, etc.

I am around to chat more around this, and I will provide help as much as I can!

The text was updated successfully, but these errors were encountered:

urish · 2023-02-08T20:39:16Z

Hi Benjamin! Thanks for offering your help here. Zephyr apps is definitely something I'd love to see running in Wokwi.

Right now, we have three ways to use Wokwi:

Online, though the web interface - very popular, but doesn't like long build times or complex build environments
As a Visual Studio Code extension - in public beta, and you're already familiar with it
As a command-line tool for CI use cases - in private beta, but we plan to open it publicly later this year

Which use cases do you have in mind?

kartben · 2023-02-12T14:53:53Z

I would love to make sure #1 gets improved, at least to the point where it allows people to get a better understanding of how they can tinker with things like device tree overlays, and project configuration. Point taken though regarding long build time possibly being a problem.
I need to play more with the VS Code extension to get a sense of how the current experience is. It might be that it just works perfectly already :-)

Cheers!

urish · 2023-02-12T19:56:25Z

How complex is it to set up a Zephyr ESP32 "Hello world" project in VS Code?

kartben · 2023-02-13T18:32:21Z

How complex is it to set up a Zephyr ESP32 "Hello world" project in VS Code?

Not too complex, apparently :)

urish · 2023-02-13T18:33:46Z

Damn, you are fast!

urish · 2023-03-23T22:56:57Z

Is there a repo with this setup?

urish · 2023-05-04T20:06:21Z

@kartben heads up on wokwi-cli, which will allow running any project you set up for VS Code in CI. Want to give it a go?

kartben · 2023-05-04T20:10:35Z

@kartben heads up on wokwi-cli, which will allow running any project you set up for VS Code in CI. Want to give it a go?

Yes, please! Is everything I'd need available at the linked URL?
And I need to do a proper write-up of some of my experiments to date with the VS Code extension.

urish · 2023-05-04T20:15:52Z

Yes, please! Is everything I'd need available at the linked URL?

It should - but this is pretty early, so you'll probably find something I forgot. Ping me when you do.

And I need to do a proper write-up of some of my experiments to date with the VS Code extension.

Yes, please!

bmeisels · 2023-05-06T21:52:42Z

wokwi-cli is really easy to use.
I got it running here https://github.com/bmeisels/wokwi-cli-github-actions-example and @urish even added a dedicated action
https://github.com/wokwi/wokwi-ci-action.

urish · 2023-08-04T13:19:13Z

@kartben which app is this? can you please share the repo?

kartben · 2023-08-12T17:12:45Z

@kartben which app is this? can you please share the repo?

I don't remember 😭 I really need to put together a quick README somewhere...

urish · 2023-08-14T21:10:02Z

People keep asking about Zephyr + Wokwi. We need your README :)

kartben · 2023-10-03T14:05:47Z

@kartben which app is this? can you please share the repo?

@urish
ok so I am finally spending some time digging this up.
Here's a working, self-contained, project: https://github.com/kartben/wokwi-zephyr-projects/tree/master/esp32c3-lvgl-with-mpu6050

Things are unfortunately really slow, and it's not clear why. I vaguely remember discussions about performance issues depending on how sleep is implemented, and also the ILI9341 simulation potentially being slow.
I will try a similar sample with the RP2040, just in case I have better luck. In the mean time, your input on how to potentially improve the performance would be most welcome!

urish · 2023-10-03T14:24:35Z

Oh yeah!

In general, transferring huge amount of data over SPI is slow. There's an experimental way to short-circuit the low level SPI stuff and speed things up. Also, DMA can help to some extent - do you know of the underlying code uses DMA?

kartben · 2023-10-03T14:34:46Z

Oh yeah!

w00t!

In general, transferring huge amount of data over SPI is slow. There's an experimental way to short-circuit the low level SPI stuff and speed things up. Also, DMA can help to some extent - do you know of the underlying code uses DMA?

Ya but I think it's more than just SPI, let me try to produce a minimal repro sample

urish · 2023-10-03T14:43:31Z

Meanwhile, I ran some benchmarks using the Wokwi Profiler. Seems like most of the ESP32 time is spent memcpy'ing and doing SPI stuff:

kartben · 2023-10-03T15:00:13Z

Nice (the profiler I mean, not the fact that LVGL is probably a huge performance killer :D). I can't easily run it myself tho, right? From the Web UI it tries to run from source, and I don't think I can launch it from VS Code?

Please see https://github.com/kartben/wokwi-zephyr-projects/tree/master/esp32c3-blinky
It should blink an LED every 1sec, but looks like the timing is off?

...
load:0x403d0000,len:0x2370
entry 0x403ce000
W (280) rtc_init: o_code calibration fail

*** Booting Zephyr OS build v3.5.0-rc1-11-g175560ff7e50 ***
LED state: ON @ 0
LED state: OFF @ 2129
LED state: ON @ 4258
LED state: OFF @ 6387

Could the rtc_init warning be relevant?

kartben · 2023-10-03T15:20:35Z

currently testing with esp32s3 and the timing for blinky seems correct

urish · 2023-10-03T19:11:24Z

Thanks! Yeah, the profiler is not available (yet) in VS Code, but I'm working on it!

Please see https://github.com/kartben/wokwi-zephyr-projects/tree/master/esp32c3-blinky

Good catch, pushed a fix. Can you please test again?

kartben · 2023-10-03T19:46:28Z

Please see https://github.com/kartben/wokwi-zephyr-projects/tree/master/esp32c3-blinky

Good catch, pushed a fix. Can you please test again?

Yay! Much better :) it looks like it speed things up a tiny bit for the lvgl sample, but that might be magical thinking.
I can't seem to get SPI to work for ESP32S2 and S3 (pretty sure I'm just doing something wrong on the Zephyr side) but before I invest more time in trying to debug this, is there a chance that Xtensa simulation will be faster than RISC-V's?

urish · 2023-10-03T20:14:09Z

Great!

It's hard to tell whether Xtensa is going to be faster - it depends on the compiler and esp-idf. For instance, it might use DMA on the S2 (or original ESP32), and that could considerably speed things up.

When you pause the simulation with S2/S3, do you see the SPI pins configured correctly?

urish · 2023-10-03T20:19:34Z

Another update: got rid of the "rtc_init: o_code calibration fail" issue, so now the program should start much faster for C3 and S3 (it won't keep polling the RTC until it time outs).

kartben · 2023-10-04T08:29:34Z

Another update: got rid of the "rtc_init: o_code calibration fail" issue, so now the program should start much faster for C3 and S3 (it won't keep polling the RTC until it time outs).

very nice -- much faster indeed!

kartben · 2023-10-04T08:33:45Z

When you pause the simulation with S2/S3, do you see the SPI pins configured correctly?

ooh, what a neat feature :)
So I am pretty sure things are wired up properly, and it looks like the pins are also OK, see below.
I've put the non-working sample here https://github.com/kartben/wokwi-zephyr-projects/tree/master/esp32s3-lvgl-with-mpu6050, in case you can maybe spot anything obvious when tracing.
Note that Zephyr+I2C+MPU6050 seems to work just fine for this same diagram, it's SPI that seems busted.
Thanks!!

urish · 2023-10-04T11:35:36Z

Thanks for making it easy to reproduce - I found out it was a bug in the simulator (SPI hardware was not signaling the code it finished transferring data correctly). Uploaded a fix - I can see it now draws on the screen, but then the program (running in the sim) seems to crash with a exception right after it finishes drawing. Is that expected?

Reading symbols from ./esp32s3-lvgl-with-mpu6050/zephyr.elf...
Remote debugging using localhost:3333
warning: multi-threaded target stopped without sending a thread-id, using first non-exited thread
_DoubleExceptionVector () at /Users/kartben/zephyrproject/zephyr/arch/xtensa/core/xtensa-asm2-util.S:444
444     /Users/kartben/zephyrproject/zephyr/arch/xtensa/core/xtensa-asm2-util.S: No such file or directory.
(gdb) bt
#0  _DoubleExceptionVector () at /Users/kartben/zephyrproject/zephyr/arch/xtensa/core/xtensa-asm2-util.S:444
#1  0xffffffff in ?? ()

kartben · 2023-10-04T11:51:56Z

Thanks for making it easy to reproduce - I found out it was a bug in the simulator (SPI hardware was not signaling the code it finished transferring data correctly). Uploaded a fix - I can see it now draws on the screen, [...]

cool, works for me too! Thanks!

but then the program (running in the sim) seems to crash with a exception right after it finishes drawing. Is that expected?

Reading symbols from ./esp32s3-lvgl-with-mpu6050/zephyr.elf...
Remote debugging using localhost:3333
warning: multi-threaded target stopped without sending a thread-id, using first non-exited thread
_DoubleExceptionVector () at /Users/kartben/zephyrproject/zephyr/arch/xtensa/core/xtensa-asm2-util.S:444
444     /Users/kartben/zephyrproject/zephyr/arch/xtensa/core/xtensa-asm2-util.S: No such file or directory.
(gdb) bt
#0  _DoubleExceptionVector () at /Users/kartben/zephyrproject/zephyr/arch/xtensa/core/xtensa-asm2-util.S:444
#1  0xffffffff in ?? ()

mmmh no (granted I don't have actual hardware to test) :)
here's the line if that helps https://github.com/zephyrproject-rtos/zephyr/blob/ecefcd7f87d73bbd7d472399127421639d5bc2c2/arch/xtensa/core/xtensa-asm2-util.S#L444

On my side I can't seem to get it to crash and it rather looks like it's stuck? Simulation rate drops to ~15%, and I can't seem to get a crashdump?

urish · 2023-10-04T11:53:53Z

Yeah, same here - it it stuck in the _DoubleExceptionVector handler. Not sure why simulation speed drops to 15% in this case, but the question is what happened just before it got to _DoubleExceptionVector.

kartben · 2023-10-04T12:08:18Z

Yeah, same here - it it stuck in the _DoubleExceptionVector handler. Not sure why simulation speed drops to 15% in this case, but the question is what happened just before it got to _DoubleExceptionVector.

This time the problem was on my side! Increased stack size of the app and now it works! 🥳
I've updated the binaries accordingly.

kartben · 2023-10-04T19:38:43Z

Alright, I think I spotted the issue with the esp32-c3, some subtle bug with interrupt handling. Pushed a fix. Can you please give try now and report?

Awesome! I can't seem to be able to crash it, no matter how hard I try :) Very cool! (updated the binaries as I had mentioned it had VT100 colors supported but it turns out it wasn't enabled in the version I had pushed to the repo)

Still investigating the esp32-s3 one

❤️

urish · 2023-10-04T20:27:27Z

Looks like the esp32-s3 issue also has to do with interrupts!

urish · 2023-10-04T21:13:11Z

pushed a fix for the esp32-s3 interrupt issue. Cool to see the VT100 colors!

kartben · 2023-10-05T10:43:38Z

pushed a fix for the esp32-s3 interrupt issue. Cool to see the VT100 colors!

Niiice! I can confirm that this seems to solve the problem for me as well 🚀

kartben · 2023-10-05T13:31:09Z

BTW how would I go about trying to refresh some of the work from @beriberikix's with the zephyr builder, i.e. how to test things "locally"? I would like to see if we could get the .overlay and .conf files directly editable from the wokwi web workspace (which right now the first blocker would be that those file extension are not supported, I guess).

urish · 2023-10-05T14:23:11Z

Here's how to test things locally

As a workaround for unsupported file extensions, you could just name them something.c (maybe also something.txt), and create a shell script inside the builder that would rename them before running the build (that's how we prototyped the Rust builders when we first introduced Cargo.toml, which was unsupported back at the time).

Word of caution though - from what I have seen, Zephyr builds tend to take a lot of time. We won't be able to afford running builds that take several minutes in scale, unless we get some sponsorship for that. Also, most users would not have patience to wait several minutes watching a spinner.

beriberikix · 2023-10-05T14:27:14Z

LMK how I can help! Zephyr's official containers are, uh, bloated. The latest CI container is ~12gb. My experiments have yielded an ESP32 container @ 1.3gb and an ESP32S3 @ 1.3gb. They're faster as well.

kartben · 2023-10-05T14:39:29Z

@urish I've just added a few RP2040 samples :) Basic helloworld and blinky work just fine (yay!) but the "shell" sample seems stuck.

urish · 2023-10-05T14:44:17Z

Did you commit the shell sample? Couldn't find it in the repo

kartben · 2023-10-05T14:45:07Z

Did you commit the shell sample? Couldn't find it in the repo

hadn't pushed :) https://github.com/kartben/wokwi-zephyr-projects/tree/master/rpi_pico-shell_module

kartben · 2023-10-05T14:52:29Z

More goodness with ESP32 and actual Wi-Fi connection :) (https://github.com/kartben/wokwi-zephyr-projects/tree/master/esp32s3-wifi)

urish · 2023-10-05T14:55:19Z

Damn you are fast!

I can see that it hangs in the UART RX code in Pi Pico shell, investigating:

(gdb) bt
#0  0x1000a826 in uart_rpi_irq_rx_ready (dev=<optimized out>)
    at /home/kartben/zephyrproject/zephyr/drivers/serial/uart_rpi_pico.c:301
#1  0x100036d8 in uart_irq_rx_ready (dev=0x1000b478 <__device_dts_ord_44>)
    at /home/kartben/zephyrproject/zephyr/include/zephyr/drivers/uart.h:1041
#2  uart_callback (dev=0x1000b478 <__device_dts_ord_44>, user_data=0x1000bd64 <shell_transport_uart_shell_uart>)
    at /home/kartben/zephyrproject/zephyr/subsys/shell/backends/shell_uart.c:174
#3  0x1000a872 in uart_rpi_isr (dev=<optimized out>) at /home/kartben/zephyrproject/zephyr/drivers/serial/uart_rpi_pico.c:353
#4  0x1000597c in _isr_wrapper () at /home/kartben/zephyrproject/zephyr/arch/arm/core/cortex_m/isr_wrapper.S:117
#5  <signal handler called>
#6  arch_cpu_idle () at /home/kartben/zephyrproject/zephyr/arch/arm/core/cortex_m/cpu_idle.S:140
#7  0x1000aa9c in k_cpu_idle () at /home/kartben/zephyrproject/zephyr/include/zephyr/kernel.h:5842
#8  idle (unused1=<optimized out>, unused2=<optimized out>, unused3=<optimized out>)
    at /home/kartben/zephyrproject/zephyr/kernel/idle.c:89
#9  0x1000186c in z_thread_entry (entry=0x1000aa91 <idle>, p1=0x20000b58 <_kernel>, p2=0x0, p3=0x0)
--Type <RET> for more, q to quit, c to continue without paging--
    at /home/kartben/zephyrproject/zephyr/lib/os/thread_entry.c:48
#10 0xaaaaaaaa in ?? ()

implement UARTTXINTR, and also clear UARTRXINTR when the Rx FIFO becomes empty. wokwi/wokwi-features#516

urish · 2023-10-05T16:21:59Z

Pico shell Issue found, pushed a fix. Can you please try now?

kartben · 2023-10-05T16:26:51Z

Pico shell Issue found, pushed a fix. Can you please try now?

Works like a charm! 👏👏

kartben · 2023-10-06T15:32:41Z

@urish Another one for you if you're bored https://github.com/kartben/wokwi-zephyr-projects/tree/master/esp32s3-lvgl-with-encoder :) Seems like it crashes when pressing the button.

urish · 2023-10-06T16:45:16Z

Seems like it crashes when pressing the button

Does it only crash on the S3? Or also in C3?

urish · 2023-10-06T18:05:15Z

A quick workaround is to disable bouncing on the button by setting the "bounce" attribute to "0":

    {
      "type": "wokwi-pushbutton",
      "id": "btn1",
      "top": 140.6,
      "left": 48,
      "attrs": { "color": "green", "bounce": "0" }
    }

Looks like with bouncing, the button is stress-testing the interrupt mechanism :)

urish · 2023-10-06T18:51:24Z

Wow, this was a super tricky one - another subtle bug in interrupt handling, this time in Xtensa. Pushed a fix, so we should now play nicely even with a bouncy button that generates tons of interrupts in quick succession.

kartben · 2023-10-06T18:54:52Z

Wow, this was a super tricky one - another subtle bug in interrupt handling, this time in Xtensa. Pushed a fix, so we should now play nicely even with a bouncy button that generates tons of interrupts in quick succession.

Nice, thanks! I will give it a try and let you know. Hopefully this will unlock rotary encoders too!
Didn't realize that the simulated button would be bouncy tho?

urish · 2023-10-06T19:00:28Z

Didn't realize that the simulated button would be bouncy tho?

Sure, we don't want folks to think their code is working just to run it on real hardware and figure out they didn't account for bouncing, right?

https://docs.wokwi.com/parts/wokwi-pushbutton#bouncing

kartben · 2023-10-06T19:04:06Z

Didn't realize that the simulated button would be bouncy tho?

Sure, we don't want folks to think their code is working just to run it on real hardware and figure out they didn't account for bouncing, right?

https://docs.wokwi.com/parts/wokwi-pushbutton#bouncing

Neat!

kartben · 2023-10-06T20:13:27Z

FWIW the encoder works like a charm too now! https://github.com/kartben/wokwi-zephyr-projects/tree/master/esp32s3-lvgl-with-encoder
Note that it's not using ESP's Pulse Counter (PCNT) IP but instead plain GPIO. PCNT might not even be supported anyway, according to the wiki at least?

urish · 2023-10-07T04:05:42Z

Yes, no PCNT yet

kartben · 2023-10-16T21:10:12Z

@urish A nice one that basically worked out of the box 🥳 https://github.com/kartben/wokwi-zephyr-projects/tree/master/esp32c3-artificial-nose
TinyML with Edge Impulse FTW.

Actually, come to think of it, not everything worked out of the box, it would crash when I tried to use a screen rotation that's not 0 or 180 degrees. Not sure if that's a bug? (here's the relevant code on the Zephyr side https://github.com/zephyrproject-rtos/zephyr/blob/433a85e7bf32ffc77ba194bb903bcca84c44c047/drivers/display/display_ili9xxx.c#L244)

urish · 2023-10-20T14:26:42Z

TinyML with Edge Impulse FTW.

Cool, I wanted to try Edge Impulse for a long time

It would crash when I tried to use a screen rotation that's not 0 or 180 degrees

Crash is in display garbage or actually the program would crash?

kartben · 2023-10-20T14:30:44Z

The latter. I will try to upload a bin as I realize it might be hard for you to test otherwise.

urish · 2023-10-20T14:32:15Z

elf please

implement UARTTXINTR, and also clear UARTRXINTR when the Rx FIFO becomes empty. wokwi/wokwi-features#516 (cherry picked from commit edf8449)

kartben added the enhancement New feature or request label Feb 8, 2023

urish added a commit to wokwi/rp2040js that referenced this issue Oct 5, 2023

fix(uart): zephyr shell broken

edf8449

implement UARTTXINTR, and also clear UARTRXINTR when the Rx FIFO becomes empty. wokwi/wokwi-features#516

c1570 pushed a commit to c1570/rp2040js that referenced this issue Feb 12, 2025

fix(uart): zephyr shell broken

d74444d

implement UARTTXINTR, and also clear UARTRXINTR when the Rx FIFO becomes empty. wokwi/wokwi-features#516 (cherry picked from commit edf8449)

c1570 pushed a commit to c1570/rp2040js that referenced this issue Feb 15, 2025

fix(uart): zephyr shell broken

1d79f30

implement UARTTXINTR, and also clear UARTRXINTR when the Rx FIFO becomes empty. wokwi/wokwi-features#516 (cherry picked from commit edf8449)

Improve support for Zephyr RTOS applications #516

Improve support for Zephyr RTOS applications #516

Comments

kartben commented Feb 8, 2023

urish commented Feb 8, 2023

kartben commented Feb 12, 2023

urish commented Feb 12, 2023

kartben commented Feb 13, 2023

urish commented Feb 13, 2023

urish commented Mar 23, 2023

urish commented May 4, 2023

kartben commented May 4, 2023

urish commented May 4, 2023

bmeisels commented May 6, 2023

urish commented Aug 4, 2023

kartben commented Aug 12, 2023

urish commented Aug 14, 2023

kartben commented Oct 3, 2023

urish commented Oct 3, 2023

kartben commented Oct 3, 2023

urish commented Oct 3, 2023

kartben commented Oct 3, 2023 • edited Loading

kartben commented Oct 3, 2023

urish commented Oct 3, 2023

kartben commented Oct 3, 2023

urish commented Oct 3, 2023

urish commented Oct 3, 2023

kartben commented Oct 4, 2023

kartben commented Oct 4, 2023

urish commented Oct 4, 2023

kartben commented Oct 4, 2023 • edited Loading

urish commented Oct 4, 2023

kartben commented Oct 4, 2023

kartben commented Oct 4, 2023

urish commented Oct 4, 2023

urish commented Oct 4, 2023

kartben commented Oct 5, 2023

kartben commented Oct 5, 2023

urish commented Oct 5, 2023

beriberikix commented Oct 5, 2023 • edited Loading

kartben commented Oct 5, 2023

urish commented Oct 5, 2023

kartben commented Oct 5, 2023

kartben commented Oct 5, 2023

urish commented Oct 5, 2023

urish commented Oct 5, 2023

kartben commented Oct 5, 2023

kartben commented Oct 6, 2023

urish commented Oct 6, 2023

urish commented Oct 6, 2023

urish commented Oct 6, 2023 • edited Loading

kartben commented Oct 6, 2023

urish commented Oct 6, 2023

kartben commented Oct 6, 2023

kartben commented Oct 6, 2023 • edited Loading

urish commented Oct 7, 2023

kartben commented Oct 16, 2023

urish commented Oct 20, 2023

kartben commented Oct 20, 2023

urish commented Oct 20, 2023

kartben commented Oct 3, 2023 •

edited

Loading

kartben commented Oct 4, 2023 •

edited

Loading

beriberikix commented Oct 5, 2023 •

edited

Loading

urish commented Oct 6, 2023 •

edited

Loading

kartben commented Oct 6, 2023 •

edited

Loading