How to collect logs for devices that don't work as expected

Submitted by dankouba on #1

I have VirtualHere set up on a Raspberry Pi to act as a hub to host various microprocessor development boards and debuggers for myself and my colleagues to use. I'm currently testing with OSX as my client. 

I've found that some boards work fine, but a couple don't and hang during certain operations. How best can I collect debug logs from VirtualHere to share here? The basic logging (via journalctl) doesn't show me anything relevant that I can see — just that the device binds to my client. 

Here are the boards that don't work:

https://www.raspberrypi.com/products/debug-probe/ (programming via https://openocd.org/)

https://www.espressif.com/en/dev-board/esp32-c3-devkit-rust-1-en

I'm able to claim them on my client and begin programming operations just fine. However, both hang at various points in the programming process and I suspect they are doing weird or non-standard things with the USB stack during the process.

Submitted by Michael on Fri, 2025/04/25 - 09:24

#2

The pico ocd works fine via virtualhere, i used it myself for a long time to develop some hardware.

Are you connected via wifi or via ethernet between the pi and the network? 

If it just stops randomly its almost always a latency issue and ethernet is mandatory. Also it wont work over the internet, the latency must be low.


Regarding the ESPS3, i tried that also and actually that one does have a race condition. It waits a few strict number of ms between flash commands and that isnt compatible with virtualhere. Not sure if expressif can do a workaround or not.

Submitted by dankouba on Tue, 2025/04/29 - 04:33

#3

Thanks for the response. Let's focus on the Pico probe for now as that is more of a priority for me.

My probe always hangs at the same point with OpenOCD. Here's a log with the debug info turned all the way up:

$ ~/openocd/src/openocd -s ~/openocd/tcl -f interface/cmsis-dap.cfg -f target/rp2350.cfg -c 'debug_level 4; init; reset init; flash erase_address 0x10000000 0x400000; shutdown'
Open On-Chip Debugger 0.12.0+dev-01943-g2aa0592e0 (2025-04-21-18:39)
Licensed under GNU GPL v2
For bug reports, read
       http://openocd.org/doc/doxygen/bugs.html
Info : [rp2350.dap.core0] Hardware thread awareness created
Info : [rp2350.dap.core1] Hardware thread awareness created
Info : [rp2350.dap.core0] Hardware thread awareness created
Info : [rp2350.dap.core1] Hardware thread awareness created
cortex_m reset_config sysresetreq
Debug: 9 2 command.c:86 script_debug(): command - init
Debug: 10 2 command.c:86 script_debug(): command - target init
Debug: 11 2 command.c:86 script_debug(): command - target names
Debug: 12 2 command.c:86 script_debug(): command - rp2350.dap.core0 cget -event gdb-flash-erase-start
Debug: 13 2 command.c:86 script_debug(): command - rp2350.dap.core0 configure -event gdb-flash-erase-start reset init
Debug: 14 2 command.c:86 script_debug(): command - rp2350.dap.core0 cget -event gdb-flash-write-end
Debug: 15 2 command.c:86 script_debug(): command - rp2350.dap.core0 configure -event gdb-flash-write-end reset halt
Debug: 16 2 command.c:86 script_debug(): command - rp2350.dap.core0 cget -event gdb-attach
Debug: 17 2 command.c:86 script_debug(): command - rp2350.dap.core0 configure -event gdb-attach halt 1000
Debug: 18 2 command.c:86 script_debug(): command - rp2350.dap.core1 cget -event gdb-flash-erase-start
Debug: 19 2 command.c:86 script_debug(): command - rp2350.dap.core1 configure -event gdb-flash-erase-start reset init
Debug: 20 2 command.c:86 script_debug(): command - rp2350.dap.core1 cget -event gdb-flash-write-end
Debug: 21 2 command.c:86 script_debug(): command - rp2350.dap.core1 configure -event gdb-flash-write-end reset halt
Debug: 22 2 command.c:86 script_debug(): command - rp2350.dap.core1 cget -event gdb-attach
Debug: 23 2 command.c:86 script_debug(): command - rp2350.dap.core1 configure -event gdb-attach halt 1000
Debug: 24 2 target.c:1597 handle_target_init_command(): Initializing targets...
Debug: 25 2 semihosting_common.c:109 semihosting_common_init():  
Debug: 26 2 semihosting_common.c:109 semihosting_common_init():  
Warn : 27 2 adapter.c:143 adapter_init(): An adapter speed is not selected in the init scripts. OpenOCD will try to run the adapter at very low speed (100 kHz).
Warn : 28 2 adapter.c:145 adapter_init(): To remove this warnings and achieve reasonable communication speed with the target, set "adapter speed" or "jtag_rclk" in the init scripts.
Debug: 29 2 adapter.c:253 adapter_config_khz(): handle adapter khz
Debug: 30 2 adapter.c:216 adapter_khz_to_speed(): convert khz to adapter specific speed value
Debug: 31 11 cmsis_dap_usb_bulk.c:170 cmsis_dap_usb_open(): found product string of 0x2e8a:0x000c 'Debug Probe (CMSIS-DAP)'
Debug: 32 11 cmsis_dap_usb_bulk.c:190 cmsis_dap_usb_open(): enumerating interfaces of 0x2e8a:0x000c
Debug: 33 14 cmsis_dap_usb_bulk.c:237 cmsis_dap_usb_open(): found interface 0 string 'CMSIS-DAP v2 Interface'
Info : 34 14 cmsis_dap_usb_bulk.c:340 cmsis_dap_usb_open(): Using CMSIS-DAPv2 interface with VID:PID=0x2e8a:0x000c, serial=E6614103E733432F
Debug: 35 15 cmsis_dap_usb_bulk.c:457 cmsis_dap_usb_read(): submit read @ 0
Debug: 36 1111 cmsis_dap_usb_bulk.c:511 cmsis_dap_usb_read(): USB timeout @ 0
Debug: 37 1111 cmsis_dap_usb_bulk.c:567 cmsis_dap_usb_write(): submit write @ 0
Debug: 38 1111 cmsis_dap_usb_bulk.c:457 cmsis_dap_usb_read(): submit read @ 0

It hangs there indefinitely. I have tried this with both the Raspberry Pi OpenOCD fork and the latest mainline version. Same result.

My setup is quite simple and all hardwired (no WiFi):

  • Raspberry Pi 5 with a 10 port USB 3 hub attached. Multiple dev boards attached to the hub. I have tried with and without the hub in place as well.
  • The Pi has wired ethernet to a switch, which my computer shares.
  • My debug probe has been updated to the latest v2.2.2 firmware

 

Submitted by Michael on Tue, 2025/04/29 - 18:33

#4

Does it OK for you with the windows client or Linux virtualhere client on those platforms? 

I reproduced the issue just on the MacOS client so i think it could be a bug in the virtualhere client on mac only which i will investigate.

 

Submitted by dankouba on Wed, 2025/04/30 - 08:00

#5

Good call — I confirmed things work as expected with the Linux client. I don't have a Windows machine around to test with.

Submitted by Michael on Wed, 2025/04/30 - 08:05

#6

Thanks for confirming, im researching this issue more today...

Submitted by Michael on Thu, 2025/05/01 - 14:13

#7

I havent been able to figure out this issue on the mac. So for the time being you will need to use Linux or Windows for this instead.

Submitted by dankouba on Fri, 2025/05/02 - 00:40

#8

Appreciate you taking a look. I unfortunately beholden to my mac client (work computer) so I will directly connect until there is a solution. I'll check in in a few weeks to see if there has been any progress.

Submitted by Michael on Fri, 2025/05/02 - 09:01

#9

I will keep investigating but not sure if i can find the cause. Oddly i have all the source code, and a hardware analyzer and the messages are identical via virtualhere and direct and it still jams....

Submitted by dankouba on Fri, 2025/05/02 - 11:01

#10

My concern is it is something inside the pico probe USB stack not playing nice, like the ESP32, rather than on your end. Odd that it works fine on other targets though.