Not able to list logs for download if high number of logs

[shervinkoushan]

The logfile_download example works fine on the Pixhawk 4, but not on the mRo Control Zero H7 OEM.

This is the output:

09:21 $ build/logfile_download udp://:14540
[09:21:55|Info ] MAVSDK version: v1.4.17 (mavsdk_impl.cpp:20)
Waiting to discover system...
[09:21:55|Info ] New system on: 10.10.0.176:14540 (with sysid: 1) (udp_connection.cpp:192)
[09:21:55|Debug] New: System ID: 1 Comp ID: 1 (mavsdk_impl.cpp:496)
[09:21:55|Debug] Component Autopilot (1) added. (system_impl.cpp:377)
[09:21:55|Warn ] Vehicle type changed (new type: 2, old type: 0) (system_impl.cpp:225)
[09:21:55|Debug] Discovered 1 component(s) (system_impl.cpp:578)
[09:21:55|Warn ] No entries received (log_files_impl.cpp:167)
[09:21:56|Warn ] sending again after 0.502617 s, retries to do: 3  (520). (mavlink_command_sender.cpp:287)
[09:21:56|Warn ] sending again after 1.00862 s, retries to do: 2  (520). (mavlink_command_sender.cpp:287)
[09:21:57|Warn ] sending again after 1.5145 s, retries to do: 1  (520). (mavlink_command_sender.cpp:287)
[09:21:57|Error] Retrying failed (520) (mavlink_command_sender.cpp:307)

Any idea why it doesn't work on the mRo?

Edit: Seems to be an issue with the log contents, not the mRo

[JonasVautherin]

Feels like MAVLink messages from MAVSDK don't reach the mRo, though MAVLink messages from the mRo do reach MAVSDK. Could it be that you have an issue there?

[shervinkoushan]

Feels like MAVLink messages from MAVSDK don't reach the mRo, though MAVLink messages from the mRo do reach MAVSDK. Could it be that you have an issue there?

Thanks for the reply. That might be, is there any way I can troubleshoot this?

[JonasVautherin]

You can try to send other commands to the autopilot (e.g. arm, or set a parameter, etc) and confirm that none of them work (if my intuition is correct, they will just time out because they never reach the autopilot).

You can also use wireshark to see what happens on the network interface between mavsdk and mRo.

If you want, you can also try to describe your network topology here, so that if there is something obviously wrong we can maybe help 😊.

[shervinkoushan]

You can try to send other commands to the autopilot (e.g. arm, or set a parameter, etc) and confirm that none of them work (if my intuition is correct, they will just time out because they never reach the autopilot).

You can also use wireshark to see what happens on the network interface between mavsdk and mRo.

If you want, you can also try to describe your network topology here, so that if there is something obviously wrong we can maybe help 😊.

Interestingly, with the pip package mavsdk 1.4.9 I am able to set parameters using the upload_params script.
The line await get_entries(drone) in the logfile_download example still hangs though.

[JonasVautherin]

Is the mRo running the same version of PX4 as the Pixhawk? Does the mRo support logs? Like can QGC list and download logs, for instance?

[shervinkoushan]

Is the mRo running the same version of PX4 as the Pixhawk? Does the mRo support logs? Like can QGC list and download logs, for instance?

Yes to both.

• The same PX4 code is running (a modified version of 1.13.3)
• I can see the logs in QGC

[JonasVautherin]

Maybe there is a bug then. Can you check what happens with MAVLink on the mRo side when MAVSDK requests the logs?

[shervinkoushan]

Maybe there is a bug then. Can you check what happens with MAVLink on the mRo side when MAVSDK requests the logs?

Can you please be more specific? Should I check a specific topic?

[JonasVautherin]

I would look at the MAVLink messages being used by the log_files implementation: https://github.com/mavlink/MAVSDK/blob/main/src/mavsdk/plugins/log_files/log_files_impl.cpp

[shervinkoushan]

Looking at this file, the output is as follows:

1. LOG_REQUEST_END
2. LOG_REQUEST_LIST
3. Then this line is called https://github.com/PX4/PX4-Autopilot/blob/f68f88b97c77d38054d54696fca5ef1a3e91822b/src/modules/mavlink/mavlink_log_handler.cpp#L175

This behavior is the same on both the Pixhawk and the mRo, but after 3. on the Pixhawk we can see the logs, whereas it just hangs when requesting from the mRo

[shervinkoushan]

It seems like I was wrong about the root cause of the issue. I tested another SD card now and it worked on the mRo.
I tested two mRos earlier with two different SD cards, I am trying to see what is different. The folder structure is pretty similar.

A card that does not work to the left, a card that works to the right:

I tried removing the -ulogs folder from the left card but it did not help.

On the PX4 side I can see that the logs are detected (165 in the left case, 23 in the right case)

Is the issue that it struggles to handle 100+ logs, or large logs in general?

I copied the contents of the card that worked to the card that didn't, and that worked fine. So the contents of the sd card is the root problem, not the card itself.

But important to note here is that there is not an issue with the mRo, sorry for the confusion.

[JonasVautherin]

It is weird that it was working with QGC and not with MAVSDK, if that was an sdcard issue 🤔

[shervinkoushan]

It is weird that it was working with QGC and not with MAVSDK, if that was an sdcard issue 🤔

Could be an issue on the PX4 side as well, not sure. If you want I can send you the SD-card contents (by email) so you can debug it further.

[shervinkoushan]

I can see that QGC triggers LOG_REQUEST_LIST two times. The second time it requests the range 50-167.

MAVSDK first triggers LOG_REQUEST_END, then LOG_REQUEST list in the range 0-167. Maybe MAVSDK simply requests too much data at a time?

It could also be that list_timeout gets called too quickly:

MAVSDK/src/mavsdk/plugins/log_files/log_files_impl.cpp

Lines 100 to 102 in b29239d

	// This first step can take a moment, on PX4 with 100+ logs I see about 2-3s.
	_system_impl->register_timeout_handler(
	[this]() { list_timeout(); }, _system_impl->timeout_s() * 10.0, &_entries.cookie);

[JonasVautherin]

Oh right, maybe there is some tuning to do there. The message definition doesn't say anything about a limit for LOG_REQUEST_LIST, so I would assume that the autopilot should do it in a way that works...

It could also be that list_timeout gets called too quickly:

Hmm I don't know, if I understand the code correctly, the timeout is refreshed everytime a LOG_ENTRY is received, and even then there is some retry logic...

Would be nice to check what PX4 does when it receives LOG_REQUEST_LIST and if it could be a problem if too many entries are requested...

[julianoes]

You could try to increase the timeout:
https://github.com/mavlink/MAVSDK/blob/v1.4/src/mavsdk/core/include/mavsdk/mavsdk.h#L267-L275

[shervinkoushan]

You could try to increase the timeout: https://github.com/mavlink/MAVSDK/blob/v1.4/src/mavsdk/core/include/mavsdk/mavsdk.h#L267-L275

Unfortunately this did not help.
I think the problem is that too many logs are requested at the same time. Should probably try to do it in batches of 100?

Have you been able to reproduce this issue on your side?
Steps to reproduce:

1. Have 150+ ulogs on a Pixhawk/mRo/whatever
2. Run the logfile_download example (I tested both the .cpp and .py examples)
3. Code execution stops on get_entries()

[JonasVautherin]

Is PX4 having an issue when there are "too many" logs? If that's the case, shouldn't that be fixed on the PX4 side?

[M4ster00gway]

Any updates on this issue?

[julianoes]

No, sorry, it's on my long todo list of things I ought to look into. Consider GitHub sponsoring if this is important and urgent for you.

[julianoes]

Actually, it might have been fixed in the meantime: #2234

Try again with v2.4.0.

See also: mavlink/MAVSDK-Python#662

[julianoes]

I'm closing this. Comment if this is not fixed in the meantime.