Question: how are Aruco markers used?

[lexvandersluijs]

Hi,

In the code I see some references to Aruco markers and April tags, but I can't seem to find information in the Wiki and website about how they are used. I can imagine a number of scenarios:

• as an aid in loop closure, as they can be identified more robustly than natural features
• as an aid to increase precision, as they can be localized in the image more precisely than natural features
• as an aid with relocalization, in case the system is used for tracking (e.g. for AR)
• or maybe they are 'just' reconstructed in 3D, to increase the ease of working with the scan-result

I'm planning to build the software from source myself, so that I can more easily follow the code-paths in Visual Studio. But if someone could answer this question in the meantime that would be very helpful.

[matlabbe]

It is quite a new feature, it is why there is no official documentation yet.

Install:

To have the built-in option in RTAB-Map standalone, OpenCV should be built with aruco module from opencv_contrib. To support apriltags2 dictionary, at least OpenCV 3.4.2 should be built.

Usage:

In RTAB-Map, go to Preferences (advanced) -> Marker Detection panel, then check the checkbox to enable marker detection. Assumptions:

• Markers have all the same size
• Markers are static
• Markers have all unique ids
• Markers should have non-zero / non-negative IDs
• If marker size parameter is not set, we should place the camera perpendicular (in front) of a marker to initialize the size. Once initialization is done, the markers can be detected in any orientations.

Effects:

• Markers are used like visual loop closure detection. When a previous marker is detected, a new constraint is added to graph and the map is optimized.
• Markers will be shown in 3D Map view as coordinate frames
• Markers can be also used as "anchors" to merge multiple sessions together.
• They are not yet used for visual odometry, only for loop closure detection and localization.

ROS:

This built-in approach can be used with rtabmap_ros. There is also an example of integration with apriltags2_ros if we want to use the external package instead of the built-in option, which may be more convenient on some systems if we don't want to rebuild another OpenCV with minimum version above. With the external approach, we can also define mosaics of tags that can be interesting for some applications.

cheers,
Mathieu

[lexvandersluijs]

Hi Mathieu,

That sounds great, thanks!
I'm quite impressed by the amount of technology that has been integrated into RTAB-Map, nice work!

I've got the Windows version compiling, building and running now. I had to make a few small changes in a few CMakeLists, what would be the best way to communicate them? In a thread like this or as a pull request?

By the way, I ran into a runtime issue, will start a new Issue for that one.

Cheers,
Lex

[matlabbe]

A pull request is better to compare changes. For the windows build, you may also check this file to see if the configuration used could have fixed your build errors:
https://github.com/introlab/rtabmap/blob/master/.appveyor.yml

If your changes are related to newer versions of the dependencies used in that build file, I'll be happy to integrate them.

cheers,
Mathieu

[molinadavid]

Hi Mathieu,

Great work integrating the apriltags to rtabmap, I have been taking a look at it but I don't know how to confirm if the tags are actually helping for loop closures or not, if you could give me some pointers on how to actually confirm that the tags are working I would be very grateful.

I am attaching 3 databases for your convenience, 2 of those have the apriltags, and for the last one I covered the tags in order to test the differences.

apriltags
apriltagsv2
notags

Best regards,
David

[malerv]

In rtabmap-databaseViewer you can look at the graph view (menu view/Graph view). it will show the occupancy grid and the results graph. In the graph, loop closures are in red. If the tag helped my guess is that you would have more loop closure around the location of the tag.

[matlabbe]

If tags are detected, you would see coordinate frames on them in 3D Map view. In graph view, you would see dark green links and nodes with negative id (note that it is not yet implemented in database viewer). Also in loop closure detection view during mapping, we can see a green background on loop closure, but you should see an orange background when closing a loop on a landmark.

In your databases, I don't see any tags added to the graph. With some testings, it is because you are using TORO as optimizer and only GTSAM/G2O are supported with landmarks. After optimization, the landmarks disappeared from the graph view. Set Optimizer/Strategy to 1 (g2o) or 2 (gtsam). Also, don't set at the same time Optimizer/Robust and RGBD/OptimizeMaxError, they are not compatible. You may also set Aruco/VarianceAngular to 9999 if you find that orientation of the landmarks are poorly estimated.

Here is an example of what you should see when landmarks are detected and added to graph:

Note the orange background and the coordinate frames of landmark printed in images under loop closure detection view and the dark green links on the graph view.

cheers,
Mathieu

[molinadavid]

Hi Mathieu,

Thank you very much for your response, I was able to confirm the presence of the tags as you suggested.

Best regards,
David

[1fabrism]

Hey guys, I have a question about using this feature. I was wondering what happens if the markers are present in the environment but the MarkerDetection parameter is set to false (I'm talking specifically about rtabmap_ros). I imagine the markers would still be used by the software as useful features, especially if they are placed on an otherwise plain wall? If so, what exactly is the difference between turning the feature on or off?
I'm planning to do tests to show how aruco markers can improve localization, so I was wondering if I can leave the markers hanging on the wall all the time and just turn the feature on and off, or if I should still make sure rtabmap is not using them "passively" by removing them from the environment altogether.

[matlabbe]

rtabmap's visual loop closure detection extracts visual features everywhere on the image. As the markers have some contrast, visual features may be extracted indirectly from the markers and used for visual loop closure detection.

When we enable RGBD/MarkerDetection parameter, another process (i.e., ArUco marker detection from OpenCV) is executed to find markers in the image. When detected, they add a new type of constraints in the graph. Like visual loop closure constraints, those "landmark" constraints are used for loop closure detection and graph optimization.

If you disable RGBD/MarkerDetection, rtabmap may still extract visual features on them, so indirectly affect visual loop closure detection performance. It is like adding artificial visual features to environment. You may cover them with a blank paper if you really don't want visual loop closure detection affected by them.

Finally, if you have want a scenario where we disable visual loop closure detection and use only marker detection, Kp/MaxFeatures can be set to -1 to disable visual loop closure detection.

cheers,
Mathieu

[molinadavid]

Hi Mathieu,

As always thank you very much for all your work. We are currently trying to test using the apriltags2 as an external package as your example, the issue we have at the moment is that when I visualize the mapping process the tf of the tag appears on the map but the TF is wrong making the tag appear on the Z axis of the robot rather than the on the front of it.

Please see this 2 screenshots for an example:

the database created is here
rtabmap_externalTags.db

Best regards,
David

[matlabbe]

Is Tag_3 the one we see in the camera feed on left? If so, the TF may appear correct from the apriltag output, referred in camera optical frame. In TF, it seems the camera is looking upward. You may change the camera_frame parameter in the appriltags2_ros example:

<arg name="camera_frame_id" default="camera_color_optical_frame"/>

to maybe camera_link.

I looked in the database and there are no landmark links. Is rtabmap connected to apriltags output like in the following screenshot?

cheers,
Mathieu

[szx0112]

Hi @matlabbe , it seems the link of the example file is gone. Could you update the link? I'd like to try the marker detection function in my project. Thanks.

[matlabbe]

See https://github.com/introlab/rtabmap_ros/blob/master/launch/tests/test_apriltag_ros.launch

[szx0112]

@matlabbe Thanks for the link. Just a follow up question. Is it possible to pre-upload the marker positions? For example, we already have the ground truth poses of the markers or the marker poses are computed from a previous mapping using RTABMAP (and then we can extract and save it for future usage).

[matlabbe]

See http://official-rtab-map-forum.206.s1.nabble.com/How-to-improve-mapping-accuracy-based-on-ArUco-identification-code-tp8393p8497.html, it is not actually implemented in rtabmap, but that post shows a way to implement it.

[ZitongLan]

Hello, I am new to rtabmap. and I have a .db file containing tag detection results. And I could use it to visualize the tag in the 3d graph view. Where is the tags' information stored? How can I get the tags location from this .db file (with their ID)
I attached a .db file for here.
https://drive.google.com/file/d/1bjzE68mvXHUCNhVxd3hRkiu6eC-2NTSz/view?usp=drive_link

[matlabbe]

See #1199

[takahiko-hasegawa-hdjp]

Does this mean that AR maker can only be used in creating a map? and not used for map matching?

[matlabbe]

Well, it depends how you want to do map matching. We can combine two maps together with same tags detected in each other like this:

rtabmap-reprocess "map1.db;map2.db" output.db

The two maps will be merged together using tag constraints.

[takahiko-hasegawa-hdjp]

Sorry when I said "map matching", i mean visual localization. During the localzation phase, even If the tags are stored in the map (.db) and all the tags are still located in the same place in the environment, it would not help localization with AR marker capabilities ON?

[matlabbe]

The tags are also used in localization mode to relocalize the robot/camera.