Next Tutorial: Detection of ArUco boards

Original authors Sergio Garrido, Alexander Panov Compatibility OpenCV >= 4.7.0

Pose estimation is of great importance in many computer vision applications: robot navigation, augmented reality, and many more. This process is based on finding correspondences between points in the real environment and their 2d image projection. This is usually a difficult step, and thus it is common to use synthetic or fiducial markers to make it easier.

One of the most popular approaches is the use of binary square fiducial markers. The main benefit of these markers is that a single marker provides enough correspondences (its four corners) to obtain the camera pose. Also, the inner binary codification makes them specially robust, allowing the possibility of applying error detection and correction techniques.

The aruco module is based on the ArUco library, a popular library for detection of square fiducial markers developed by Rafael Muñoz and Sergio Garrido [99].

The aruco functionalities are included in:

#include aruco_detector.hppMarkers and Dictionaries

An ArUco marker is a synthetic square marker composed by a wide black border and an inner binary matrix which determines its identifier (id). The black border facilitates its fast detection in the image and the binary codification allows its identification and the application of error detection and correction techniques. The marker size determines the size of the internal matrix. For instance a marker size of 4x4 is composed by 16 bits.

Some examples of ArUco markers:

Example of markers images

It must be noted that a marker can be found rotated in the environment, however, the detection process needs to be able to determine its original rotation, so that each corner is identified unequivocally. This is also done based on the binary codification.

A dictionary of markers is the set of markers that are considered in a specific application. It is simply the list of binary codifications of each of its markers.

The main properties of a dictionary are the dictionary size and the marker size.

The dictionary size is the number of markers that compose the dictionary.The marker size is the size of those markers (the number of bits/modules).

The aruco module includes some predefined dictionaries covering a range of different dictionary sizes and marker sizes.

One may think that the marker id is the number obtained from converting the binary codification to a decimal base number. However, this is not possible since for high marker sizes the number of bits is too high and managing such huge numbers is not practical. Instead, a marker id is simply the marker index within the dictionary it belongs to. For instance, the first 5 markers in a dictionary have the ids: 0, 1, 2, 3 and 4.

More information about dictionaries is provided in the "Selecting a dictionary" section.

Marker Creation

Before their detection, markers need to be printed in order to be placed in the environment. Marker images can be generated using the generateImageMarker() function.

For example, lets analyze the following call:

cv::Mat markerImage;cv::aruco::Dictionary dictionary = cv::aruco::getPredefinedDictionary(cv::aruco::DICT_6X6_250);cv::aruco::generateImageMarker(dictionary, 23, 200, markerImage, 1);cv::imwrite("marker23.png", markerImage);cv::Matn-dimensional dense array classDefinition mat.hpp:812cv::aruco::DictionaryDictionary is a set of unique ArUco markers of the same size.Definition aruco_dictionary.hpp:29cv::imwriteCV_EXPORTS_W bool imwrite(const String &filename, InputArray img, const std::vector< int > ¶ms=std::vector< int >())Saves an image to a specified file.cv::aruco::generateImageMarkervoid generateImageMarker(const Dictionary &dictionary, int id, int sidePixels, OutputArray img, int borderBits=1)Generate a canonical marker image.cv::aruco::getPredefinedDictionaryDictionary getPredefinedDictionary(PredefinedDictionaryType name)Returns one of the predefined dictionaries defined in PredefinedDictionaryType.cv::aruco::DICT_6X6_250@ DICT_6X6_2506x6 bits, minimum hamming distance between any two codes = 11, 250 codesDefinition aruco_dictionary.hpp:110

First, the cv::aruco::Dictionary object is created by choosing one of the predefined dictionaries in the aruco module. Concretely, this dictionary is composed of 250 markers and a marker size of 6x6 bits (cv::aruco::DICT_6X6_250).

The parameters of cv::aruco::generateImageMarker() are:

The first parameter is the cv::aruco::Dictionary object previously created.The second parameter is the marker id, in this case the marker 23 of the dictionary cv::aruco::DICT_6X6_250. Note that each dictionary is composed of a different number of markers. In this case, the valid ids go from 0 to 249. Any specific id out of the valid range will produce an exception.The third parameter, 200, is the size of the output marker image. In this case, the output image will have a size of 200x200 pixels. Note that this parameter should be large enough to store the number of bits for the specific dictionary. So, for instance, you cannot generate an image of 5x5 pixels for a marker size of 6x6 bits (and that is without considering the marker border). Furthermore, to avoid deformations, this parameter should be proportional to the number of bits + border size, or at least much higher than the marker size (like 200 in the example), so that deformations are insignificant.The fourth parameter is the output image.Finally, the last parameter is an optional parameter to specify the width of the marker black border. The size is specified proportional to the number of bits. For instance a value of 2 means that the border will have a width equivalent to the size of two internal bits. The default value is 1.

The generated image is:

Generated marker

A full working example is included in the create_marker.cpp inside the samples/cpp/tutorial_code/objectDetection/.

The samples now take input from the command line using cv::CommandLineParser. For this file the example parameters will look like:

"marker23.png" -d=10 -id=23

Parameters for create_marker.cpp:

const char* keys = "{@outfile |res.png| Output image }" "{d | 0 | dictionary: DICT_4X4_50=0, DICT_4X4_100=1, DICT_4X4_250=2," "DICT_4X4_1000=3, DICT_5X5_50=4, DICT_5X5_100=5, DICT_5X5_250=6, DICT_5X5_1000=7, " "DICT_6X6_50=8, DICT_6X6_100=9, DICT_6X6_250=10, DICT_6X6_1000=11, DICT_7X7_50=12," "DICT_7X7_100=13, DICT_7X7_250=14, DICT_7X7_1000=15, DICT_ARUCO_ORIGINAL = 16}" "{cd | | Input file with custom dictionary }" "{id | 0 | Marker id in the dictionary }" "{ms | 200 | Marker size in pixels }" "{bb | 1 | Number of bits in marker borders }" "{si | false | show generated image }";}Marker Detection

Given an image containing ArUco markers, the detection process has to return a list of detected markers. Each detected marker includes:

The position of its four corners in the image (in their original order).The id of the marker.

The marker detection process is comprised of two main steps:

Detection of marker candidates. In this step the image is analyzed in order to find square shapes that are candidates to be markers. It begins with an adaptive thresholding to segment the markers, then contours are extracted from the thresholded image and those that are not convex or do not approximate to a square shape are discarded. Some extra filtering is also applied (removing contours that are too small or too big, removing contours too close to each other, etc).After the candidate detection, it is necessary to determine if they are actually markers by analyzing their inner codification. This step starts by extracting the marker bits of each marker. To do so, a perspective transformation is first applied to obtain the marker in its canonical form. Then, the canonical image is thresholded using Otsu to separate white and black bits. The image is divided into different cells according to the marker size and the border size. Then the number of black or white pixels in each cell is counted to determine if it is a white or a black bit. Finally, the bits are analyzed to determine if the marker belongs to the specific dictionary. Error correction techniques are employed when necessary.

Consider the following image:

Image with an assortment of markers

And a printout of this image in a photo:

Original image with markers

These are the detected markers (in green). Note that some markers are rotated. The small red square indicates the marker’s top left corner:

Image with detected markers

And these are the marker candidates that have been rejected during the identification step (in pink):

Image with rejected candidates

In the aruco module, the detection is performed in the cv::aruco::ArucoDetector::detectMarkers() function. This function is the most important in the module, since all the rest of the functionality is based on the detected markers returned by cv::aruco::ArucoDetector::detectMarkers().

An example of marker detection:

cv::Mat inputImage;// ... read inputImage ...std::vector markerIds;std::vector markerCorners, rejectedCandidates;cv::aruco::DetectorParameters detectorParams = cv::aruco::DetectorParameters();cv::aruco::Dictionary dictionary = cv::aruco::getPredefinedDictionary(cv::aruco::DICT_6X6_250);cv::aruco::ArucoDetector detector(dictionary, detectorParams);detector.detectMarkers(inputImage, markerCorners, markerIds, rejectedCandidates);cv::aruco::ArucoDetectorThe main functionality of ArucoDetector class is detection of markers in an image with detectMarkers(...Definition aruco_detector.hpp:276cv::aruco::DetectorParametersstruct DetectorParameters is used by ArucoDetectorDefinition aruco_detector.hpp:25

When you create an cv::aruco::ArucoDetector object, you need to pass the following parameters to the constructor:

A dictionary object, in this case one of the predefined dictionaries (cv::aruco::DICT_6X6_250).Object of type cv::aruco::DetectorParameters. This object includes all parameters that can be customized during the detection process. These parameters will be explained in the next section.

The parameters of cv::aruco::ArucoDetector::detectMarkers() are:

The first parameter is the image containing the markers to be detected.The detected markers are stored in the markerCorners and markerIds structures:markerCorners is the list of corners of the detected markers. For each marker, its four corners are returned in their original order (which is clockwise starting with top left). So, the first corner is the top left corner, followed by the top right, bottom right and bottom left.markerIds is the list of ids of each of the detected markers in markerCorners. Note that the returned markerCorners and markerIds vectors have the same size.The final optional parameter, rejectedCandidates, is a returned list of marker candidates, i.e. shapes that were found and considered but did not contain a valid marker. Each candidate is also defined by its four corners, and its format is the same as the markerCorners parameter. This parameter can be omitted and is only useful for debugging purposes and for ‘refind’ strategies (see cv::aruco::ArucoDetector::refineDetectedMarkers()).

The next thing you probably want to do after cv::aruco::ArucoDetector::detectMarkers() is check that your markers have been correctly detected. Fortunately, the aruco module provides a function to draw the detected markers in the input image, this function is drawDetectedMarkers(). For example:

cv::Mat outputImage = inputImage.clone();cv::aruco::drawDetectedMarkers(outputImage, markerCorners, markerIds);cv::Mat::cloneCV_NODISCARD_STD Mat clone() constCreates a full copy of the array and the underlying data.cv::aruco::drawDetectedMarkersvoid drawDetectedMarkers(InputOutputArray image, InputArrayOfArrays corners, InputArray ids=noArray(), Scalar borderColor=Scalar(0, 255, 0))Draw detected markers in image.outputImage is the input/output image where the markers will be drawn (it will normally be the same as the image where the markers were detected).markerCorners and markerIds are the structures of the detected markers returned by the cv::aruco::ArucoDetector::detectMarkers() function.Image with detected markers

Note that this function is only provided for visualization and its use can be omitted.

With these two functions we can create a basic marker detection loop to detect markers from our camera:

cv::aruco::ArucoDetector detector(dictionary, detectorParams); cv::VideoCapture inputVideo; int waitTime; if(!video.empty()) { inputVideo.open(video); waitTime = 0; } else { inputVideo.open(camId); waitTime = 10; } double totalTime = 0; int totalIterations = 0; // set coordinate system cv::Mat objPoints(4, 1, CV_32FC3); objPoints.ptr(0)[0] = Vec3f(-markerLength/2.f, markerLength/2.f, 0); objPoints.ptr(0)[1] = Vec3f(markerLength/2.f, markerLength/2.f, 0); objPoints.ptr(0)[2] = Vec3f(markerLength/2.f, -markerLength/2.f, 0); objPoints.ptr(0)[3] = Vec3f(-markerLength/2.f, -markerLength/2.f, 0); while(inputVideo.grab()) { cv::Mat image, imageCopy; inputVideo.retrieve(image); double tick = (double)getTickCount(); vector ids; vector corners, rejected; // detect markers and estimate pose detector.detectMarkers(image, corners, ids, rejected); size_t nMarkers = corners.size(); vector rvecs(nMarkers), tvecs(nMarkers); if(estimatePose && !ids.empty()) { // Calculate pose for each marker for (size_t i = 0; i < nMarkers; i++) { solvePnP(objPoints, corners.at(i), camMatrix, distCoeffs, rvecs.at(i), tvecs.at(i)); } } double currentTime = ((double)getTickCount() - tick) / getTickFrequency(); totalTime += currentTime; totalIterations++; if(totalIterations % 30 == 0) { cout