Here is all what you wanted to know about Mat... but it doesn't talk much about what you put inside (the images itself), so for an easier ride (hopefully), keep reading: :)
You can read one element with get(x,y), with x/y the position of the element or a group of elements with get(x,y,byte[]) with x,y=0,0 (for the full array; does it indicate the origin?). The result will get stored in the argument array byte[]. You can change one element with put, in reverse way as get... See example here.
On the other side, BufferedImage is a java subclass that describes an image with an accessible buffer of image data. A BufferedImage is comprised of a ColorModel and a Raster of image data. The 2nd is what holds the image pixels. See how to access them here.
To answer how to read/set the pixels on those structures, we will try to answer the title of this post, i.e., how to go from Mat (the openCV resulting image) to BufferedImage (to display), and do it in the most efficient way. Therefore, we proceed to benchmark few methods (see full code here). Hint: if you want to skip all the reading, jump to Method 4 :)
Method 1 Go through a jpg. Although looks the cleanest code (actually suggested by one commenter), another commenter thought that this would take the highest computation effort, and basically trigger this whole post :).
public boolean MatToBufferedImage(Mat matrix) {
long startTime = System.nanoTime();
MatOfByte mb=new MatOfByte();
Highgui.imencode(".jpg", matrix, mb);
try {
image = ImageIO.read(new ByteArrayInputStream(mb.toArray()));
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
return false; // Error
}
long endTime = System.nanoTime();
System.out.println(String.format("Elapsed time: %.2f ms", (float)(endTime - startTime)/1000000));
return true; // Successful
}
Detected 2 faces
Elapsed time: 25.94 ms
Detected 1 faces
Elapsed time: 27.56 ms
Detected 2 faces
Elapsed time: 27.37 ms
Detected 1 faces
Elapsed time: 26.96 ms
Detected 1 faces
Elapsed time: 35.70 ms
Detected 1 faces
Elapsed time: 27.32 ms
Method 2 Extract the data from Mat into an array, flip the blue and the red channels/columns and store in BufferedImage.
public boolean MatToBufferedImage(Mat matrix) {
long startTime = System.nanoTime();
int cols = matrix.cols();
int rows = matrix.rows();
int elemSize = (int)matrix.elemSize();
byte[] data = new byte[cols * rows * elemSize];
int type;
matrix.get(0, 0, data);
switch (matrix.channels()) {
case 1:
type = BufferedImage.TYPE_BYTE_GRAY;
break;
case 3:
type = BufferedImage.TYPE_3BYTE_BGR;
// bgr to rgb
byte b;
for(int i=0; i<data.length; i=i+3) {
b = data[i];
data[i] = data[i+2];
data[i+2] = b;
}
break;
default:
return false; // Error
}
image = new BufferedImage(cols, rows, type);
image.getRaster().setDataElements(0, 0, cols, rows, data);
long endTime = System.nanoTime();
System.out.println(String.format("Elapsed time: %.2f ms", (float)(endTime - startTime)/1000000));
return true; // Successful
}
Detected 2 faces
Elapsed time: 2.27 ms
Detected 2 faces
Elapsed time: 2.81 ms
Detected 2 faces
Elapsed time: 2.25 ms
Detected 2 faces
Elapsed time: 2.75 ms
Detected 2 faces
Elapsed time: 2.22 ms
Substantial (10x!!) improvement, as the anonymous commenter had anticipated...
Method 3 We do the color conversion in OpenCV (see here for color conversions within OpenCV) and then save to BufferedImage:
public boolean MatToBufferedImage(Mat matrix) {
long startTime = System.nanoTime();
Imgproc.cvtColor(matrix, matrix, Imgproc.COLOR_BGR2RGB);
int cols = matrix.cols();
int rows = matrix.rows();
int elemSize = (int)matrix.elemSize();
byte[] data = new byte[cols * rows * elemSize];
matrix.get(0, 0, data);
image = new BufferedImage(cols, rows, BufferedImage.TYPE_3BYTE_BGR);
image.getRaster().setDataElements(0, 0, cols, rows, data);
long endTime = System.nanoTime();
System.out.println(String.format("Elapsed time: %.2f ms", (float)(endTime - startTime)/1000000));
return true; // Successful
}
Detected 2 faces
Elapsed time: 2.19 ms
Detected 2 faces
Elapsed time: 12.68 ms
Detected 3 faces
Elapsed time: 2.04 ms
Detected 2 faces
Elapsed time: 2.91 ms
Detected 3 faces
Elapsed time: 2.05 ms
Detected 2 faces
Elapsed time: 2.84 ms
Maybe slightly faster than doing it by hand but... Notice also the long 12ms case above. Nevertheless, I observed that in the other algorithms too, so, I feel that is because the processor gets distracted with some other function...
Method 4 Finally, we get to what is the most efficient method. It basically goes straight from the BGR to the BufferedImage.
public boolean MatToBufferedImage(Mat matBGR){
long startTime = System.nanoTime();
int width = matBGR.width(), height = matBGR.height(), channels = matBGR.channels() ;
byte[] sourcePixels = new byte[width * height * channels];
matBGR.get(0, 0, sourcePixels);
// create new image and get reference to backing data
image = new BufferedImage(width, height, BufferedImage.TYPE_3BYTE_BGR);
final byte[] targetPixels = ((DataBufferByte) image.getRaster().getDataBuffer()).getData();
System.arraycopy(sourcePixels, 0, targetPixels, 0, sourcePixels.length);
long endTime = System.nanoTime();
System.out.println(String.format("Elapsed time: %.2f ms", (float)(endTime - startTime)/1000000));
return true;
}
In this 4th method suggested by the anonymous commenter we get the best. Even up to 50x improvement respect to the method 1:
Detected 2 faces
Elapsed time: 0.51 ms
Detected 2 faces
Elapsed time: 1.22 ms
Detected 1 faces
Elapsed time: 0.47 ms
Detected 1 faces
Elapsed time: 1.32 ms
Detected 1 faces
Elapsed time: 0.48 ms
Detected 1 faces
Elapsed time: 1.77 ms
I still need to understand why this other method does not require to flip R and B, but hey, it works... So, thank you, sir! :)
Cheers!!
PS.: This other post talks about how to get the picture in BufferedImage into an array and does a nice work benchmarking two methods, either using getRGB or using ((DataBufferByte) bufferedImage.getRaster().getDataBuffer()).getData();
PS2.: More theory on BufferedImages.
PS3.: By the way, the method may be nicer if it was returning the image instead of accessing it as a variable of the mother class, but anyhow...
No comments:
Post a Comment