Sunday, July 7, 2013

Tracking a ball with Java/OpenCV

Here we are going to explain how to track a ball. There are few places in the web that explain similar things (see PS) but without much details. So, I am going to put few posts explaining what I learned. Hopefully it is useful to you... Just to call your attention on few topics:
  1. Basically we are capturing images like we have done so far.
  2. Then we detect the ball in two ways and do an AND of both. One way is by color segmentation, and the other by the shape.
  3. You can learn here about how to select objects by color. Basically, you got to segment the image in the HSV space, which is much easier than on RGB.
  4. The instruction for thresholding is "Core.inRange(distance,new Scalar(0.0), new Scalar(200.0), thresholded2);"
  5. Notice that we do two thresholdings, one over the Hue, and another over the saturation/value of the sample. We combine both here "Core.bitwise_and(thresholded, thresholded2, thresholded);"
  6. After that we blur out noise (smoothing): "Imgproc.GaussianBlur(thresholded, thresholded, new Size(9,9),0,0);" By the way, somewhere I read that this may be included by default in the Canny transform. Not sure if right or not, but I left it, just in case.
  7. Then we apply the Hough Circles Transform. I detailed further this in this post. And you are done!
Other things that are interesting, and just list here, for search engines to find out:
  1. How to initialize an array (Mat) to a constant value: "Mat array255=new Mat(webcam_image.height(),webcam_image.width(),CvType.CV_8UC1);
            array255.setTo(new Scalar(255)); "
  2. Notice in the code how the individual channels in the HSV Mat are extracted (split the channels)."List<Mat> lhsv = new ArrayList<Mat>(3); Core.split(hsv_image, lhsv); /*We get 3 2D one channel Mat*/ Mat S = lhsv.get(1); Mat V = lhsv.get(2);"
  3. How to convert from one type of Mat to another: S.convertTo(S, CvType.CV_32F);
  4. Read the code to understand how to position text in the screen. For instance, don't use JLabel to position text in the screen. It can't. It can only do center, or left, or stuff like that... JTextField may do it, but we can also use OpenCV, which is what I did...  
  5. Notice that to display the final image, first you do everything in the Mat and then upload it to panel/image.
See below for the final code. I apologize for not being all that clean, but I got to keep going with the project. Here are some screenshots of the results :)

The original BGR with the resulting detection (purple circle around the ball):
The HSV:
 The distance to the maximum saturation/value: sqrt((255-S)^2+(255-V)^2)):
 The results after a Canny transform (edge detection): Imgproc.Canny(thresholded, thresholded, 500, 250);

The Canny transform happens inside the Hough Circles Transform, but here I had it apart just for display. Unfortunately I didn't get a picture of the result after HSV threshold and SV distance are AND together.

Overall, to tell you the truth, worked pretty well, but sometimes I got false detections on my face and also no detections if I move the ball fast. I believe the secret to it is in the adjustment of the parameters, but in the end, there is only so much one can do (imagine the background had circular figures of the same color as the ball...). We have other methods in mind that will help the performance of our overall project, mainly consisting on time filtering (a ball can't just appear and disappear instantly...)

Here is the code. I used to get an exception but no more, thanks to Abdel's comment below! Thank you! :)
 import java.awt.Graphics;  
 import java.awt.image.BufferedImage;  
 import java.util.ArrayList;  
 import java.util.List;  
 import javax.swing.JFrame;  
 import javax.swing.JPanel;  
 import org.opencv.core.Core;  
 import org.opencv.core.Mat;   
 import org.opencv.core.Point;  
 import org.opencv.core.Scalar;  
 import org.opencv.core.Size;  
 import org.opencv.highgui.VideoCapture;  
 import org.opencv.imgproc.Imgproc;  
 import org.opencv.core.CvType;  
 class Panel extends JPanel{  
   private static final long serialVersionUID = 1L;  
   private BufferedImage image;    
   // Create a constructor method  
   public Panel(){  
   private BufferedImage getimage(){  
     return image;  
   public void setimage(BufferedImage newimage){  
   public void setimagewithMat(Mat newimage){  
    * Converts/writes a Mat into a BufferedImage.  
    * @param matrix Mat of type CV_8UC3 or CV_8UC1  
    * @return BufferedImage of type TYPE_3BYTE_BGR or TYPE_BYTE_GRAY  
   public BufferedImage matToBufferedImage(Mat matrix) {  
     int cols = matrix.cols();  
     int rows = matrix.rows();  
     int elemSize = (int)matrix.elemSize();  
     byte[] data = new byte[cols * rows * elemSize];  
     int type;  
     matrix.get(0, 0, data);  
     switch (matrix.channels()) {  
       case 1:  
         type = BufferedImage.TYPE_BYTE_GRAY;  
       case 3:  
         type = BufferedImage.TYPE_3BYTE_BGR;  
         // bgr to rgb  
         byte b;  
         for(int i=0; i<data.length; i=i+3) {  
           b = data[i];  
           data[i] = data[i+2];  
           data[i+2] = b;  
         return null;  
     BufferedImage image2 = new BufferedImage(cols, rows, type);  
     image2.getRaster().setDataElements(0, 0, cols, rows, data);  
     return image2;  
   protected void paintComponent(Graphics g){  
      //BufferedImage temp=new BufferedImage(640, 480, BufferedImage.TYPE_3BYTE_BGR);  
      BufferedImage temp=getimage();  
      //Graphics2D g2 = (Graphics2D)g;
      if( temp != null)
        g.drawImage(temp,10,10,temp.getWidth(),temp.getHeight(), this);  
 public class ball {  
   public static void main(String arg[]){  
     // Load the native library.  
     // It is better to group all frames together so cut and paste to  
     // create more frames is easier  
     JFrame frame1 = new JFrame("Camera");  
     frame1.setBounds(0, 0, frame1.getWidth(), frame1.getHeight());  
     Panel panel1 = new Panel();  
     JFrame frame2 = new JFrame("HSV");  
     frame2.setBounds(300,100, frame2.getWidth()+300, 100+frame2.getHeight());  
     Panel panel2 = new Panel();  
     JFrame frame3 = new JFrame("S,V Distance");  
     frame3.setBounds(600,200, frame3.getWidth()+600, 200+frame3.getHeight());  
     Panel panel3 = new Panel();  
     JFrame frame4 = new JFrame("Threshold");  
     frame4.setBounds(900,300, frame3.getWidth()+900, 300+frame3.getHeight());  
     Panel panel4 = new Panel();  
     //-- 2. Read the video stream  
     VideoCapture capture =new VideoCapture(0);  
     Mat webcam_image=new Mat();  
     Mat hsv_image=new Mat();  
     Mat thresholded=new Mat();  
     Mat thresholded2=new Mat();;  
     Mat array255=new Mat(webcam_image.height(),webcam_image.width(),CvType.CV_8UC1);  
     array255.setTo(new Scalar(255));  
     /*Mat S=new Mat();  
     S.ones(new Size(hsv_image.width(),hsv_image.height()),CvType.CV_8UC1);  
     Mat V=new Mat();  
     V.ones(new Size(hsv_image.width(),hsv_image.height()),CvType.CV_8UC1);  
         Mat H=new Mat();  
     H.ones(new Size(hsv_image.width(),hsv_image.height()),CvType.CV_8UC1);*/  
     Mat distance=new Mat(webcam_image.height(),webcam_image.width(),CvType.CV_8UC1);  
     //new Mat();//new Size(webcam_image.width(),webcam_image.height()),CvType.CV_8UC1);  
     List<Mat> lhsv = new ArrayList<Mat>(3);      
     Mat circles = new Mat(); // No need (and don't know how) to initialize it.  
                  // The function later will do it... (to a 1*N*CV_32FC3)  
     Scalar hsv_min = new Scalar(0, 50, 50, 0);  
     Scalar hsv_max = new Scalar(6, 255, 255, 0);  
     Scalar hsv_min2 = new Scalar(175, 50, 50, 0);  
     Scalar hsv_max2 = new Scalar(179, 255, 255, 0);  
     double[] data=new double[3];  
     if( capture.isOpened())  
      while( true )  
      { ;  
        if( !webcam_image.empty() )  
          // One way to select a range of colors by Hue  
          Imgproc.cvtColor(webcam_image, hsv_image, Imgproc.COLOR_BGR2HSV);  
          Core.inRange(hsv_image, hsv_min, hsv_max, thresholded);           
          Core.inRange(hsv_image, hsv_min2, hsv_max2, thresholded2);  
          Core.bitwise_or(thresholded, thresholded2, thresholded);  
          // Notice that the thresholds don't really work as a "distance"  
          // Ideally we would like to cut the image by hue and then pick just  
          // the area where S combined V are largest.  
          // Strictly speaking, this would be something like sqrt((255-S)^2+(255-V)^2)>Range  
          // But if we want to be "faster" we can do just (255-S)+(255-V)>Range  
          // Or otherwise 510-S-V>Range  
          // Anyhow, we do the following... Will see how fast it goes...  
          Core.split(hsv_image, lhsv); // We get 3 2D one channel Mats  
          Mat S = lhsv.get(1);  
          Mat V = lhsv.get(2);  
          Core.subtract(array255, S, S);  
          Core.subtract(array255, V, V);  
          S.convertTo(S, CvType.CV_32F);  
          V.convertTo(V, CvType.CV_32F);  
          Core.magnitude(S, V, distance);  
          Core.inRange(distance,new Scalar(0.0), new Scalar(200.0), thresholded2);  
          Core.bitwise_and(thresholded, thresholded2, thresholded);  
          // Apply the Hough Transform to find the circles  
          Imgproc.GaussianBlur(thresholded, thresholded, new Size(9,9),0,0);  
          Imgproc.HoughCircles(thresholded, circles, Imgproc.CV_HOUGH_GRADIENT, 2, thresholded.height()/4, 500, 50, 0, 0);   
          //Imgproc.Canny(thresholded, thresholded, 500, 250);  
          //-- 4. Add some info to the image  
          Core.line(webcam_image, new Point(150,50), new Point(202,200), new Scalar(100,10,10)/*CV_BGR(100,10,10)*/, 3);  
, new Point(210,210), 10, new Scalar(100,10,10),3);  
          data=webcam_image.get(210, 210);  
          Core.putText(webcam_image,String.format("("+String.valueOf(data[0])+","+String.valueOf(data[1])+","+String.valueOf(data[2])+")"),new Point(30, 30) , 3 //FONT_HERSHEY_SCRIPT_SIMPLEX  
               ,1.0,new Scalar(100,10,10,255),3);  
          //int cols = circles.cols();  
           int rows = circles.rows();  
           int elemSize = (int)circles.elemSize(); // Returns 12 (3 * 4bytes in a float)  
           float[] data2 = new float[rows * elemSize/4];  
           if (data2.length>0){  
             circles.get(0, 0, data2); // Points to the first element and reads the whole thing  
                           // into data2  
             for(int i=0; i<data2.length; i=i+3) {  
               Point center= new Point(data2[i], data2[i+1]);  
               //Core.ellipse( this, center, new Size( rect.width*0.5, rect.height*0.5), 0, 0, 360, new Scalar( 255, 0, 255 ), 4, 8, 0 );  
               Core.ellipse( webcam_image, center, new Size((double)data2[i+2], (double)data2[i+2]), 0, 0, 360, new Scalar( 255, 0, 255 ), 4, 8, 0 );  
          Core.line(hsv_image, new Point(150,50), new Point(202,200), new Scalar(100,10,10)/*CV_BGR(100,10,10)*/, 3);  
, new Point(210,210), 10, new Scalar(100,10,10),3);  
          data=hsv_image.get(210, 210);  
          Core.putText(hsv_image,String.format("("+String.valueOf(data[0])+","+String.valueOf(data[1])+","+String.valueOf(data[2])+")"),new Point(30, 30) , 3 //FONT_HERSHEY_SCRIPT_SIMPLEX  
               ,1.0,new Scalar(100,10,10,255),3);  
          distance.convertTo(distance, CvType.CV_8UC1);  
          Core.line(distance, new Point(150,50), new Point(202,200), new Scalar(100)/*CV_BGR(100,10,10)*/, 3);  
, new Point(210,210), 10, new Scalar(100),3);  
          data=(double[])distance.get(210, 210);  
          Core.putText(distance,String.format("("+String.valueOf(data[0])+")"),new Point(30, 30) , 3 //FONT_HERSHEY_SCRIPT_SIMPLEX  
               ,1.0,new Scalar(100),3);   
          //-- 5. Display the image  
            //distance.convertTo(distance, CvType.CV_8UC1);  
           System.out.println(" --(!) No captured frame -- Break!");  

PS.: Other examples on the same exercise:
  1. A lot of the learning came from this one. Unfortunately no java or deep level of explanation... so, took me a while to make it work... 
  2. Different one which I didn't use, but may be useful...
  3. And one more.
  4. In this example the software will give you the angle set by the line between two cursors of different colors and the horizontal.
PS2.: Click here to see the index of these series of posts on OpenCV


  1. Hello,

    thanks for this post.
    Do you know how to get X and Y coordinates for tracking object in Java? I don't know how to do it this with Moments class.


  2. Hi,

    Thanx for this guide, but I've a question for you: how can I choose the color to detect? Where do I find the interval of color; or anything else?


  3. Hi,

    I am not sure I understand your question... Is your problem to know what is the numeric value of the color you want to detect? If so, there are two things you can use:
    1. In the code (see my pics), I have included a reporting in the screen of the values for a given pixel (in the circle). You can move your camera till the position of the pixel is in the color that you want to know...
    2. You can follow this link: to understand what all these numbers mean...

    If your question is about how in the code I am choosing the color to detect, you can see the instruction "Core.inRange(distance,new Scalar(0.0), new Scalar(200.0), thresholded2);" and the explanation in the points #4 and #5 of my post.

    Does this answer your question?

  4. Hi ,

    The exception is coming from the paint function because your image object reference is null the first time.
    Just do the following :
    if( temp != null)
    g.drawImage(temp,10,10,temp.getWidth(),temp.getHeight(), this);
    and there will be no exception anymore :-)

    Regards ,

    1. Hey Abdel! Thanks a lot!! Will update the code when I get a chance! :)

    2. Done and working! Thanks again! :)

  5. Hi!

    I've seen that in the example code there are TWO intervals of hsv colors..The first came out from the HSV PANEL as you answered to Anonymous in a previous comment, but how do I find the second interval?


    1. Hi,

      I think you are referring to:
      Scalar hsv_min = new Scalar(0, 50, 50, 0);
      Scalar hsv_max = new Scalar(6, 255, 255, 0);
      Scalar hsv_min2 = new Scalar(175, 50, 50, 0);
      Scalar hsv_max2 = new Scalar(179, 255, 255, 0);

      If you look at the Hue bar in you can see that the redish colors are actually in both extremes of the bar (left and right). So, I call red anything with a Hue below 6 (from 0 to 6) or bigger than 175 (and smaller than 179 which is anyhow max Hue). Then I "OR" both results into one single array with Core.bitwise_or(thresholded, thresholded2, thresholded);

      Hope this is what you were asking...

    2. I saw the Hue bar and i noticed what you said about red, but how can I find these two intervals for the colors Blue, Green and Yellow? I want to track red, blue, green and yellow circles but I'm in trouble with these last three colors..Could you give me them please? :) I'm doing a lot of tries, but I'm not finding any good result!

      Hope you could help me again, thanx a lot!

    3. Not sure I follow you... For the Blue or the Green or Yellow, you actually don't have two intervals. You only have a min and a max as the bar does not wrap around. So, for instance, as explained on the link if you pick Hue around 129 (say min hue=120 and max hue=135), you got the right envelop for blue... For green is ~61. Yellow around 30. Maybe your issue is not on the Hue but on the Saturation or Value?

      Or a different way, if you use my software and point the circle on the screen to an object that you want to identify/segment by color, it'll tell you what the HSV values are... Can you tell me what exactly is not working? Does the red alone work?

  6. Hi,

    Great Tutorial. I wanted to know how to find the X and Y coordinates of the object you are tracking.

    1. Thank you!! Not sure I follow your question... It is not very easy to see, but the final center coordinates of the circle around the detected ball are explicit in the code above in the instruction: Point center= new Point(data2[i], data2[i+1]);. data2[i] is the X, and data2[i+1] is Y. Hope it helps!

    2. Thanks for the help. For some reason when the program is tracking the ball it doesn't track the whole thing but different spots on the ball. Is there some solution to this.

    3. OH! I see what you mean! Yes, you are right. I never finished improving that. It felt like a lot of tweaking but I thought I would go back with more time to get a solid solution (never got the time, though). I think it comes down to the HoughCircles parameters, which may include too the Canny (edge detection) that happens within the HoughCircles.

      For one side, you have sometimes, false positives outside the ball. As you see in the black and white bottom picture above (after Canny edge detection), there are other red areas that got edged around and as such, they are candidates to be detected as circles by the HoughCircles algo.

      For the same reason, as there also edges running inside the ball real edges, you may have the smaller spots in the ball, as you describe.

      If I remember right, I did play with some of the parameters, to try to filter this out. For instance, with minimum acceptable radius, etc... It worked well but then, when you had the ball further away (smaller) from the camera, it was failing for being too small, below that threshold. Again, it felt like a lot of tweaking...

      There may be smarter strategies. For instance, assume that the ball is the biggest circle object in the image and go from there. Or run the algorithms with different parameters, not just one set, and learn from all the results assuming, for instance, that you know for sure that the ball is in the image. I guess that one has to make assumptions to really cut the "noise" down.

      Anyhow, sorry is not perfect. If you or anybody finds a method to get this better, please let me know! :)

    4. I was able to make the tracking more accurate by using contouring. I had a question on how to do multiple color tracking with opencv and java.

    5. Hi Chickenator! Not familiar with contouring... Quickly googled it...Is that something that you apply between the gaussianBlur filter and the HoughCircles?

      And is the second ("multiple color tracking...") a question? Not sure I follow...

      And Happy New Year! :)

    6. Yeah I added contouring after the HoughCirlces. I wanted to know if you have ever tried to do multiple color tracking. I am trying one way right know but it is really slow.

      here is the code I used for contouring:
      Imgproc.findContours(thresholded, contours, thresholded2, Imgproc.RETR_LIST,Imgproc.CHAIN_APPROX_SIMPLE);
      Imgproc.drawContours(webcam_image, contours, -1, new Scalar(0, 0, 0));

    7. When I try to draw a line using Core.line() the line disappears when I want to continue drawing

    8. the line is supposed to follow the object I am tracking

    9. Sorry, I've been traveling and just got to this... Probably you solved it already but I thought at least I would say something. I have no experience with contouring/multiple color tracking. The typical trick to expedite things is to use smaller resolution images. That's a general one (duh!) but anyhow, thought I would mentione it... About the core.line, it's strange. I use that along my code... Do those work for you?

  7. It works like a charm m8!
    Thumbs up and Thanks! :)


  8. Hi, great tutorial.

    I am trying to detect and track an orange ping pong ball.

    I need some directions on how to set parameters in the code.

    Placing the orange tennis ball in the spot I can read on the various windows the following values:

    S,V Distance 55
    Camera 0,65,175
    HSV 13,255,80

    Can You explain me what values and where do I have to change the code?

    Sorry for my poor understanding, I only have competence on robots, no competence in OpenCV/Video/etc.

    Thank a lot


    1. Hi,

      I can try to help you to the extend of what I remember :)
      To understand what HSV means, check my other post:
      It is actually very easy (I freaked out first time I saw this but give it 5 min and you'll get it). Get the software on the link. Will be very helpful to understand HSV...
      Just for you :) I checked your values and certainly it is orange. Basically the H 13 will tell you is orange-like. The S how really orange is vs "diluted" (in your case it is really orange, max out at 255). And 80 is the brightness. 80 is kind of dark, so, looks more like maroon.
      Now you need to have your software decide how far a color value is from here to be considered your ball color. The first line doing that is the "inRange". InRange looks between two thresholds (min and max). All 3 HSV values of a point got to be between the two thresholds to be considered in range (and get a "1"). Thresholds are set in the 4 lines like Scalar hsv_min = new Scalar(0, 50, 50, 0);
      You will ask why 4 and not 2? Well, the HSV wheel kind of wraps around in the red (red can be a very low or very high H number). So, if you are trying to detect a red, there are 2 "ranges" where the red can belong to. But if you are in green, then one is enough... In that case, set the second range to be the same as the first.
      Then there is a line that just ORs both answers.
      Till here, you got something that is now "orangy". That would probably do. But if you want to pick something that really looks like orange, then the next step (distance) will get you there. Not saying that you need this step as an orange ball may not look orange in your screen (depending on lighting), but hey, your call...
      So, to understand that, get the software on the link and look at the big square on the bottom left. The more to the top right of that square, the more orange you are. Read my HSV post to understand the rest.
      Follow the code to see how I compute the distance from an (S,V) coordinate to the top right corner. Finally, you set the distance threshold in the same way as before, with the inRange line: Core.inRange(distance,new Scalar(0.0), new Scalar(200.0), thresholded2); As long as your distance is below 200 you good.

      Hope that works for you. You could choose to skip the distance part. For all of this it is best to look at the intermediate results and see how much false positives or how many negatives for a true value you get and tweak things...

      Selection done this way is not everything. Later I use other techniques to narrow down things (like detecting a circle). And there could be others that I didn't go into (like temporal analysis, like for instance, a ball can't jump from one side to the other of the image immediately, etc...)

      Good luck!

  9. This comment has been removed by the author.

  10. Hello, it's a great tutorial. But I'm wondering that how do I extract the amount of detected circles by houghCircles?Or if it can only detect one circle. Can it detect multi-circles?

  11. Exception in thread "main" java.lang.UnsatisfiedLinkError: no opencv_java245 in java.library.path
    How to solve this error??

  12. Exception in thread "main" java.lang.UnsatisfiedLinkError: no opencv_java245 in java.library.path
    How to solve this error??

    1. CHANGE TO THIS, System.loadLibrary(Core.NATIVE_LIBRARY_NAME);

    2. Hi Raghav! Thank you for contributing! It's been so long that I did anything with this that I wouldn't be able to answer anything now. Appreciate you going through the effort. Cheers!