OpenCV: How to get started and index of posts

So, you want to do some cool image processing... so, here is a very basic explanation on how to get started with link/index to the stuff I posted so far.

Where to start

Basically OpenCV is a set of nice libraries that will highly expedite a lot of your work. See more detail here. You will need to call them from your program, say in C/C++ or Java.

To develop the program you will need an IDE (integrated development environment) which takes care of editing (more than a text editor, with nice support features), and allows you to "plug in", compilers, debuggers, etc... so that you can run everything from there. In my case, I am using Eclipse. Notice that you could use a different IDE. During configuration of the IDE, you tell it what compiler to use, etc... So that when you press the compile button, Eclipse calls it to generate the object file... I think you get the point...

So, install Eclipse if you don't have it. Of course, you could choose a different IDE, but sorry, no experience on that, and honestly, I have no complains about Eclipse. Great tool. In my case, I did the installation when I was planning to use it with Qt (a set of widgets for something that I was doing completely unrelated). Still, the initial portion of the installation is the same. You can see my installation experience here.

So, what do I choose, C/C++ or Java?

Not an expert on this but I believe that a lot of the initial stuff on OpenCV was done for C development. Nevertheless, I think that almost everything in OpenCV has a port to Java already. I.e., there are packages in Java equivalent to libraries in C. So, probably depends more on what language you feel comfortable with.

One thing I was thinking (and I was wrong) is that if I want to use OpenCV in Android, I better get a handle of the Java version, as that is the language to develop in that platform (see this overview on Android programming environment). Nevertheless, that it's not that truth. In Android development, there is something called the NDK which allows you to call pieces of C code (native code) from the java code (see more info here on the NDK). The funny part is that in the end, as OpenCV is pretty processing intensive, you may want to use the NDK for the OpenCV routines (native code runs much faster than java code, if you do things properly...). Bottom line, I went down the path of working with OpenCV in Java, but probably I (and you, if you prefer) could have saved part of that work, as I was more familiar with C. But as I went down that path, I end-up with Eclipse supporting both languages (that is done with the "views"). So, to install each of the languages to work with OpenCV in a PC CPU, follow:

1. Installation to work with C/C++
2. Installation to work with Java

To develop for an Android machine, follow this.

Note: The 3 links above are in the order of what I historically did on my case. I.e., the 3 steps may not be completely independent of each other. For instance, something got installed in step 2 that is needed for Android, but I never realized and if I had done only the third, it would not have worked. Please let me know if you got any trouble...

Coding examples

So, now that you got it installed, let's do some coding:

1. As part of the installation of OpenCV in Java (the same link as above) we did a first tutorial where we detect faces in an image file in a PC.
2. Creating windows in Java and drawing in them. We strip everything to get familiar with the java drawing/workings. There is no OpenCV here. I read an image file from disk and display it. No detection yet.
3. Next, we create windows and capture webcam, i.e., not from a file, like in #1. Although we do not do any image processing, we do use OpenCV structures to be ready for the next step (processing). 
4. One of the keys on that post is how to take a Mat structure from OpenCV and pass it to BufferedImage, for display in Java. I created a post just for that.
5. And into image processing, we now detect the faces on the webcam stream. Notice that this is very similar (a port to Java) of the original C tutorial
6. Then we move into detecting a ball in the image. Part of the detection is based on color. So, you got to understand the color space. See this.
7. And then we track the ball with the PC camera/CPU.
8. Then we start moving into Android with a first app using OpenCV.
9. Here we track faces with OpenCV in Android, using the smart phone camera. And here I use the Android SDK for the same thing.
10. Using the face tracking I created a 3D display, which seems to be a similar technique to what Amazon is going to be using on their phone.
11. And finally, we port the tracking of the ball to Android, i.e., we use the smart phone camera/CPU.
12. The last final thing that I had in plan but haven't done yet is to actually port it to Android but have the OpenCV CPU intensive routines done in a C library, with the NDK.

For the next posts I am going to be concentrating in Android development, not related to OpenCV, but I'll be back :)

Cheers!!

Copy project in Eclipse

The following is an extension of this and is center around Android/Java but gives you an idea...
Here I wanted to create an app based on the second tutorial that comes with OpenCV (see more here). So, I wanted to copy that and start working on it. This is what I did:

1. Ctrl+C, Ctrl+V on the project that you want to copy (make sure the project is open)
2. Enter new project name when prompted.
3. Change the package name on the AndroidManifes.xml to whatever you want it to be. Click save.
4. On the "Do you want to update your launch configurations?" say "yes".
5. Right click on the package name, within src folder, and select refactor-rename. Enter new name. If you click in Preview it may show you a warning (package already exists in gen folder...). I ignored it (continue).
6. Change the name of the application in the AndroidManifest. The real name may actually be in the strings.xml, so change it there too...
7. Right click on the class name, inside the package, and select refactor-rename. Change main class name. It will update also any reference to it inside the code.
8. Note: you may get an error on the link to native library. Just click continue and fix it later... Hint: the easiest way to fix errors is clicking on them and picking from the options they give you.
9. Now if you click Run, select Android Application, should work...

Update 1: I did similar thing now for a C project I was using in ODE. Steps 1 and 2 are the same. Then in properties-Run/Debug settings delete the old launch configuration and enter a new one for the new project.

Update 2: After I copy the ODE project (ctrl+C, ctrl+V, rename the cpp) the whole file is marked with red warnings/errors. It is solved by going to properties --> C/C++ General --> Indexer and then mark Enable project specific settings, Enable indexer, Use active build configuration.

Cheers!

Android Native Development

I decided to do a quick Android app that would do the addition of two numbers, with one version fully in java, and the second where the UI is Java but the addition itself is native (in C).

In the Java code, the key things I learn were (see code):

1. How to reference the elements in the UI
2. Use of EditText and TextView.
3. How to pass from string to numbers/int or vice versa.

The code is:

 package com.example.adder;  
 import android.os.Bundle;  
 import android.app.Activity;  
 import android.view.Menu;  
 import android.view.View;  
 import android.widget.Button;  
 import android.widget.EditText;  
 import android.widget.TextView;  
 public class MyAdderActivity extends Activity {  
      private EditText mnumberA, mnumberB;  
      private TextView mResult;  
      private Button madd;  
      @Override  
      protected void onCreate(Bundle savedInstanceState) {  
           super.onCreate(savedInstanceState);  
           setContentView(R.layout.activity_my_adder);  
           mResult=(TextView)findViewById(R.id.adderResult);  
           mnumberA=(EditText)this.findViewById(R.id.editNumberA);  
           mnumberB=(EditText)this.findViewById(R.id.editNumberB);  
           madd=(Button)this.findViewById(R.id.button1);  
           madd.setOnClickListener(new View.OnClickListener() {  
                @Override  
                public void onClick(View v) {  
                     // TODO Auto-generated method stub  
                     int A=Integer.parseInt(mnumberA.getText().toString());  
                     int B=Integer.parseInt(mnumberB.getText().toString());  
                     mResult.setText(Integer.toString(A+B));  
                }  
           });  
      }  
      @Override  
      public boolean onCreateOptionsMenu(Menu menu) {  
           // Inflate the menu; this adds items to the action bar if it is present.  
           getMenuInflater().inflate(R.menu.my_adder, menu);  
           return true;  
      }  
 }  

In the native case, we replace the addition (yeah, I know, just the addition, but it is just to practice) by a call in C to add the two numbers. We talk a bit about this in here (points 8 and 9). See references in the end, but here is my short explanation:

1. Create your java Android project, as usual (like on the previous case). Say we call the project MyProject. This will appear in Eclipse as the "top name" in your branch of the workspace and as the name of the directory, within your workspace folder.
2. The java source will be in a package. Something like com.myname.mypackage. This is in fact the directory tree structure under MyProject/src where Eclipse places the .java file.
3. Inside your main activity class (say MyActivity) include/load the native (C) library you are going to create with something like "static { System.loadLibrary("MyLib"); }. MyLib is the .dll (Windows) or .so (Unix/Android). As we are centered around Android, it'll be the second. By the way, as we will see below, the actual file name of the library will be "libMyLib.so", but the first "lib" and the ".so" are not mentioned.
4. Declare within MyActivity class a native method (java) that will be used to call the native function (C). Something like "private (or public) native [return type] MyMethod([arguments...]);". For instance:  "private native int add_numbers(int A, int B);" and call it in your java code, like usual... As usual, if you declared it public, then you will be able to call it from outside your class.
5. Create a jni directory in your project. You can do it from inside Eclipse, or outside, and do "refresh" for Eclipse to find it.
6. Put inside it the .c sources (New-->File-->whatever.c...). Somehow when I used .cpp it was not working (I'll get a "java.lang.UnsatisfiedLinkError exception !" at run time because it does not find the functions).
7. The wording within the c code is not "straighforward". Basically it is telling the jni and the NDK how to create the library so that the methods calls refer to these c functions. Here are the main things:
1. Add "#include <jni.h>" at the beginning. Note: JNI stands for Java Native Interface and  it is a programming framework that enables Java code running in a Java Virtual Machine (JVM) to call, and to be called by, native applications (programs specific to a hardware and operating system platform) and libraries written in other languages such as C, C++ and assembly
2. The name of the function needs to be declared as "Java"+{package name}+{class name}+{method name}, using "_" between words. For instance, for our example, it would be "Java_com_myname_mypackage_MyActivity_MyMethod"
3. As arguments, at the very least (even if the C function does not need any), we will need to write "JNIEnv *env, jobject thisObj".
4. The rest of the arguments need to be the usual ones that the C function would have. Nevertheless, as java types and C types are not 100% compatible, look at the links below to see what to do in each case. On my example, this is straightforward (see below). Notice that the arguments use java types, not C types. Same thing for returned results...
8. Add the Android.mk which basically tells the ndk-build (see next) what to build and how to name it. Basically:
   LOCAL_PATH := $(call my-dir)
   include $(CLEAR_VARS)
   LOCAL_MODULE    := MyLib
   LOCAL_SRC_FILES := whatever.c
   include $(BUILD_SHARED_LIBRARY)
   The two bold lines are the ones changing most of the time. Notice that LOCAL_MODULE is the final name of the library, although the real file name is slightly different (libMyLib.so).
9. To then compile and create the library, in Windows Explorer go to the project folder, SHIFT+right click --> Open command window here. Then type {NDK path}+ndk-build. If you had previous builds that you want to clean up, use "ndk-build clean". Note: this may be doable within Eclipse (allowing you to clean up compile errors...), but have not experimented yet with it.
10. Now you should be able to just run the java project (launch your app).
11. Watch out when you launch your app, as some times the previous launch may be still active (go to the task in your phone and kill it...) or may have the old library in there and don't notice that it is missing (in that case, uninstall the app). I know... this shouldn't be the case, but something like that just happened to me...

Here is my code for the adder case:
AdderJNI.java

 package com.example.jniadder;  
 import android.os.Bundle;  
 import android.app.Activity;  
 import android.view.Menu;  
 import android.view.View;  
 import android.widget.Button;  
 import android.widget.EditText;  
 import android.widget.TextView;  
 public class AdderJNI extends Activity {  
      private EditText mnumberA, mnumberB;  
      private TextView mResult;  
      private Button madd;  
      @Override  
      protected void onCreate(Bundle savedInstanceState) {  
           super.onCreate(savedInstanceState);  
           setContentView(R.layout.activity_adder_jni);  
           mResult=(TextView)findViewById(R.id.adderResult);  
           mnumberA=(EditText)this.findViewById(R.id.editNumberA);  
           mnumberB=(EditText)this.findViewById(R.id.editNumberB);  
           madd=(Button)this.findViewById(R.id.button1);  
           mResult.setText( stringFromJNI() );  
           madd.setOnClickListener(new View.OnClickListener() {  
                @Override  
                public void onClick(View v) {  
                     // TODO Auto-generated method stub  
                     //setView R.layout.adderResult="3";  
                     int A=Integer.parseInt(mnumberA.getText().toString());  
                     int B=Integer.parseInt(mnumberB.getText().toString());  
                     //mResult.setText(Integer.toString(A+B));                    // Just using Java  
                     mResult.setText(Integer.toString(addnumbers(A,B)));      // Using native  
                }  
           });  
      }  
      @Override  
      public boolean onCreateOptionsMenu(Menu menu) {  
           // Inflate the menu; this adds items to the action bar if it is present.  
           getMenuInflater().inflate(R.menu.adder_jni, menu);  
           return true;  
      }  
      private native int addnumbers(int A,int B);  
      public native String stringFromJNI();  
      static {  
     System.loadLibrary("JNIadder2");  
   }  
 }  

JNIlib.c

 #include <jni.h>  
 #include <string.h>  
 jstring  
 Java_com_example_jniadder_AdderJNI_stringFromJNI( JNIEnv* env,  
                                                             jobject thisObj )  
 {  
   return (*env)->NewStringUTF(env, "Your Result");  
 }  
 jint  
 Java_com_example_jniadder_AdderJNI_addnumbers(JNIEnv *env, jobject thisObj, jint n1, jint n2)  
 {  
   // jint is mapped to int  
   return (n1+n2);  
 }  

And a screenshot of eclipse with the Android.mk:

References:
These can be a bit confusing as some times they don't explain what is what, they refer to java native development, not necessarily related to Android NDK (slight change of notation) or they are a bit obsolete (refer to previous versions):

1. Look  in the NDK folder for a "docs" folder, and open the file OVERVIEW.html. Nice explanation...
2. There is a nice video tutorial here.  
3. Coderwall 
4. Java Native Interface (very detailed)
5. Another tutorial, lots of details, but a bit less than the previous one.

PS.: Please, click here to see an index of other posts on Android.

OpenCV in Android - Quick install guide

I found a good explanation of OpenCV in Android and the inner workings of calling this library... I feel it is a must read before you start. Will save you work/time.

Then we go into the tutorial. Top level, check this first. Then you need to download/add the OpenCV for Android library. Then create a new workspace, this library should be one of the projects on the workspace, and any OpenCV Android project will refer to that project. When you run it, your apk gets loaded and a Manager (that gets downloaded the first time, if you don't have it in your phone), loads the library apk.

Still trying to wrap my brain around this, but here are the details of how I did till I got to run one of the sample demos (puzzle15). Follow the tutorial and if that doesn't work, read below to see my experience...

We have stuff installed from the past (like Eclipse, Basic4Android, java...). So, we need to figure out how to make it work without reinstalling or creating a parallel copy of all this (maybe only alternative).

 Sun JDK 6 or higher:

1. I seem to have 7 (jdk1.7.0_03). JDK stands for Java Development Kit. Check in Program Files/Java...
2. I also have JRE6 and JRE7. JRE stands for Java Runtime Environment, which contains  everything required to run Java applications on your system. Do I really need both?

Android SDK:

This is nicely documented by the Android folks. Just look around, as I guess pages may change, but I found this and this.

1. Got an old installation (about a year ago) that I have been leveraging with Basic4Android. So, I launched my Android SDK Manager and selected the updates. The screen shows Android 4.2.2 (API17) empty when I had Android 4.1 (API16), indicating possible updates. I clicked the updates (not a new install) and the Manager suggested other dependencies I needed to get...
2. Nevertheless, I think I still should install the new API17 (the updates just update the old API16). This is what the SDK Manager looks like:
   
3. To develop in Eclipse, I need the ADT plug-in. Follow the instructions there, but remember that if you want to develop in Android with C++ (which is likely for any kind of intensive processing), you should get also the NDK. This is how my Eclipse - Help - About Eclipse looks like...
   
4. Get the NDK. Follow links... Installation is just to unzip anywhere...
5. At this moment, things seem to look very complicated (maybe because of the NDK), so, we go build a simple first app in Android, nothing to do with OpenCV. Just to make sure all the above installation is working... If you do not need this, jump to #8.
6. While doing this, Eclipse didn't detect the HTC One at launch. Also, if you run the adb (Android Debug Bridge, go to command prompt and type adb devices) nothing shows up. In the Windows Device Manager the driver shows nicely, but it is NOT the one we need. Found a solution here. You should see at the TOP of the device manager "Android USB devices". If not, keep uninstalling driver and installing the right one till it does show up like that. What is the right one it is tough to say. Initially it worked when I got the USB Composite Device. But lately works with "My HTC". Key seems to be to have an "Android USB devices" at the top of the list. Now the adb  will see it... Restart Eclipse and now the launch should show the phone! Note, during this whole thing, I uninstalled HTC sync (but I don't think it had nothing to do with it). Other things that I am not sure they can work are here and here. A good summary of things to try is actually on the original tutorial we are following.
   
   
7. Even after knowing the app launches, you may want to finish the tutorial. It is very useful and a nicely explained to quickly get a background on Android app development, if you didn't have one... Just as a quick note, it uses a intent to create the screen with the result of an app, so, that when you click the "back" key (which I think that is what he calls "up") it can go back to the app.
8. Then we take this further by trying to use the NDK. We will go for the Hello-jni example which works right away. Note: JNI stands for Java Native Interface and it is a programming framework that enables Java code running in a Java Virtual Machine (JVM) to call, and to be called by, native applications (programs specific to a hardware and operating system platform) and libraries written in other languages such as C, C++ and assembly. To understand what is all this NDK about, look  in the NDK folder for a "docs" folder, and open the file OVERVIEW.html. Nice explanation.
9. We do also the 2nd example, a fully native app. I.e., no java involved, only the .C with the NDK. Notice how the src file is empty.
   
    Notice that the original sample source only had the C stuff in jni folder and the res folder.
   
   Most of the final stuff (in "gen", etc...) is generated by Eclipse when creating the project from an existing source. Before running NDK... The ndk-build will create the binary of our native libraries and place them in the right places for Eclipse to use them...
10. Anyhow, back to our OpenCV installation tutorial!! The above NDK tutorial shows how to include native code in your app. But you can build the native part of your application from within Eclipse. That is where this tutorial continues...
11. It is a bit confusing, though, in some way... Maybe I missed it, but so far, we have not downloaded the OpenCV for Android SDK! Follow these instructions... Note: there are Android OpenCV samples on the original OpenCV package (not the specific one for Android but the one I was using in my previous examples), under "samples\android", but the tutorial can't assume that...
12. The tutorial then goes into importing those samples. Either of the two tutorials (Introduction to Android Development or OpenCV Android SDK) does this, but it is not clear if we need to add the C Nature to the samples that don't even have a jni. I am not 100% it will even allow you to do it... but seems to ask for that.
13. So, simply do the import to the point we see the projects in the workspace but nothing else. And we try to load puzzle15. Lots of errors on the package from OpenCV. Left click on the error and one of the solutions (all the way to the bottom) sounds like "fix project setup" or something like that. It will suggest to add OpenCV library - 2.4.6 to the build path. Notice that it is on the same workspace. These errors go away:
    
14. Nevertheless, I was still having lots of errors, some of them on the OpenCV library project. As the other projects link to this one, they didn't run. So, after running the SDK manager and updating everything (which was just some minor stuff as I recently installed everything), I deleted the projects (not from the drive) and run again the import. Now everything looks clean... Don't ask me why...
15. Still, when I launch the project, I get an error: "Could not find OpenCV-2.3.1.apk!". This was solved following this.
16. So, now, when you launch it, the installer realizes that is missing a package and gives you the following notice in the phone "Package not found. Do you want to install it?" Answer yes and that launches Google play and looks for OpenCV Manager. Install it as any other app... The Manager will get the right openCV apk for your hardware.
17. Now your app should run!!

In the next post we will create our own app...

PS.: Click here to see the index of these series of posts on OpenCV

Programming in Android - Top Level

Programming in Android:

1. Eclipse: IDE (Integrated Development Environment). The "editor". 
2. Eclipse plug-ins: the brains (compilers, linkers...) and the customization of the interface to that particular development environment (for instance, adding debugging view, etc...). Usually Eclipse comes in packages of plug-ins targeting different solutions. See what is on each package. But some times, like on this case, you need to get few more on your own. For Android development, you will need:
1. As the whole Android development is based on Java, you will need the JDT plug-in. If when you got Eclipse, you got the Java package, then no need...
2. For Android specific, you will need also the ADT plug-in.
3. If you want to develop native applications or libraries (in C/C++) you will need CDT plug-in. You may have got it already if you were developing in C/C++ before, with Eclipse...
4. Finally, also for native development, you will need also the NDK (Native Development Kit) plug-in.
3. To run java on your PC, you will need the JRE (Java Runtime Environment).
4. JDK: Java Development Kit (set of Java libraries).
5. Android SDK: Android Software Development Kit. Set of libraries, but also tools to help in the development, like emulator (AVD, Android Virtual Device), stuff to monitor connected devices (Android Debugging Bridge)...
6. NDK: Android Native Development Kit. Allows you to compile the C/C++ (native) code and be able to link it into your Android app.

In many cases, you will not need any kind of C development, so, you can ignore the NDK stuff. But for OpenCV you may need it.

Cheers!
PS.: Click here to see the index of these series of posts on OpenCV

OpenCV in C/Eclipse - Quick install guide

We are going to setup now OpenCV in Eclipse. For that, we will follow the tutorial in: 
http://docs.opencv.org/doc/tutorials/objdetect/cascade_classifier/cascade_classifier.html#cascade-classifier

We start by creating a project. Follow these instructions.
http://docs.opencv.org/doc/tutorials/introduction/linux_eclipse/linux_eclipse.html

Nevertheless, we got stuck in several places, so, here are the tips. First place was because I was selecting the wrong project, to start with.

 

I was going with GNU Autotools which, it seems, needs to be setup, etc... The right choice is "Executable" and the I picked the MinGW GCC. Found the solution here.

Also got stuck because I didn't know how to properly add the libraries in Eclipse. For details, read this and this. But even with that, I was doing some mistakes, so, here is the basics: 

1. Add the path to the include (header files) directory. In our case it was (C:/eclipse/opencv/build/include). The individual paths are listed in the code, respect to this root... FYI, I did this in the C++ compiler options. Not in the C compiler.
2. Add the path to the lib (compiled libraries) AND the libraries themselves. When adding their name, do not add the extension.
3. Remember to use fwd slash "/" on the paths although I think it is ok because Eclipse will put it between "".
4. Also, watch how the C libraries are defined. I didn't really have to add the path for them, but some examples didn't work when used without the .h. For instance, use #include <stdio.h> and not #include <stdio>.

Here is a screen shot of the whole thing, in case it helps with this or something else:

Notice that if you download the Opencv code of the link above is slightly different than the listed... They have few more includes, like:
#include "opencv2/core/utility.hpp"
#include "opencv2/highgui/highgui_c.h"

But I didn't need those to build.

Not sure why I need to use the x86 >> This may be because we may have selected this version of eclipse because of Qt?? Compiling with the libraries on x64 will not generate an exe but we don't get more errors than the one we get with x86 (make: *** No rule to make target `all'.). This error is weird because everything else works. And anyhow, I shouldn't need to create a rule because we have it in auto. On that sense, looking into the directories, notice that I don't have a makefile. I think that is normal. The tool has that internal but does not share it with us.

Another thing was that the program didn't start because libpencv_highgui245.dll is missing from your computer. That was just the first one that he looked for. All of them were missing.It basically has to do with dynamic linking. The dll is not included in the code (like it would with static), so, at run time needs to look for it and can't find it. See http://qt-project.org/forums/viewthread/12525. I solved by moving the 4 dlls to the debug directory.

GEEEE! Almost there! Just a last one... Remember to get the xmls. I put them in resources, following a procedure like the one described in the java tutorial. I had to add resources/ in front of the text. Works now from run. If you want to execute the exe, you then have to move it to the root of the project with all its dlls...

DONE!

PS.: Click here to see the index of these series of posts on OpenCV