Thursday, August 22, 2013

Basic4Android - Baby Flashcard

This is a very simple application in Basic4Android, a paid tool (~$100), where you can program an Android device in Visual Basic (almost fully compatible). If I understand it correct, it is a wrapper that takes the VB and turns it into java, before running it through the Android SDK and generating the apk.

The app displays images every 5 seconds that pulls from a folder. It also pulls an mp3 with the same name which plays some audio (for instance, the name of the object being displayed). That way, a baby can see an object, say an apple, hear apple, see "apple" written below, and then move randomly to the next object.

The app illustrates a bit of a GUI/display images, access to files, mediaplayer, timer and multithreading.
 'Bits: flash cards present for my one year old nephew  
 '8/18/2013  
 'The program opens randomly a picture from a language directory, and the same name .mp3 file from that directory.  
 'Options eventually will include to select the directory (therefore, several languages can be selected), select the time  
 'the pic is in display, and to mute the voice, so, that the parents can be the ones saying the word. Also we want to have   
 'the option to pass card with the press of a Button.  
 'To avoid accidental exits, as the baby may touch the screen with the hand, the exit of presentation mode (to go into the   
 'menu) works by pressing two buttons at the same time.  
 'Thanks to Gigatron (GTR) as I looked at some of his code for learning...  
 'Now we want the system to cover the screen, no matter what screen and orientation...  
 'Changing things with reference to 100% is not enough as everything is bound by Panel.  
 'See for that the two lines of code below.  
 'Then we refer the position of the objects also respect to the screen/Activity (% notation)  
 'Notice that the notation is top and left. So, 0% means all the way to the left, and all the   
 'way to the top.  
 'Things explained in detail in page 72 of the beginners guide.  
 '  
 'Then, we finally add to it the capability to rotate the screen. For that, all what we need to do is to go   
 'to "Project>>Orientations supported" AND change it To "both" (it was in "landscape").  
 'In other words, if both are allowed, when we rotate the screen, the activity is recreated   
 'with the new reference axis following the position of the tablet. Otherwise, the axis just  
 'follow the orientation.  
 '  
 'We will present an image every number of seconds, with timer, eventually selected by customer, but we will load the  
 'files in the background process.   
 '   
 'With every timer click the UI will make sure the loading process is done, then take the info and move it to a   
 'save place before running again the loading thread. Not sure the copy works for MediaPlayer elements, so, I have two  
 'player objects and I switch from one to the other. Play one while loading the other.  
 'Activity module  
 Sub Process_Globals  
      'These global variables will be declared once when the application starts.  
      'These variables can be accessed from all modules.  
      Dim time As Timer  
 End Sub  
 Sub Globals  
      'These global variables will be redeclared each time the activity is created.  
      'These variables can only be accessed from this module.  
      Dim cnv As Canvas  
      Dim graphpanel As Panel  
      Dim clr As Int       
      Dim Label1 As Label  
      Dim Orientation As String  
      'Directory name to be selected by menu  
      Dim Question_Number,Old_Question_Number As Int  
      Dim FileList As List       
      Dim Card_Name, Card_Name_Buffer As String  
      Dim Card, Card_Buffer As Bitmap  
      Dim SrcRect, DestRect As Rect  
      Dim MP1,MP2 As MediaPlayer 'One is loaded while the other is played  
      Dim MP1_Busy As Boolean  
      'Dim tic As Int 'Number of seconds in each picture  
      Dim LoadFiles As Thread  
 End Sub  
 Sub Activity_Create(FirstTime As Boolean)  
           Activity.LoadLayout("bitsscreen")  'nothing special just graphpanel + 1 imageView in layout  
           graphpanel.Width=100%x  
           graphpanel.Height=100%y  
           Label1.Width=100%x  
           Label1.Height=20%y  
           Label1.TextSize=15%y  
           Label1.Left=50%x-Label1.Width/2  
           Label1.Top=80%y  
           Label1.Gravity=Gravity.CENTER_HORIZONTAL  
           FileList=File.ListFiles("/sdcard/Pictures/bits")  
           cnv.Initialize(graphpanel)  
           MP1.Initialize2("MP")  
           MP2.Initialize2("MP")  
           MP1_Busy=False  
           Load_card  
           LoadFiles.Initialise("GO")  
           time.Initialize("time",1000)  
           time.Enabled=True  
 End Sub  
 Sub Activity_Resume  
   time.Enabled = True  
           time_Tick  
 End Sub  
 Sub Activity_Pause (UserClosed As Boolean)  
   If MP1.IsPlaying Then MP1.Pause  
           If MP2.IsPlaying Then MP2.Pause  
   time.Enabled = False  
 End Sub  
 Sub time_Tick  
      time.Enabled=False   
      Do While LoadFiles.Running==True   
      Loop  
      MP1_Busy=Not(MP1_Busy)  
      Card=Card_Buffer  
      Card_Name=Card_Name_Buffer  
      Old_Question_Number=Question_Number  
      LoadFiles.Start(Me,"Load_card", Null)  
      time.Initialize("time",5000)  
      time.Enabled=True  
      Show_card  
 End Sub  
 Sub Show_card          'This shows whatever image was on the buffer when timer call it  
      Dim ratio As Float   
      Dim Rec As Rect  
      Rec.Initialize(0,0,100%x,100%y)  
      cnv.DrawRect(Rec,Colors.White,True,5dip)  
      Label1.Text=Card_Name       
      SrcRect.initialize(0,0,Card.Width,Card.Height)  
      'Fit picture to the limits without distorting  
      ratio=100%x/Card.Width  
      If (ratio>80%y/Card.Height)     Then  
           ratio=80%y/(2*Card.Height)  
           DestRect.initialize(50%x-Card.Width*ratio,0,50%x+Card.Width*ratio,80%y)  
      Else  
           ratio=ratio/2  
           DestRect.initialize(0,40%y-Card.Height*ratio,100%x,40%y+Card.Height*ratio)  
      End If  
      cnv.DrawBitmap(Card,SrcRect,DestRect)  
  graphpanel.Invalidate ' invalidate panel after drawing the stars (make it visible)            
      If MP1_Busy=True Then   
           MP1.Play  
      Else   
           MP2.Play  
      End If  
 End Sub  
 Sub Load_card     'This loads an image in the background  
      Dim ratio As Float   
      Question_Number=Rnd(0,FileList.Size)  
      Card_Name_Buffer=FileList.Get(Question_Number)  
      Do While (File.IsDirectory("/sdcard/Pictures/bits",Card_Name_Buffer) OR _  
           Not(Card_Name_Buffer.EndsWith(".jpg")) OR (Question_Number==Old_Question_Number))  
           Question_Number=Rnd(0,FileList.Size)  
           Card_Name_Buffer=FileList.Get(Question_Number)  
      Loop       
      Card_Buffer.Initialize("/sdcard/Pictures/bits",Card_Name_Buffer)  
      Card_Name_Buffer=Card_Name_Buffer.SubString2(0,Card_Name_Buffer.Length-4)  
      If MP1_Busy==True Then   
           MP2.Load("/sdcard/Pictures/bits",Card_Name_Buffer&".mp3")   
      Else  
           MP1.Load("/sdcard/Pictures/bits",Card_Name_Buffer&".mp3")         
      End If  
 End Sub  

PS.: And no, I am not going backwards from Java into Basic4Android. :) I started a year ago on Basic4Android because it was easier for me to ramp up but then I decided to learn Java... Now I still go to Basic4Android as I am more familiar with its libraries, but only till I get better with the Android SDK :)

Sunday, August 4, 2013

Copy project in Eclipse

The following is an extension of this and is center around Android/Java but gives you an idea...
Here I wanted to create an app based on the second tutorial that comes with OpenCV (see more here). So, I wanted to copy that and start working on it. This is what I did:
  1. Ctrl+C, Ctrl+V on the project that you want to copy (make sure the project is open)
  2. Enter new project name when prompted.
  3. Change the package name on the AndroidManifes.xml to whatever you want it to be. Click save.
  4. On the "Do you want to update your launch configurations?" say "yes".
  5. Right click on the package name, within src folder, and select refactor-rename. Enter new name. If you click in Preview it may show you a warning (package already exists in gen folder...). I ignored it (continue).
  6. Change the name of the application in the AndroidManifest. The real name may actually be in the strings.xml, so change it there too...
  7. Right click on the class name, inside the package, and select refactor-rename. Change main class name. It will update also any reference to it inside the code.
  8. Note: you may get an error on the link to native library. Just click continue and fix it later... Hint: the easiest way to fix errors is clicking on them and picking from the options they give you.
  9. Now if you click Run, select Android Application, should work...

Update 1: I did similar thing now for a C project I was using in ODE. Steps 1 and 2 are the same. Then in properties-Run/Debug settings delete the old launch configuration and enter a new one for the new project.

Update 2: After I copy the ODE project (ctrl+C, ctrl+V, rename the cpp) the whole file is marked with red warnings/errors. It is solved by going to properties --> C/C++ General --> Indexer and then mark Enable project specific settings, Enable indexer, Use active build configuration.

Cheers!

Tracking a ball in Android, with OpenCV

I am going to port to Android the example we did in PC/Java with OpenCV, about tracking a ball.
As that app was displaying different levels of processing of the incoming camera stream, it looks like a good idea to start from the second tutorial that comes with OpenCV. I follow these steps in order to make a copy of that project and start editing it.
    Note: we do not want to use jni at this time, but the tutorial 2 actually was doing so. I feel like leaving it in there without calling any of it... (so we can grow it later).

    The following is the code running OpenCV ball detection, all in java, no native, and (this is not good, so, will work on it later), running on the main/UI thread. I.e., if your phone is not powerful enough, it may just hang... Also, obviously, I want to port a lot of that processing into native, so, that I expedite things. There are also some other issues with the detection, but will work on a clean up version later.

    This is my Ball3Activity.java
     package com.cell0907.ball3;  
     import java.util.ArrayList;  
     import java.util.List;  
     import org.opencv.android.BaseLoaderCallback;  
     import org.opencv.android.CameraBridgeViewBase;  
     import org.opencv.android.CameraBridgeViewBase.CvCameraViewFrame;  
     import org.opencv.android.CameraBridgeViewBase.CvCameraViewListener2;  
     import org.opencv.android.LoaderCallbackInterface;  
     import org.opencv.android.OpenCVLoader;  
     import org.opencv.core.Core;  
     import org.opencv.core.CvType;  
     import org.opencv.core.Mat;  
     import org.opencv.core.Scalar;  
     import org.opencv.core.Size;  
     import org.opencv.imgproc.Imgproc;  
     import org.opencv.core.Point;  
     import android.app.Activity;  
     import android.os.Bundle;  
     import android.util.Log;  
     import android.view.Menu;  
     import android.view.MenuItem;  
     import android.view.WindowManager;  
     public class Ball3Activity extends Activity implements CvCameraViewListener2 {  
       private static final String  TAG = "OCVSample::Activity";  
       private static final int    VIEW_MODE_RGBA   = 0;  
       private static final int    VIEW_MODE_GRAY   = 1;  
       private static final int    VIEW_MODE_CANNY  = 2;  
       private static final int    VIEW_MODE_FEATURES = 5;  
       private int          mViewMode;  
       private Mat          mRgba;  
       private Mat          mIntermediateMat;  
       private Mat          mGray;  
       private Mat                           mHSV;  
       private Mat                           mThresholded;  
       private Mat                           mThresholded2;  
          private Mat                       array255;  
          private Mat                       distance;  
       private MenuItem        mItemPreviewRGBA;  
       private MenuItem        mItemPreviewGray;  
       private MenuItem        mItemPreviewCanny;  
       private MenuItem        mItemPreviewFeatures;  
       private CameraBridgeViewBase  mOpenCvCameraView;  
       private BaseLoaderCallback mLoaderCallback = new BaseLoaderCallback(this) {  
         @Override  
         public void onManagerConnected(int status) {  
           switch (status) {  
             case LoaderCallbackInterface.SUCCESS:  
             {  
               Log.i(TAG, "OpenCV loaded successfully");  
               // Load native library after(!) OpenCV initialization  
               //System.loadLibrary("mixed_sample");  
               mOpenCvCameraView.enableView();  
             } break;  
             default:  
             {  
               super.onManagerConnected(status);  
             } break;  
           }  
         }  
       };  
       public Ball3Activity() {  
         Log.i(TAG, "Instantiated new " + this.getClass());  
       }  
       /** Called when the activity is first created. */  
       @Override  
       public void onCreate(Bundle savedInstanceState) {  
         Log.i(TAG, "called onCreate");  
         super.onCreate(savedInstanceState);  
         getWindow().addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON);  
         setContentView(R.layout.tutorial2_surface_view);  
         mOpenCvCameraView = (CameraBridgeViewBase) findViewById(R.id.tutorial2_activity_surface_view);  
         mOpenCvCameraView.setCvCameraViewListener(this);  
       }  
       @Override  
       public boolean onCreateOptionsMenu(Menu menu) {  
         Log.i(TAG, "called onCreateOptionsMenu");  
         mItemPreviewRGBA = menu.add("RGBA");  
         mItemPreviewGray = menu.add("HSV");  
         mItemPreviewCanny = menu.add("Thresholded");  
         mItemPreviewFeatures = menu.add("Ball");  
         return true;  
       }  
       @Override  
       public void onPause()  
       {  
         super.onPause();  
         if (mOpenCvCameraView != null)  
           mOpenCvCameraView.disableView();  
       }  
       @Override  
       public void onResume()  
       {  
         super.onResume();  
         OpenCVLoader.initAsync(OpenCVLoader.OPENCV_VERSION_2_4_3, this, mLoaderCallback);  
       }  
       public void onDestroy() {  
         super.onDestroy();  
         if (mOpenCvCameraView != null)  
           mOpenCvCameraView.disableView();  
       }  
       public void onCameraViewStarted(int width, int height) {  
         mRgba = new Mat(height, width, CvType.CV_8UC4);  
         mHSV = new Mat(height, width, CvType.CV_8UC4);  
         mIntermediateMat = new Mat(height, width, CvType.CV_8UC4);  
         mGray = new Mat(height, width, CvType.CV_8UC1);  
         array255=new Mat(height,width,CvType.CV_8UC1);  
         distance=new Mat(height,width,CvType.CV_8UC1);  
         mThresholded=new Mat(height,width,CvType.CV_8UC1);  
         mThresholded2=new Mat(height,width,CvType.CV_8UC1);  
       }  
       public void onCameraViewStopped() {  
         mRgba.release();  
         mGray.release();  
         mIntermediateMat.release();  
       }  
       public Mat onCameraFrame(CvCameraViewFrame inputFrame) {  
            final int viewMode = mViewMode;  
            mRgba = inputFrame.rgba();  
            if (viewMode==VIEW_MODE_RGBA) return mRgba;  
               List<Mat> lhsv = new ArrayList<Mat>(3);             
               Mat circles = new Mat(); // No need (and don't know how) to initialize it.  
                                              // The function later will do it... (to a 1*N*CV_32FC3)  
               array255.setTo(new Scalar(255));  
               Scalar hsv_min = new Scalar(0, 50, 50, 0);  
               Scalar hsv_max = new Scalar(6, 255, 255, 0);  
               Scalar hsv_min2 = new Scalar(175, 50, 50, 0);  
               Scalar hsv_max2 = new Scalar(179, 255, 255, 0);  
               //double[] data=new double[3];  
            // One way to select a range of colors by Hue  
            Imgproc.cvtColor(mRgba, mHSV, Imgproc.COLOR_RGB2HSV,4);  
            if (viewMode==VIEW_MODE_GRAY) return mHSV;  
            Core.inRange(mHSV, hsv_min, hsv_max, mThresholded);                     
            Core.inRange(mHSV, hsv_min2, hsv_max2, mThresholded2);   
            Core.bitwise_or(mThresholded, mThresholded2, mThresholded);  
            /*Core.line(mRgba, new Point(150,50), new Point(202,200), new Scalar(100,10,10)CV_BGR(100,10,10), 3);  
                 Core.circle(mRgba, new Point(210,210), 10, new Scalar(100,10,10),3);  
                 data=mRgba.get(210, 210);  
                 Core.putText(mRgba,String.format("("+String.valueOf(data[0])+","+String.valueOf(data[1])+","+String.valueOf(data[2])+")"),new Point(30, 30) , 3 //FONT_HERSHEY_SCRIPT_SIMPLEX  
                       ,1.0,new Scalar(100,10,10,255),3);*/  
            // Notice that the thresholds don't really work as a "distance"  
            // Ideally we would like to cut the image by hue and then pick just  
            // the area where S combined V are largest.   
            // Strictly speaking, this would be something like sqrt((255-S)^2+(255-V)^2)>Range  
            // But if we want to be "faster" we can do just (255-S)+(255-V)>Range  
            // Or otherwise 510-S-V>Range  
            // Anyhow, we do the following... Will see how fast it goes...  
            Core.split(mHSV, lhsv); // We get 3 2D one channel Mats  
            Mat S = lhsv.get(1);  
            Mat V = lhsv.get(2);  
            Core.subtract(array255, S, S);  
            Core.subtract(array255, V, V);  
            S.convertTo(S, CvType.CV_32F);  
            V.convertTo(V, CvType.CV_32F);  
            Core.magnitude(S, V, distance);  
            Core.inRange(distance,new Scalar(0.0), new Scalar(200.0), mThresholded2);  
            Core.bitwise_and(mThresholded, mThresholded2, mThresholded);  
     /*       if (viewMode==VIEW_MODE_CANNY){  
                 Imgproc.cvtColor(mThresholded, mRgba, Imgproc.COLOR_GRAY2RGB, 4);  
                 return mRgba;   
            }*/  
            // Apply the Hough Transform to find the circles  
            Imgproc.GaussianBlur(mThresholded, mThresholded, new Size(9,9),0,0);  
            Imgproc.HoughCircles(mThresholded, circles, Imgproc.CV_HOUGH_GRADIENT, 2, mThresholded.height()/4, 500, 50, 0, 0);       
            if (viewMode==VIEW_MODE_CANNY){  
                 Imgproc.Canny(mThresholded, mThresholded, 500, 250); // This is not needed.  
                                                                                 // It is just for display  
                 Imgproc.cvtColor(mThresholded, mRgba, Imgproc.COLOR_GRAY2RGB, 4);  
                 return mRgba;   
            }   
            //int cols = circles.cols();  
             int rows = circles.rows();  
             int elemSize = (int)circles.elemSize(); // Returns 12 (3 * 4bytes in a float)  
             float[] data2 = new float[rows * elemSize/4];  
             if (data2.length>0){  
                  circles.get(0, 0, data2); // Points to the first element and reads the whole thing  
                                                  // into data2  
                  for(int i=0; i<data2.length; i=i+3) {  
                       Point center= new Point(data2[i], data2[i+1]);  
                       Core.ellipse( mRgba, center, new Size((double)data2[i+2], (double)data2[i+2]), 0, 0, 360, new Scalar( 255, 0, 255 ), 4, 8, 0 );  
                }  
              }  
            return mRgba;  
       }  
       public boolean onOptionsItemSelected(MenuItem item) {  
         Log.i(TAG, "called onOptionsItemSelected; selected item: " + item);  
         if (item == mItemPreviewRGBA) {  
           mViewMode = VIEW_MODE_RGBA;  
         } else if (item == mItemPreviewGray) {  
           mViewMode = VIEW_MODE_GRAY;  
         } else if (item == mItemPreviewCanny) {  
           mViewMode = VIEW_MODE_CANNY;  
         } else if (item == mItemPreviewFeatures) {  
           mViewMode = VIEW_MODE_FEATURES;  
         }  
         return true;  
       }  
       //public native void FindFeatures(long matAddrGr, long matAddrRgba);  
     }  
    

    And this is my tutorial2_surface_view.xml
     <LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"  
       xmlns:tools="http://schemas.android.com/tools"  
       xmlns:opencv="http://schemas.android.com/apk/res-auto"  
       android:layout_width="match_parent"  
       android:layout_height="match_parent" >  
       <org.opencv.android.JavaCameraView  
         android:layout_width="fill_parent"  
         android:layout_height="fill_parent"  
         android:id="@+id/tutorial2_activity_surface_view"  
         opencv:show_fps="true"  
         opencv:camera_id="any" />  
     </LinearLayout>  
    

    Notice the use of opencv:show_fps="true" to see the fps on screen. To include this line, you will need to also add on top xmlns:opencv="http://schemas.android.com/apk/res-auto". On my HTC One, the fps will go from 10fps when just capturing the images down to barely 2 when is doing all the processing...

    PS.: Click here to see the index of these series of posts on OpenCV