Sunday, October 13, 2013

Tutorial for Drawing in Android - SurfaceView and SurfaceHolder

Right into the water! The next paragraph is my 5 seconds explanation, so, hold on to anything coz is not straightforward (it was not for me, at least)!! But don't worry. Gets broken down below... If you can't learn it here, we will return your money, guaranteed... lol! Also, you can start by checking "simpler but less powerful" approaches here.

Soooo.... Bottom line, within the Activity, we need an ObjectA that has the Surface object (the screen with pixels) of the application, that we want to show at that instant. This ObjectA has an ObjectB inside that implements a bunch of methods that allow access to that surface. The ObjectA launches a separate thread and gives it ObjectB. The thread as such, independent of the UI, gets a hold of the canvas provided by ObjectB (in the background, no display, and no interfering with the main thread/UI thread), draws on it, and releases the hold, for it to get updated by the UI.

The pending questions that I haven't found answer for are:
  1. Why do we need Object B and not implement the methods directly in Object A? Stackoverflow question on the same thing is unanswered.
  2. Do we really update the UI in the thread or in the UI? I thought we were supposed to do this always in the UI.
Anyhow, let's put names to all this. In simple graphics (see this) to update the  screen one uses an object from View class. Nevertheless, in this case, we want something with a bit more control, with more methods, among other things, to do live graphics. An example is shown below. The following app has a menu which allows the selection between two views, a simple one that does nothing, created with a layout, and the 2nd one, with SurfaceView, that keeps adding a random circle every 100ms.


To do the 2nd one (the one we care to explain) we build a class extending SurfaceView class, which is a child of View. This, in the example below, happens in the DotsSurfaceView definition (DotsSurfaceView.java). An instance of this class is what we call ObjectA above. In the code below, this is dots_screen_view (see Dots1.java). Whenever we want the app to switch to this view, we will use setContentView(dots_screen_view);

Dots1.java
 package com.cell0907.dots1;  
   
 import android.os.Bundle;  
 import android.app.Activity;  
 import android.view.Menu;  
 import android.view.MenuItem;  
 import android.view.View;  
 import android.widget.Button;  
   
 public class Dots1 extends Activity {  
      // USER INTERFACE  
      static int screen_selected=0;  
      private static final int MENU_SIMPLE_UI = 1;     // SIMPLE UI  
      private Button button1;  
      private static final int MENU_DOTS = 2;          // RANDOM CIRCLES "dots"  
      DotsSurfaceView dots_screen_view;  
        
   @Override  
      protected void onCreate(Bundle savedInstanceState) {  
           super.onCreate(savedInstanceState);  
      }  
        
   @Override  
   public void onPause(){  
        if (screen_selected==2){  
             dots_screen_view.surfaceDestroyed(dots_screen_view.getHolder());  
        }  
        super.onPause();  
   }  
        
   @Override  
      protected void onResume(){  
           super.onResume();  
           switch (screen_selected) {  
           case 0:  
                screen_selected=MENU_SIMPLE_UI;  
           case MENU_SIMPLE_UI:  
           screen_selected=1;  
              set_simple_UI();  
       return;  
     case MENU_DOTS:  
           screen_selected=2;  
           set_dots();  
       return;  
     }  
      }  
             
   @Override  
      protected void onStop() {  
           super.onStop();  
   }   
     
       /**  
    * Invoked during init to give the Activity a chance to set up its Menu.  
    *  
    * @param menu the Menu to which entries may be added  
    * @return true  
    */  
   @Override  
   public boolean onCreateOptionsMenu(Menu menu) {  
     super.onCreateOptionsMenu(menu);  
     menu.add(0, MENU_SIMPLE_UI, 0, R.string.menu_simple_ui);  
     menu.add(0, MENU_DOTS, 0, R.string.menu_dots);  
     return true;  
   }       
     
   /**  
    * Invoked when the user selects an item from the Menu.  
    *  
    * @param item the Menu entry which was selected  
    * @return true if the Menu item was legit (and we consumed it), false  
    *     otherwise  
    */  
   @Override  
   public boolean onOptionsItemSelected(MenuItem item) {  
           switch (screen_selected) {  
     case MENU_SIMPLE_UI:  
          break;  
     case MENU_DOTS:  
          dots_screen_view.surfaceDestroyed(dots_screen_view.getHolder());  
          break;  
     }  
     switch (item.getItemId()) {  
       case MENU_SIMPLE_UI:  
            screen_selected=1;  
               set_simple_UI();  
         return true;  
       case MENU_DOTS:  
            screen_selected=2;  
            set_dots();  
         return true;  
       }  
     return false;  
   }  
        
      void set_simple_UI(){  
           setContentView(R.layout.activity_main);  
           button1=(Button)this.findViewById(R.id.button1);       
           button1.setOnClickListener(new View.OnClickListener() {  
                @Override  
                public void onClick(View v) {  
                     //~Activity;  
                }  
           });  
      }  
        
      void set_dots(){  
           dots_screen_view=new DotsSurfaceView(this);  
           setContentView(dots_screen_view);  
      }  
 }  

Ok, so, DotsSurfaceView inherits from SurfaceView class, among other things, few Surface objects (like the one we are using now, and the one we are going to use...) and methods, like callbacks to react to stuff that happens, etc... Also, SurfaceView has an internal (private) object created from an anonymous class (see references below) that implements the SurfaceHolder interface. This is what we call Object B above. This is an object that allows access to the Surfaces. This approach effectively gives body to the methods of the SurfaceHolder interface. See in Google Source (line 694) the creation of the internal object mSurfaceHolder:
private SurfaceHolder mSurfaceHolder = new SurfaceHolder() { methods... }
Still, in our code, we need to implement in the DotsSurfaceView the SurfaceHolder.Callback interface. That is a nested interface to the SurfaceHolder interface. To see this, one can simply look at the SurfaceHolder source. Notice that inside that code (line 69), nested, there is the definition of the Callback interface (i.e., SurfaceHolder.Callback) which defines 3 more methods (see example code). Those are the ones that we need to implement when we extend SurfaceView. If not, we will get a compiler error as we are saying "implement SurfaceHolder.Callback" in the header of the class, but we don't.

DotsSurfaceView.java
 package com.cell0907.dots1;  
   
 import android.content.Context;  
 import android.view.SurfaceHolder;  
 import android.view.SurfaceView;  
   
 // We extend SurfaceView. Internally (private) SurfaceView creates an object SurfaceHolder  
 // effectively defining the methods of the SurfaceHolder interface. Notice that it does  
 // not create a new class or anything, it just defines it right there. When we extend  
 // the SurfaceView with the SurfaceHolder.Callback interface, we need to add in that extension  
 // the methods of that interface.  
   
 public class DotsSurfaceView extends SurfaceView implements SurfaceHolder.Callback {  
      private SurfaceHolder holder;     // This is no instantiation. Just saying that the holder  
                                              // will be of a class implementing SurfaceHolder  
      private DotsThread DotsThread;// The thread that displays the dots  
             
      public DotsSurfaceView(Context context) {  
           super(context);  
           holder = getHolder();          // Holder is now the internal/private mSurfaceHolder inherit   
                                              // from the SurfaceView class, which is from an anonymous  
                                              // class implementing SurfaceHolder interface.  
           holder.addCallback(this);  
      }  
        
      @Override  
      public void surfaceCreated(SurfaceHolder holder) {  
      }  
        
      @Override  
      // This is always called at least once, after surfaceCreated  
      public void surfaceChanged(SurfaceHolder holder, int format, int width, int height) {  
           if (DotsThread==null){  
                DotsThread = new DotsThread(holder);  
                DotsThread.setRunning(true);  
                DotsThread.setSurfaceSize(width, height);  
                DotsThread.start();  
           }  
      }  
   
      @Override  
      public void surfaceDestroyed(SurfaceHolder holder) {  
           boolean retry = true;  
           DotsThread.setRunning(false);  
           while (retry) {  
                try {  
                     DotsThread.join();  
                     retry = false;  
                } catch (InterruptedException e) {}  
           }  
   }  
        
   public Thread getThread() {  
     return DotsThread;  
   }  
 }  

But still, where do we draw anything? All that happens on a separate object thread. We want to have that running on its own pace, faster or slower, without interfering with the UI. We define this object class in DotsThread.java. And we create the instance of the object in one of the callbacks of the SurfaceHolder.Callback interface (surfaceChanged). Whenever the view is created, it goes through surfaceCreated and then this call. So, within surfaceChanged, we will lunch the thread that will do the actual drawing. We will give that process a Holder to our view, so, that it can manipulate it.

The main magic here happens within the run method. Basically there is a loop running continuously as long as the variable running is truth. One can make that variable truth or false from outside, through the method setRunning, which effectively will allow the loop to run or stop it. See how we stop it in surfaceDestroyed, the 3rd callback of the SurfaceHolder.Callback interface. Basically, we turn the variable off and wait for the thread to disappear.

Within the thread loop, we wait for certain time and then we create a dot. This gets added to the list and then we lock the canvas. Now we can draw on it. That routine basically goes through the list drawing all the dots. Notice that we write them all, not only the last one. Basically, making sure that even if something changed the canvas while it was out of our control (lock), we get it back as we like it. After we are done, we unlock it, which effectively will refresh it in the screen.

DotsThread.java
 package com.cell0907.dots1;  
   
 import java.util.ArrayList;  
   
 import android.graphics.Canvas;  
 import android.graphics.Color;  
 import android.graphics.Paint;  
 import android.graphics.Paint.Style;  
 import android.view.SurfaceHolder;  
   
 public class DotsThread extends Thread {       
      private int mCanvasWidth;  
   private int mCanvasHeight;  
      private ArrayList<dot> Dots= new ArrayList<dot>(); // Dynamic array with dots  
      private SurfaceHolder holder;  
   private boolean running = false;  
   private final Paint paint = new Paint(Paint.ANTI_ALIAS_FLAG);  
   private final int refresh_rate=100;      // How often we update the screen, in ms  
   
   public DotsThread(SurfaceHolder holder) {  
     this.holder = holder;  
   }  
   
   @Override  
   public void run() {  
        int x,y,radius;  
        float[] color=new float[3]; // HSV (0..360,0..1,0..1)  
     long previousTime, currentTime;  
        previousTime = System.currentTimeMillis();  
     Canvas canvas = null;  
     while(running) {  
          // Look if time has past  
       currentTime=System.currentTimeMillis();  
       while ((currentTime-previousTime)<refresh_rate){  
            currentTime=System.currentTimeMillis();  
       }  
       previousTime=currentTime;  
       // ADD ONE MORE DOT TO THE SCREEN  
       x=100 + (int)(Math.random() * (mCanvasWidth-200));  
       y=100 + (int)(Math.random() * (mCanvasHeight-200));  
       radius=1 + (int)(Math.random() * 99);  
       color[0]=(float)(Math.random()*360);  
       color[1]=1;  
       color[2]=1;  
       dot mdot=new dot(x,y,radius,Color.HSVToColor(128,color));  
       Dots.add(mdot);  
       // PAINT  
       try {  
         canvas = holder.lockCanvas();  
         synchronized (holder) {  
               draw(canvas);           
         }  
       }  
       finally {  
            if (canvas != null) {  
                 holder.unlockCanvasAndPost(canvas);  
                 }  
       }  
       // WAIT  
                try {  
                     Thread.sleep(refresh_rate-5); // Wait some time till I need to display again  
                } catch (InterruptedException e) {  
                     // TODO Auto-generated catch block  
                     e.printStackTrace();  
                }       
     }  
   }  
   
   // The actual drawing in the Canvas (not the update to the screen).  
   private void draw(Canvas canvas)  
   {  
        dot temp_dot;  
        canvas.drawColor(Color.BLACK);  
        paint.setStyle(Style.FILL_AND_STROKE);  
        for (int i=0;i<Dots.size();i++){  
             temp_dot=Dots.get(i);  
             paint.setColor(temp_dot.get_color());  
             canvas.drawCircle((float)temp_dot.get_x(),  
                       (float)temp_dot.get_y(),   
                       (float)temp_dot.get_radius(),  
                       paint);  
        }  
    }  
     
   public void setRunning(boolean b) {  
     running = b;  
   }  
     
   public void setSurfaceSize(int width, int height) {  
        synchronized (holder){  
          mCanvasWidth = width;  
          mCanvasHeight = height;  
        }  
   }  
     
   private class dot{  
           private int x,y,radius,color;  
             
           dot(int x, int y, int radius, int color){  
                this.x=x;  
                this.y=y;  
                this.radius=radius;  
                this.color=color;  
           }  
             
           public int get_x(){  
                return this.x;  
           }  
   
           public int get_y(){  
                return this.y;  
           }  
   
           public int get_radius(){  
                return this.radius;  
           }  
   
           public int get_color(){  
                return this.color;  
           }  
      }  
 }  

And just for Richard (my first commenter below) :), the layout file activity_main.xml:
 <RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"  
   xmlns:tools="http://schemas.android.com/tools"  
   android:layout_width="match_parent"  
   android:layout_height="match_parent"  
   android:paddingBottom="@dimen/activity_vertical_margin"  
   android:paddingLeft="@dimen/activity_horizontal_margin"  
   android:paddingRight="@dimen/activity_horizontal_margin"  
   android:paddingTop="@dimen/activity_vertical_margin"  
   tools:context=".Dots1" >  
   <TextView  
     android:id="@+id/textView1"  
     android:layout_width="wrap_content"  
     android:layout_height="wrap_content"  
     android:text="@string/hello_world" />  
   <Button  
     android:id="@+id/button1"  
     android:layout_width="wrap_content"  
     android:layout_height="wrap_content"  
     android:layout_alignLeft="@+id/textView1"  
     android:layout_below="@+id/textView1"  
     android:layout_marginLeft="32dp"  
     android:layout_marginTop="146dp"  
     android:text="@string/button1" />  
 </RelativeLayout>  

And as we are on it, let me add the AndroidManifest.xml:
 <?xml version="1.0" encoding="utf-8"?>  
 <manifest xmlns:android="http://schemas.android.com/apk/res/android"  
   package="com.cell0907.dots1"  
   android:versionCode="1"  
   android:versionName="1.0" >  
   <uses-sdk  
     android:minSdkVersion="9"  
     android:targetSdkVersion="17" />  
   <application  
     android:allowBackup="true"  
     android:icon="@drawable/ic_launcher"  
     android:label="@string/app_name"  
     android:theme="@style/AppTheme" android:debuggable="true">  
     <activity  
       android:name="com.cell0907.dots1.Dots1"  
       android:label="@string/app_name" >  
       <intent-filter>  
         <action android:name="android.intent.action.MAIN" />  
         <category android:name="android.intent.category.LAUNCHER" />  
       </intent-filter>  
     </activity>  
   </application>  
 </manifest>  

So, that's it. Other things to learn from the example:
  1. Notice that the app may have different "screens or views" objects and at a given moment an app will pick one. 
  2. Also notice the HSV use to create the color of the dots. That helps us to avoid "grey" dots by only manipulating the Hue. If questions about that, check my other post here.
  3. Do not use the width and height of a view inside its constructor. When a view’s constructor is called, Android doesn’t know yet how big the view will be, so the sizes are set to zero. The real sizes are calculated during the layout stage, which occurs after construction but before anything is drawn. You can use the onSizeChanged() method to be notified of the values when they are known, or you can use the getWidth( ) and getHeight() methods later, such as in the onDraw( ) method.
  4. What do we do when the surface is destroyed? See closing surfaceView properly
  5. I don't want to ignore the line Holder.addCallback(this). Maybe the best explanation I have seen on this is here
  6. The whole painting in the canvas it can be pretty straightforward but it can also get advanced. I list here a bunch of links. Eventually may write a tutorial on this:
    1. Clearing canvas with Canvas.drawColor()
    2. Android Bitmap Blending - Color Channels 
    3. How to paint with alpha
    4. PorterDuff.Mode
    5. Wikipedia Alpha Compositing 
    6. Stackoverflow question
Finally, as summary, let me try to describe the different players on this approach. The best attempt to this is here. Worth reading. Also thanks to Android Student here and Sajan and Lawrence D'Oliveiro for trying to do the same:
  1. SurfaceView is a subclass of View. "Provides a dedicated drawing surface embedded inside of a view hierarchy. You can control the format of this surface and, if you like, its size; the SurfaceView takes care of placing the surface at the correct location on the screen."
  2. SurfaceHolder is an object which provide us with the canvas we can draw on. Allows you to control the surface size and format, edit the pixels in the surface, and monitor changes to the surface. All this methods are part of the SurfaceHolder interface definition.
  3. SurfaceHolder.Callback is just an interface, a list of method headers, not the implementation, but just a description of how those methods interface with the external world. It is a nested class of SurfaceHolder and it is actually what we implement in the SurfaceView class. For "implement" we mean that when we define the a class extending SurfaceView we have to implement (program, create) the methods defined by SurfaceHolder.Callback interface. We tell the compiler that we are going to do so by writing in the class definition the word "implement", like class Panel extends SurfaceView implements SurfaceHolder.Callback. Therefore if we fail to create the methods defined by the SurfaceHolder.Callback interface, the compiler will throw an error. As SurfaceHolder.Callback is a nested class of SurfaceHolder, the other methods of the SurfaceHolder interface can be also implemented, but do not have to.
Just for completeness, I list here two parent classes that we actually do not use directly (you can skip):
  1. View class, direct descendent from Object, and parent of SurfaceView, "this class represents the basic building block for user interface components. A View occupies a rectangular area on the screen and is responsible for drawing and event handling. View is the base class for widgets, which are used to create interactive UI components (buttons, text fields, etc.)."
  2. Surface class.  Handle onto a raw buffer that is being managed by the screen compositor. I believe the SurfaceHolder actually can provide access directly to this, but I have seldom seen do that.
Finally, please, click here to see an index of other posts on Android.

References:
  1. Probably the best tutorial I found on this is here and although it doesn't completely explain it either it gives plenty to work with. 
  2. Anonymous class examples here and here.
  3. A whole series of  tutorials from less to more complicated.  
  4. Create a circle at the touch point of surfaceview
  5. Example without launching a separate thread
  6. tic tac toe example
  7. http://blog.infrared5.com/2011/07/android-graphics-and-animation-part-ii-animation/  
  8. One tutorial that covers SurfaceView with and without the thread. The issue with this one is that the code is not completely proven so it won't run if you just cut and paste.
  9. Can we create an instance of an interface?
  10. Note that there are other methods to do advanced graphics, like OpenGL...

18 comments:

  1. Great article. Would be nice if the xml for the layout was added as well. :)

    ReplyDelete
    Replies
    1. Wops!! Good point! Just added! Apologies and thank you for stopping by!! :)

      Delete
  2. Nice tutorial. Really what I was looking for.

    ReplyDelete
    Replies
    1. I'm slowly starting to partly understand small fragments of some portions of this stuff. Finally. This was the first tutorial that did not end up in a null pointer assignment after one hour of cut and paste and tears and "wtf now?"s.
      As I can not buy you a beer, you have to promise me that you will buy yourself a beer from me.

      THANKS!

      /Erik (55 year old assembler programmer.)

      Delete
    2. HA! Yeah, sure, you certainly will not have to force me to do that! But do me a favor and buy yourself or somebody nearby one too! Cheers!! :)

      Delete
  3. Mission accomplished. But seriously, your tuts are the best I've ever found out there. And I really need good tutorials. Taking the step from linear, procedural coding, mainly in assembler, to the abstraction level of Java is not a small leap for an old ox like me.
    I will keep checking your blog, for sure.
    As my mother tongue is not english, I might have missed something, but your reaction to the beer proposal and your nick made me start wondering if it is that bad? If so, I must say that you really do make good use of the abundance of available time.

    Thanks again.
    /Erik


    ReplyDelete
    Replies
    1. You are very welcome! My mother language is not English either, so, not sure what I missed :). But all what I meant to say is that I'll have a beer to cheer for your support. Every time somebody post something like this, it makes my day! Honestly, I am just returning back to the community all what I can... that's where I learn ALL this... And by the way, I also grew up with assembly... I know exactly what you mean :)

      Delete
  4. Great tutorial! :) Thank cell0907 alot!
    But if I want to redraw all of DotsSurfaceView from the beginning. How can I do this?

    ReplyDelete
  5. error in dot1.java at line
    menu.add(0, MENU_SIMPLE_UI, 0, R.string.menu_simple_ui);
    menu.add(0, MENU_DOTS, 0, R.string.menu_dots);

    ReplyDelete
  6. This comment has been removed by the author.

    ReplyDelete
  7. @Anonymous,
    Just add new values as following to "res/values/string.xml" then enjoy!

    name="menu_simple_ui" Value=Simple UI
    name="menu_dots" Value=Dots

    Thanks alot Mr. cell0907, this was EXACTLY what I was looking for, for long! :)
    Best regards and good luck!

    ReplyDelete
    Replies
    1. Thanks a lot to you Thair for stopping by, helping Anonymous, and leaving your greetings. Good luck to you too!! :)

      Delete
  8. خیلی زیاد نوشتین =)

    ReplyDelete
  9. خیلی زیاد نوشتین =)

    ReplyDelete
  10. Great tutorial, great help, many thanks!

    ReplyDelete
  11. You also forgot the string for the button text..
    Here is my complete strings.xml:
    <?xml version="1.0" encoding="utf-8"?>
    <resources>
    <string name="app_name">Dots</string>
    <string name="hello_world">Hello world!</string>
    <string name="action_settings">Settings</string>
    <string name="button1">Start SurfaceView Drawing</string>
    <string name="menu_simple_ui">Simple UI</string>
    <string name="menu_dots">Dots</string>
    </resources>

    For the onClick event in Dots1.java I also added this (so the button does something):

    @Override
    public void onClick(View v) {
    //~Activity;
    set_dots();
    }

    Good multi-threaded surface demo code though... ;)

    ReplyDelete
    Replies
    1. By the way..because your comments doesn't support either the html pre or code tags
      I was forced to use a special tool to get the xml (for the strings.xml) to show correctly:

      http://www.htmlescape.net/htmlescape_tool.html

      Delete