Department of Physical Sciences, Meru University of Science and Technology (MUST)

From Raspberry Pi Min-Grant project
Jump to: navigation, search

About Meru University of Science and Technology 

Meru University of Science and Technology (MUST) is a young public university in Meru County of the Republic of Kenya having been chartered in March 2013 established under The Universities Act of 2012. Prior to the University status, MUST was a constituent college of Jomo Kenyatta University of Agriculture and Technology (JKUAT).

Meru University of Science and Technology Raspberry Pi Project

Meru University of Science and Technology has won a KENET Raspberry mini grant. This project is geared towards using a credit size gadget called Raspberry Pi (RPI) to teach computational units in the University. Raspberry Pi is a small device that was developed for teaching Computer Science in schools. Besides being portable, RPI is affordable and can be easily interfaced with the outside and used to perform automation projects.

As a pilot project, we will use RPI to teach three computational units on offer in the university during the April - August 2015 semester. We will also be conducting RPI related projects with an aim of publishing the results in referred journals. Students will be encouraged to develop their own projects using the pi and eventually showcase them within the University and other forums organized by RPI stake holders. Read more.

Executive Summary

The overall objective is to establish a Raspberry Pi teaching Laboratory in Meru University of Science and Technology (MUST). To achieve this a minimum of three computational units will be used to pilot the use of Raspberry Pi in teaching. Instructional manuals for teaching these units using Raspberry Pi will be developed and provided to the students. Student learning outcomes will be assessed throughout the semester. Publications regarding novel findings observed in the pilot stage will be presented in a refereed conference. Prior to the grant the university had already acquired a number of Raspberry Pi units and its accessories, developed exhibits of how Raspberry Pi can be used for teaching computer related courses in schools and showcased the projects in Commission for University Education exhibitions and ASK shows. Work is also in progress to develop a security surveillance system and an electronic device for early detection of crop diseases and other plant stress using Raspberry Pi. Upon successful completion of the pilot stage the university will upscale the use of Raspberry Pi in teaching more courses and launch a campaign for popularizing the adoption of Raspberry Pi in teaching computer related course in tertiary institutions of learning, secondary and primary schools within the environs of the university.

Team Members

Methodology of Implementation and Assessment

Setting up.jpg

The University currently has four teaching computer laboratories fully connected to the internet. One of these computer labs will be used to establish a Raspberry pi based teaching laboratory. Facilities such as computer monitors, USB keyboards and mice, internet connectivity and computer desks will be shared by the existing PCs and the Raspberry Pi modules to be acquired. A suitable mode of embedding the Raspberry Pi modules in the system unit will be adopted.

Meru University RPI team first training

On 2nd march 2015 RPI project team underwent their first training on getting started with RPI. The course focused on the key components of the Pi, downloading and load the Raspbian (Raspberry Pi official OS) to the SD card, powering up the Pi, configuring the pi OS, launching basic Pi applications for word processing, spread sheet, presentation and internet browsing. Members were also introduced to Python programming using the Python IDE which comes with Raspbian.

Launch of RPI to Computing students

Class.jpg

On 15th March  2015, the Meru University RPI team introduced RPI to computer science students. The students were explained the capabilities of the gadgets with demonstrations. In this meeting students resolved to organize themselves into groups to learn and develop more RPI applications. 

RPI related projects in progress

RPI imaging module for crop stress monitoring

Imaging.jpg

We are developing a system for detecting crop stress in green houses using Raspberry imaging module. The module comprises of the RPI connected to RPI camera. An executable Linux script for periodically taking photos is written and incorporated to the Linux task scheduler (cron) so that upon reboot the pi starts taking photos after a specified period. The system is powered by solar. So far the imaging module is complete and has been tested in the green house and has been found to be working fine. We are now working on suitable algorithms for processing the images captured for stress. We intend to use python and open CV imaging processing library to accomplish the task.

Raspberry pi Tutor project

Meru University Raspberry pi team is working on making Raspberry pi an ideal gadget for learning. The team is specifically developing educational applications for running on the Pi geared at making learning interesting right from early childhood. So far applications incorporating elementary English, mathematics, and technology lessons have been developed. Images, videos, and sound are used as learning aids in these subjects. 
The project has been showcased in various forums including the Commission of University exhibition held in Machakos in from 19th to 21st March, 2015, the NACOSTI exhibition week held in the University of Nairobi from 11th to 16th May 2015, and the Meru National ASK show held in Meru ASK show ground from 3rd to 6th June, 2015. members of the public were interested by the capabilities of the little device and agreed the Pi can make a great learning companion to the pupils and students as well as teachers.
The team main agenda as pertaining this project is to advocate for the adoption of the Raspberry pi (RPI) as an alternative to desktop/laptops/tablets computers in learning of IT related courses.

The presentation was categorized as an exhibition, and apart from showcasing our amazing gadget, the team was also able to interact and learn where to improve and the many possible challenges that the project is likely to encounter in the near future.

PROJECT DESCRIPTION (By developers: students from Meru University)
Target Audience: Class 1-4 students, Teachers
Application: Teaching Aid, Student management
Detailed Description:

Welcome.jpg


“CHEESE” is an innovative program that is aimed at teaching young students different subjects in an immersive and well interactive manner.
The Graphical user interface is developed in such a way to allow different students to log in so as to start the lesson
The student is required to type in their name and the teachers name after which a log is kept.
The teacher can then use the log to trace and make a follow up of which students were present for the class and also be able to access the student performance.
The program also allows working in groups by allowing login of different group members. This data is very useful to the teacher for progress assessment and grading.
From the login phase the student can choose from a range of different subject. Our personal favorite is Technology as a subsidiary of Science. In this subject the program present different logos and graphics and the students is supposed to recognize them and key in the correct answer. Points are awarded for correct answers.
But thats not all.


Teachers also have the ability to upload their own images that they feel may be appropriate and defining the correct answers thereby further extending the capability of the program.
This is because during the development of this program we not only have the students in mind but also the teachers who are supposed to be supervising the students.
There is a total of six subjects that students can select from:

SUBJECT DESCRIPTION
Mathematics

The subject has simple arithmetic questions that the student is tasked with finding the answers to, awarding points for each correct answer answered

English

Aimed at the young children, this subject requires that the student identifies pictures and spells the correctly

Kiswahili

Graphics is core to the subject with students being tasked with translating and spelling different pictures.

Christian Education

Morals are an important lesson both to us and the society, which is why as part of our development in the project we designed simple stories some borrowed from famous tales to teach children different morals and life skills

Social Studies

<Under development>

Music

<Under development>

Art and Craft

<Under development>

Technology

In this subject the program present different logos and graphics and the students is supposed to recognize them and key in the correct answer.  


NB: For all of the above subjects, point are awarded for each correct answer

Developed by Mr. Michael Kamau and Steve Kisinga, both students from Meru University, we believe that the application has huge potential as a teaching aid as well as a good student management program that is user friendly.

About.jpg

We also presented the same project at the Meru ASK show where we scooped various awards, among them being the best innovation stand. Major efforts are being carried at ensuring that the project succeeds, and a lot of dedication is being displayed by the team members

RASPBERRY AS A TEACHING AID

CIT 3102 – FUNDAMENTALS OF COMPUTER PROGRAMMING

IMG 20150702 092618.jpg

The raspberry pi has been used in Meru University as a teaching aid to teach among other units, Fundamentals of computer programming.

Students were introduced to the raspberry pi, what it is, it's capabilities and its advantages. As part of the lesson students were required to use the raspberry pi to access the internet and use various search engines to find out the technical specifications of the device.

Students were intoduced to programming basic programming and taught basic proramming skills. Geany was used as the IDE of choice to teach C programming. Ranging from displaying simple text on the terminal to the more complex programming skills of assigning variables and accepting input, the raspberry pi has offered a great deal in helping teach the unit.

Geany.png
==

Introduction Raspberry Pi and GPIO programming

Professor Simon Cox and his team at the University of Southampton connected 64 Raspberry Pi boards to build 
an experimental supercomputer, held together with Lego bricks.

The project was able to cut the cost of a supercomputer from millions of dollars to thousands or even hundreds.

Introduction
System on a Chip

What does System on a Chip (SoC)mean?

A system on a chip (SoC) combines the required electronic circuits of various computer components onto 
a single, integrated chip (IC). 

A SoC is a complete electronic substrate system that may contain analog, digital, mixed-signal or radio frequency 
functions. 

Its components usually include a graphical processing  unit (GPU), a central processing unit (CPU) that may be 
multi-core, and system memory (RAM).

Because a SoC includes both the hardware and software,  it uses less power, has better performance, requires less 
space and is more reliable than multi-chip systems. Most system-on-chips today come inside mobile devices like 
smartphones and tablets.

A distinction between a microprocessor and a microcontroller should also be explained.

A microprocessor is an integrated circuit (IC) which has only the central processing unit (CPU) on it. Such as the 
Intel i7 CPU.

Microcontrollers have a CPU, memory, and other peripherals embedded on them.

A microcontroller can be programmed to perform certain functions.
A very popular microcontroller is the Arduino Uno.
The difference between a SoC and a microcontroller often times is defined by the amount of random access 
memory (RAM).
A SoC is capable of running its own operating system.

System on a Chip

The SoC we will use today is the Raspberry Pi B+.

The heart of this credit card sized computer is the Broadcom BCM2835 chipset that contains an ARM CPU 
and a Videocore4 graphics processing unit.

The B+ has a 700 MhzCPU and 512 MB of RAM.
Pi Assembly and Raspbian Install

We will now discuss how to assemble the components that make up the Raspberry Pi.
You will need the following (should be on the bench):
•Raspberry Pi B+ board
•HDMI to DVI cable
•Monitor or TV
•Micro SD card
•USB keyboard and mouse
•Micro USB power supply


Pi Assembly and Raspbian Install

We will now bake (assemble) the Pi!

Once you have it put together, power it on.

You should see the New Out Of Box Software (NOOBS)  operating system installer. 

Select the Raspbian [RECOMMENDED] option, change the  language to English (US), change the keyboard to US, and 
click install.

While the OS is installing we will discuss some electronics  information.


Voltage, Current, and Resistance

When a potential difference exists between two charged bodies that are connected by a conductor, electrons flow 
in the conductor. This flow is from the negatively charged body to the positively charged body.

The voltage does not flow, only the charge. Voltage supplies the “push” or “pressure”.


Plumbing Analogy

It is difficult to keep straight the concepts of voltage, current, charge, resistance, etc., but for 
electrical circuits there is a helpful analogy with plumbing systems. Wires are, of course, like pipes; 
current is like the rate of water flow (gallons per minute); and resistance is like friction in the 
pipes. Now, here is the most helpful part of the analogy: voltage is like water pressure, and 
batteries and generators are like pumps. Like all analogies, however, it doesn't quite work. 

Voltage, Current, and Resistance


Voltage can also be looked at as water pressure.We are mainly concerned with the potential 
difference between two points.

Ground is the reference point and is considered to have zero potential.
Electric current –directed flow of free electrons. 

Moves from a region of negative potential to a region of positive potential. 

Therefore it can be stated that electric current flows from negative to positive.

Voltage, Current, and Resistance


Breadboarding Basics

What is a Breadboard?
What circuit are we going to breadboard today?
Are these wires connected?
Which is Correct???
Longer 
Lead of 
LED goes 
on top
Connect Circuit to Pi

Now that you have a working resistor and LED circuit, we  can now connect it to the Raspberry Pi.

Located on the Pi is a 40 pin general purpose input output (GPIO) connector.

The 40 pins can be connected to various inputs or outputs. In our experiment we will connect our LED circuit to two of the outputs.

To connect the GPIO header to the breadboard we will 
use a 40 pin cable.

The red stripe on the cable indicates pin number 1, which 
is in the upper left corner of the GPIO header.

We will connect the ribbon cable as follows:

Pin 6 will be used for ground (common to both LEDs)

Pin 11 will be used for our output to the Blue LED

Pin 12 will be used for our output to the Red LED

Connect a black or brown wire to pin 6, and a red or orange wire to pin 11 and pin 12 on the ribbon cable.

Plug the black wire to the negative side of the circuit and the red wire to the positive side.

Your project should look like this:


Program the Pi

Now that the Pi is setup and the electronics are connected, we will now begin programming the GPIO output using a programming language known as Python.Python is an interpreted, object-oriented, high-level programing language.Python’s simple, easy to learn syntax emphasizes readability, which makes it easy to maintain and update.Python is included in the Raspbian distribution.We will run a simple python program to verify that our Pi is ready to go.Logon to the Pi with the username of piand the password of raspberryProgram the Pi

You should now see the shell prompt like:
Type sudopythonand you should see the following:

Type quit() to exit from the python interpreter.

This shows us that Python is installed and ready for action.

Program the Pi

We will now write our first Python program.

Type sudonanohelloworld.py and press Enter.

Nanois a text editor that can be used to type our code.

Type print “Hello World!”;

Once you have the code written, press CTRLand xat the  same time and press y to save the file.

Now type sudopython helloworld.py You should see:

We will now write some code to control the LED circuit.

Type sudonanoled.py and press Enter.

import RPi.GPIOas GPIO

import time

GPIO.setmode(GPIO.BOARD)

GPIO.setup(11,GPIO.OUT)

GPIO.output(11,GPIO.HIGH)

time.sleep(1)

GPIO.output(11,GPIO.LOW)

GPIO.cleanup()

Note, these are capital O’s, not 0’s (zeroes).

Add code to light up pin 12.

Program the Pi
Once you have the code written, press CTRLand xat the 
same time and press y to save the file.

Now type sudo python led.pyYou should see your LEDs 
light up for 1 second and turn off.

Modify your code to make the LEDs blink 3 times.
Program the Pi

import RPi.GPIOas GPIO

import time

GPIO.setmode(GPIO.BOARD)

GPIO.setup(11,GPIO.OUT)

for iin range (0,3):

GPIO.output(11, GPIO.HIGH)

time.sleep(1)

GPIO.output(11, GPIO.LOW)

time.sleep(1)

GPIO.cleanup()

{Be sure to use a tab instead of space here

Add code 
for pin 12!


Putting it all Together

You now have the skills to build a complete light show 
system!

Enjoy!

Introduction to Python


A readable, dynamic, pleasant, flexible, fast and powerful language

Nowell Strite 
Manager of Tech Solutions @ PBSnowell@strite.org
Overview
•Background
•Syntax
•Types / Operators / Control Flow
•Functions
•Classes
•Tools


What is Python
•Multi-purpose (Web, GUI, Scripting, etc.)
•Object Oriented
•Interpreted
•Strongly typed and Dynamically typed
•Focus on readability and productivity

Features
•Batteries Included
•Everything is an Object
•Interactive Shell
•Strong Introspection
•Cross Platform
•CPython, Jython, IronPython, PyPy


Who Uses Python
•Google
•PBS
•NASA
•Library of Congress
•the ONION
•...the list goes on...


Releases

•Created in 1989 by Guido Van Rossum
•Python 1.0 released in 1994
•Python 2.0 released in 2000
•Python 3.0 released in 2008
•Python 2.7 is the recommended 
version
•3.0 adoption will take a few years


Syntax


Hello World

hello_world.py


Indentation
•Most languages don’t care about  indentation
•Most humans do
•We tend to group similar things together


IndentationThe else here actually belongs to the 2nd if  statement 

IndentationThe else here actually belongs to the 2nd if statement
IndentationI knew a coder like this
IndentationYou should always be explicit
Indentation

Python embraces indentation
Comments


Types
Strings
Numbers
Null
Lists
Lists
Dictionaries
Dictionary Methods
Booleans
Operators
Arithmetic
String Manipulation
Logical Comparison
Identity Comparison


Arithmetic 
Comparison
Control Flow


Conditionals


For Loop


Expanded For Loop While Loop
List Comprehensions
•Useful for replacing simple for-loops.


Functions
Basic Function
Function Arguments
Arbitrary Arguments
Fibonacci
Fibonacci Generator
Classes
Class Declaration


Class Attributes
•Attributes assigned at class 
declaration should always be 
immutable


Class Methods


Class Instantiation & Attribute Access


Class Inheritance
Python’s Way
•No interfaces
•No real private attributes/functions
•Private attributes start (but do not end) with double underscores.
•Special class methods start and end with double underscores.
•__init__, __doc__, __cmp__, 
__str__



Imports
•Allows code isolation and re-use
•Adds references to variables/classes/functions/etc. into current namespace


Imports
More Imports
Error Handling
Documentation
Docstrings
Tools
Web Frameworks

•Django
•Flask
•Pylons
•TurboGears
•Zope
•Grok


IDEs
•Emacs
•Vim
•Komodo
•PyCharm
•Eclipse (PyDev)


Package Management


Resources
http://python.org/
http://diveintopython.org/
http://djangoproject.com/


Example
Going Further
•Decorators
•Context Managers
•Lambda functions
•Generators


COMPUTER VISION USING  SIMPLECV AND THE RASPBERRY PI 

Reference: 
Practical Computer Vision with SimpleCV -Demaagd (2012) 

Enabling Computers To See 

SimpleCV is an open source framework for building  computer vision applications.  
With it, you get access to several high-powered  computer vision libraries such as OpenCV – without having to first learn about bit depths, file formats,  color spaces, buffer management, eigenvalues, or  matrix versus bitmap storage. 

This is computer vision made easy. 

SimpleCV is an open source framework 
It is a collection of libraries and software that you can use to develop vision applications. 
It lets you work with the images or video streams that come from webcams, Kinects, FireWire and IP cameras, or mobile phones. 

It’s helps you build software to make your various technologies not only see the world, but understand it 
too.  
SimpleCV is written in Python, and it's free to use. It runs on Mac, Windows, and Ubuntu Linux, and it's licensed 
under the BSD license. 
Features 

Convenient "Superpack" installation for rapid deployment Feature detection and discrimination of Corners, Edges,
Blobs, and Barcodes  Filter and sort image features by their location, color, quality, and/or size An integrated iPython interactive shell makes developing code easy 

Image manipulation and format conversion 

Capture and process video streams from Kinect, Webcams, Firewire, IP Cams, or even mobile phones Learn how to build your own computer vision (CV)  applications quickly and easily with SimpleCV. You can access the book online through the Safari collection of the ITESM CEM Digital Library. 


SimpleCV Framework 6 
Installing SimpleCV for Ubuntu is done through a .deb package. From the SimpleCV download page, click the latest stable release link. This will  download the package and handle the installation of all the required dependencies. 
From the command line, type the following two commands: 
$ sudo apt-get install ipython python-opencv pythonscipy python-numpy python-pygame python-setuptoolspython-pip

$ sudo pip install https://github.com/ingenuitas/SimpleCV/zipball/master&nbsp;
The first command installs the required Python packages, and the second 
command uses the newly installed pip tool to install the latest version ofSimpleCV from the GitHub repository. 
Once everything is installed, you type 'simplecv' on the command line to launch the SimpleCV interactive shell. 
Note, however, that even recent distributions of Ubuntu may have an outdated version of OpenCV, one of the major dependencies for SimpleCV. If the installation 
throws errors with OpenCV, in a Terminal window enter: 
$ sudo add-apt-repository ppa:gijzelaar/opencv2.3$ sudo apt-get update 
Once SimpleCV is installed, start the SimpleCV interactive Python shell by opening a command prompt and entering python -m SimpleCV.__init__. A majority 
of the examples can be completed from the SimpleCV shell. 

Hello World 
from SimpleCV import Camera, Display, Image# Initialize the camera 
cam = Camera()
# Initialize the displaydisplay = Display()
# Snap a picture using the cameraimg = cam.getImage()
# Show the picture on the screenimg.save(display) 


Retrieve an Image-object from the camera with the highest quality possible. 

from SimpleCV import Camera, Display, Imageimport time 

  1. Initialize the camera 

cam = Camera()

  1. Initialize the display

display = Display()

  1. Snap a picture using the camera

img = cam.getImage()

  1. Show some text 

img.drawText("Hello World!")

  1. Show the picture on the screen

img.save(display)

  1. Wait five seconds so the window doesn't close right away

time.sleep(5) 
The shell is built using IPython, an interactive shell for Python development. 

Most of the example code in this book is written so that it could be executed as a standalone script. 
However, this code can be still be run in the SimpleCV shell. 
The interactive tutorial is started with the tutorial command: 
>>> tutorial
Introduction to the camera 
The simplest setup is a computer with a built-in webcam or an external video camera. These usually fall into a category called a USB Video Class (UVC) 
device. UVC has emerged as a “device class” which provides a standard way to control video streaming over USB. Most webcams today are now supported by UVC, and do not require additional drivers for basic operation. Not all UVC devices support all functions, so when in doubt, tinker with the camera in a program like guvcview to see what works and what does not. RPi Verified Peripherals: USB Webcams 

Camera Initialization 
The following line of code initializes the camera: 
from SimpleCV import Camera 

  1. Initialize the camera 

cam = Camera() 

..This approach will work when dealing with just one 
camera, using the default camera resolution, without 
needing any special calibration. 
Shortcut when the goal is simply to initialize the camera and 
make sure that it is working:
from SimpleCV import Camera 

  1. Initialize the camera 

cam = Camera()

  1. Capture and image and display it

cam.getImage().show()


The show() function simply pops up the image from the 
camera on the screen. It is often necessary to store the 
image in a variable for additional manipulation instead ofsimply calling show() after getImage(), but this is a 
good block of code for a quick test to make sure that the 
camera is properly initialized and capturing video. 

Example output of the basic camera example 
To access more than one camera, pass the camera_id as an 
argument to the Camera() constructor. 
• 
On Linux, all peripheral devices have a file created for them in the 
/dev directory. For cameras, the file names start with video and end 
with the camera ID, such as /dev/video0 and /dev/video1. 
from SimpleCV import Camera 

  1. First attached camera 

cam0 = Camera(0)

  1. Second attached camera 

cam1 = Camera(1)

  1. Show a picture from the first camera

cam0.getImage().show()

  1. Show a picture from the second camera

cam1.getImage().show() 
The SimpleCV framework can control many other camera properties. An easy example is forcing the resolution. Almost every webcam supports the standard resolutions of 320×240, and 640×480 (often called “VGA” resolution). Many newer webcams can handle higher resolutions such as 800×600, 1024×768, 1200×1024, or 
1600×1200. Here is an example of how to move text to the upper left quadrant of an image, starting at the coordinates (160, 120) on a 640×480 image: 

from SimpleCV import Camera 
cam = Camera(0, { "width": 640, "height": 480 }
img = cam.getImage(
)
img.drawText("Hello World", 160, 120)
img.show(

Notice that the camera’s constructor passed a new argument in the form of{"key":value} 

Drawing on Images in SimpleCV 20 

Layers 
Layers can be unlimited and what's ever above or on top ofthe other layer will cover the 
layer below it. Let's look at the simplecv logo with some text written on it. 
• 
Which in reality is an image with 3 layers, each layer has the text displayed. If we rotate the image and expand the 
layers you can get a better idea of what is really happening with image layers.
>>> scv = Image('simplecv') 
>>> logo = Image('logo') 
>>> #write image 2 on top of image 
>>> scv.dl().blit(logo) 

NOTE: Run help DrawingLayer for more information. 
Here is an example of how to move text to the upper left quadrant of an image, starting at the coordinates (160, 120) on a 640×480 image: 
from SimpleCV import Camera 
cam = Camera(0, { "width": 640, "height": 480 }
img = cam.getImage()
img.drawText("Hello World", 160, 120)
img.show(

Camera class attributes 
The Camera() function has a properties argument for basic cameracalibration. Multiple properties are passed in as a comma delimited list, with the entire list enclosed in brackets.  
Note that the camera ID number is NOT passed inside the brackets, since it is a separate argument. The configuration options are: 

  • width and height 
  • brightness 
  • contrast 
  • saturation 
  • hue 
  • gain 
  • exposure 

The available options are part of the computer’s UVC system. 

A live camera feed 
To get live video feed from the camera, use the live() function. 
from SimpleCV import Camera 
cam = Camera(
)
cam.live(

In addition to displaying live video feed, the live() function has two  other very useful properties. The live feed makes it easy to find both  the coordinates and the color of a pixel on the screen. To get the coordinates or color for a pixel, use the live() function as outlined in the example above. After the window showing the video feed appears, click the left mouse button on the image for the pixel of interest. The coordinates and color for the pixel at that location will then be displayed on the screen and also output to the shell. The coordinates will be in the (x, y) format, and the color will be displayed as an RGB triplet (R,G,B). 

Demonstration of the live feed 
To control the closing of a window based on the user interaction with the window: 

from SimpleCV import Display, Imageimport time 
display = Display()
Image("logo").save(display)
print "I launched a window" 
# This while loop will keep looping untilthe window is closed 
while not display.isDone():
time.sleep(0.1)
print "You closed the window"
 


The user will not be able to close the window by clicking the close button in the corner of the window. Checks the event queue and returns True if a quit event has been issued. It outputs to thecommand prompt, and not the image. Information about the mouse 
While the window is open, the following information about 
the mouse is available: 

mouseX and mouseY (Display class) 
The coordinates of the mouse 
mouseLeft, mouseRight, and mouseMiddle 
Events triggered when the left, right, or middle buttons on the mouse are clicked  
mouseWheelUp and mouseWheelDown Events triggered then the scroll wheel on the mouse is moved 

How to draw on a screen 
from SimpleCV import Display, Image, Color
winsize = (640,480)
display = Display(winsize)
img = Image(winsize) 

If the button is clicked, draw the circle. 

img.save(display) 
The image has a drawing layer, which is 
accessed with the dl() function. The 
drawing layer then provides access to 

while not display.isDone(): the circle() function. 
if display.mouseLeft: 
img.dl().circle((display.mouseX,display.mouseY),4,

Color.WHITE, filled=True)
img.save(display)
img.save("painting.png"

Example using the drawing application 
The little circles from the drawing act like a paint brush, coloring in a small region of the screen wherever the mouse is clicked. 

Examples=
from SimpleCV import Camera, Imageimport time 
cam = Camera()

  1. Set the number of frames to capture

numFrames = 10 

  1. Loop until we reach the limit set in numFrames

for x in range(0, numFrames):
img = cam.getImage()
filepath = "image-" + str(x) + ".jpg"
img.save(filepath)
print "Saved image to: " + filepath

time.sleep(60) 


Color

Introduction 

Although colorsounds like a relatively straightforward concept, differentrepresentations ofcolor are useful in different contexts. 
The following examples workwith an image ofThe Starry Night by Vincent van Gogh (1889)

In the SimpleCV framework, the colors of an individual pixel are extracted with the getPixel() function. from SimpleCV import Imageimg = Image('starry_night.png')
print img.getPixel(0, 0) Prints the RGB triplet for the pixel at (0,0), which will equal (71.0, 65.0, 54.0). 
Example RGB 
Original Image R-Component 

G-Component B-Component 

HSV 
One criticism of RGB is that it does not specifically model luminance. Yet the luminance/brightness is one of the most common properties to manipulate.In theory, the luminance is the relationship of the of R, G, and B values. In practice, however, it is sometimes more convenient to separate the color values from the luminance values. The solution is HSV, which stands for hue, saturation, and value. The color is defined according to the hue and saturation, while value is the measure of the luminance/brightness.  
The HSV color space is essentially just a transformation of the RGB color space because all colors in the RGB space have a corresponding unique color in the HSV space, and vice versa. 

Example HSV 
Original Image Hue 

Saturation Value (Intensity) 

RGB ñ 
HSV 
The HSV color space is often used by people because it corresponds better to how people experience color than the RGB color space does. 

It is easy to convert images between the RGB and HSV color spaces, as is demonstrated below. from SimpleCV import Imageimg = Image('starry_night.png')
hsv = img.toHSV()
print hsv.getPixel(25,25)
rgb = hsv.toRGB()
print rgb.getPixel(25,25) 

It converts the image from the original RGB to HSV. 
it prints the HSV 
values for the pixel 
It converts the image back 
to RGB 
it prints the HSV 
values for the pixel 
The HSV color space is particularly useful when dealing with an object that has a lot of specular highlights or reflections.In the HSV color space, specular reflections will have a 
high luminance value (V) and a lower saturation (S) component. The hue (H) component may get noisy depending on how bright the reflection is, but an object of solid color 
will have largely the same hue even under variable lighting. A grayscale image represents the luminance of the image, but lacks any color components. 
An 8-bit grayscale image has many shades of grey, usually on a scale from 0 to 255. 

The challenge is to create a single value from 0 to 255 out of the three values of red, green, and blue found in an RGB image. There is no single scheme for doing this, but it is done by taking a weighted average of the three.

from SimpleCV import Imageimg = Image('starry_night.png')
gray = img.grayscale()
print gray.getPixel(0,0)
 

getPixel returns the same number three times. This keeps a consistent format with RGB and HSV, which both return three values. To get the grayscale value for a particular pixel 
without having to convert the image to grayscale, 

use getGrayPixel(). 
The Starry Night, converted to grayscale Segmentation is the process of dividing an image into areas of related content.Color segmentation is based on subtracting away the 
pixels that are far away from the target color, while preserving the pixels that are similar to the color.  
The Image class has a function calledcolorDistance() that computes the distance between every pixel in an image and a given color. 
This function takes as an argument the RGB value of the target color, and it returns another image representing the distance from the specified color. Segmentation Example 


46 
from SimpleCV import Image, ColoryellowTool = Image("yellowtool.png") 

yellowDistBin = yellowDist.binarize(50).invert()
onlyYellow = yellowTool -yellowDistBinonlyYellow.show() 

yellowDist = yellowTool.colorDistance((223, 191, 29))1 

Basic Feature Detection

Introduction 
The human brain does a lot of pattern recognition to makesense of raw visual inputs. After the eye focuses on an object, the brain identifies the characteristics of the object —such as its shape, color, or texture— and then compares these to the characteristics offamiliar objects to match and recognize the object. 
In computer vision, that process of deciding what to focus on is called feature detection. A feature can be formally defined as “one or more measurements of some quantifiable property of an object, computed so that it quantifies some significant characteristics ofthe object” (Kenneth R. Castleman, Digital Image Processing, Prentice Hall, 1996). 
Easier way to think of it: a feature is an “interesting” part of an image. 
A good vision system should not waste time—or processing power—analyzing the unimportant or uninteresting parts of an image, so feature detection helps determine which pixels to focus on. In this session we will focus on the most basic types of features: blobs, lines, circles, and corners. If the detection is robust, a feature is something that could be reliably detected across multiple images. 

Detection criteria 
How we describe the feature can also determine the situations in which we can detect the feature. Our detection criteria for the feature determines whether we can: Find the features in different locations of the picture (position invariant) Find the feature if it’s large or small, near or far (scale invariant)Find the feature if it’s rotated at different orientations 
(rotation invariant) 

Blobs 


51 
• 
Blobs are objects or connected components, 
regions of similar pixels in an image. 
• 
Examples: 
a group of brownish pixels together, which might 
represent food in a pet food detector. 
• 
a group of shiny metal looking pixels, which on a 
door detector would represent the door knob 
• 
a group of matte white pixels, which on a 
medicine bottle detector could represent the cap. 
• 
Blobs are valuable in machine vision because 
many things can be described as an area of 
a certain color or shade in contrast to a 
background. 

52 
52 


• 
findBlobs() can be used to find objects that are 
lightly colored in an image. If no parameters are 
specified, the function tries to automatically detect 
what is bright and what is dark. 
Left: Original image of pennies; Right: Blobs detected 


Blob measurements 


53 
After a blob is identified we can: 

• 
measure a lot of different things: 
• 
area 
• 
width and height 
• 
find the centroid 
• 
count the number of blobs 
• 
look at the color of blobs 
• 
look at its angle to see its rotation 
• 
find how close it is to a circle, square, or rectangle —or 
compare its shape to another blob 

Blob detection and measurement 


from SimpleCV import Imagepennies = Image("pennies.png")
binPen = pennies.binarize()
blobs = binPen.findBlobs()
blobs.show(width=5) 
54 



1. 
Blobs are most easily detected on a binarized image. 
2. 
Since no arguments are being passed to the findBlobs() function, it 
returns a FeatureSet 
• 
list of features about the blobs found 
• 
has a set of defined methods that are useful when handling features 
3. 
show() function is being called on blobs and not the Image object 
• 
It draws each feature in the FeatureSet on top of the original image and 
then displays the results. 

Blob detection and measurement (3) 


55 
• 
After the blob is found, several other functions 
provide basic information about the feature, such as 
its size, location, and orientation. 
from SimpleCV import Image
pennies = Image("pennies.png"
)
binPen = pennies.binarize(
)
blobs = binPen.findBlobs(
)
print "Areas: ", blobs.area(
)
print "Angles: ", blobs.angle(
)
print "Centers: ", blobs.coordinates(


Blob detection and measurement (2) 


56 
• 
area function: returns an array of the area of each 
feature in pixels. 
• 
By default, the blobs are sorted by size, so the areas should 
be ascending in size. 
• 
angle function: returns an array of the angles, as 
measured in degrees, for each feature. 
• 
The angle is the measure of rotation of the feature away 
from the x-axis, which is the 0 point. (+: counter-clockwise 
rotation; -: , clockwise rotation) 
• 
coordinates function: returns a two-dimensional array of 
the (x, y) coordinates for the center of each feature. 

57 
57 


• 
If the objects of interest are darkly colored on a 
light background, use the invert() function. 
from SimpleCV import Imageimg = Image("chessmen.png")
invImg = img.invert()
blobs = invImg.findBlobs()
blobs.show(width=2)
img.addDrawingLayer(invImg.dl())
img.show() 



58 
58 


1. 
invert() function: turns the black chess pieces white and 
turns the white background to black 
2. 
findBlobs() function: can then find the lightly colored 
blobs as it normally does. 
3. 
Show the blobs. Note, however, that this function will 
show the blobs on the inverted image, not the original 
image. 
4. 
To make the blobs appear on the original image, take 
the drawing layer from the inverted image (which is 
where the blob lines were drawn), and add that layer 
to the original image. 

59 
59 


• 
In many cases, the actual color is more important 
than the brightness or darkness of the objects. 
• 
Example: find the blobs that represent the blue 
candies 

Finding Blobs of a Specific Color (2) 


60 
from SimpleCV import Color, Imageimg = Image("mandms.png")
blue_distance = 
img.colorDistance(Color.BLUE).invert()
blobs = blue_distance.findBlobs()
blobs.draw(color=Color.PUCE, width=3)
blue_distance.show()
img.addDrawingLayer(blue_distance.dl())
img.show() 




Left: the original image; Center: blobs based on the 
blue distance; Right: The blobs on the original image 


Finding Blobs of a Specific Color (3) 


61 
1. 
colorDistance() function: returns an image that shows how 
far away the colors in the original image are from the 
passed in Color.BLUE argument. 
• 
To make this even more accurate, we could find the RGB triplet 
for the actual blue color on the candy. 
• 
Because any colors close to blue are black and colors far awayfrom blue are white, we again use the invert() function to switchthe target blue colors to white instead. 
2. 
We use the new image to find the blobs representing the 
blue candies. 
• 
We can also fine-tune what the findBlobs() function discovers bypassing in a threshold argument. The threshold can either be an 
integer or an RGB triplet. When a threshold value is passed in, 
the function changes any pixels that are darker than the 
threshold value to white and any pixels above the value to 
black. 

Finding Blobs of a Specific Color (4) 


62 
3. 
In the previous examples, we have used the 
FeatureSet show() method instead of these two lines 
(blobs.show()). That would also work here. We’ve 
broken this out into the two lines here just to show that 
they are the equivalent of using the other method. 
• 
To outline the blue candies in a color not otherwise found in 
candy, they are drawn in puce, which is a reddish color. 
4. 
Similar to the previous example, the drawing ends up 
on the blue_distance image. Copy the drawing layer 
back to the original image. 

63 
63 

• 
Sometimes the lighting conditions can make color 
detection more difficult. 
• 
To resolve this problem use hueDistance() instead of 
colorDistance(). 
• 
The hue is more robust to changes in light 
from SimpleCV import Color, Imageimg = Image("mandms-dark.png")
blue_distance = img.hueDistance(Color.BLUE).invert()
blobs = blue_distance.findBlobs()
blobs.draw(color=Color.PUCE, width=3)
img.addDrawingLayer(blue_distance.dl())
img.show() 

Blob detection in less-than-ideal light 
conditions (2)
64 

Blob detection in less-than-ideal light 
conditions (2)
64 
Blobs detected with

Blobs detected with 
hueDistance() colorDistance() 


Lines and Circles65 

Lines 


66 
• 
A line feature is a straight edge in an image that 
usually denotes the boundary of an object. 
• 
The calculations involved for identifying lines can be a 
bit complex. The reason is 
• 
because an edge is really a list of (x, y) coordinates, and 
any two coordinates could possibly be connected by a 
straight line. 
Left: Four coordinates; Center: One possible scenario for lines connecting the points; 
Right: An alternative scenario 


Hough transform 


• 
The way this problem is handled behind-the-scenes 
is by using the Hough transform technique. 
• 
This technique effectively looks at all of the possible 
lines for the points and then figures out which lines 
show up the most often. The more frequent a line is, 
the more likely the line is an actual feature. 

68 
68 


Utilizes the Hough transform and returns a FeatureSet ofthe lines found 

• 
coordinates() 
• 
Returns the (x, y) coordinates of the starting point of the 
line(s). 
• 
width() 
• 
Returns the width of the line, which in this context is the 
difference between the starting and ending x coordinates 
of the line. 
• 
height() 
• 
Returns the height of the line, or the difference between the 
starting and ending y coordinates of the line. 
• 
length() 
• 
Returns the length of the line in pixels. 

69 
69 

• 
The example looks for lines on a 
block of wood. 
from SimpleCV import Imageimg = Image("block.png")
lines = img.findLines()
lines.draw(width=3)
img.show() 

• 
The findLines() function returns a 
FeatureSet of the line features. 
• 
This draws the lines in green on the 
image, with each line having a 
width of 3 pixels. 

70 
70 


• 
Threshold 
• 
This sets how strong the edge should before it is recognized as a 
line (default = 80) 
• 
Minlinelength 
• 
Sets what the minimum length of recognized lines should be. 
• 
Maxlinegap 
• 
Determines how much of a gap will be tolerated in a line. 
• 
Cannyth1 
• 
This is a threshold parameter that is used with the edge detection 
step. It sets what the minimum “edge strength” should be. 
• 
Cannyth2 
• 
This is a second parameter for the edge detection which sets the 
“edge persistence.” 

71 
71 


from SimpleCV import Image
img = Image("block.png"
)


# Set a low threshold 


lines = img.findLines(threshold=10)
lines.draw(width=3)
img.show(


Line detection at a lower threshold 


Circles 


72 
• 
The method to find circular features is called 
findCircle() 
• 
It returns a FeatureSet of the circular features it 
finds, and it also has parameters to help set its 
sensitivity. 

73 
73 


• 
Canny 
• 
This is a threshold parameter for the Canny edge detector. The 
default value is 100. If this is set to a lower number, it will find a 
greater number of circles. Higher values instead result in fewer 
circles. 
• 
Thresh 
• 
This is the equivalent of the threshold parameter for findLines(). It 
sets how strong an edge must be before a circle is recognized. 
The default value for this parameter is 350. 
• 
Distance 
• 
Similar to the maxlinegap parameter for findLines(). It determines 
how close circles can be before they are treated as the same 
circle. If left undefined, the system tries to find the best value,
based on the image being analyzed. 

74 
74 


• 
radius() 
• 
diameter() 
• 
perimeter() 
• 
It may seem strange that this isn’t called circumference, 
but the term perimeter makes more sense when dealing 
with a non-circular features. Using perimeter here 
allows for a standardized naming convention. 

75 
75 

• 
from SimpleCV import Image
• 
img = Image("pong.png")
1. circles = 
img.findCircle(canny=200,thresh=250,distance=15)
2. circles = circles.sortArea()
3. circles.draw(width=4)
4. circles[0].draw(color=Color.RED, width=4)
5. img_with_circles = img.applyLayers()
6. edges_in_image = img.edges(t2=200)
7. final = 
img.sideBySide(edges_in_image.sideBySide(img_with_circles)).scale(0.5)
• 
final.show() 
applyLayers() Render all of the layers onto the current image and return the result. 
Indicies can be a list of integers specifying the layers to be used. 
sideBySide() Combine two images as a side by side images. 


76 
76 


Image showing the detected circles 


Corners 


77 
• 
are places in an image where two lines meet. 
• 
unlike edges, are relatively unique and effective for 
identifying parts of an image. 
• 
For instance, when trying to analyze a square, a vertical line 
could represent either the left or right side of the square. 
• 
Likewise, detecting a horizontal line can indicate either the top or 
the bottom. 
• 
Each corner is unique. For example, the upper left corner could 
not be mistaken for the lower right, and vice versa. This makes 
corners helpful when trying to uniquely identify certain parts of a 
feature. 
• 
Note: a corner does not need to be a right angle at 90 
degrees 

78 
78 


• 
analyzes an image and returns the locations of all 
of the corners it can find 
• 
returns a FeatureSet of all of the corner features it 
finds 
• 
has parameters to help fine-tune the corners that 
are found in an image. 

79 
79 


from SimpleCV import Image= Image('corners.png')
imgimg.findCorners.show() 

• 
Notice that the example 
finds a lot of corners 
(default: 50) 
• 
Based on visual inspection,
it appears that there arefour main corners. 
• 
To restrict the number of 
corners returned, we can 
use the maxnum 
parameter. 

80 
80 


from SimpleCV import Imageimg = Image('corners.png')
img.findCorners.(maxnum=9).show() 


Limiting findCorners() to a maximum of 
nine corners 


The XBox Kinect 81 

Introduction 


82 
• 
Historically, the computer vision market has been dominated 
by 2D vision systems. 
• 
3D cameras were often expensive, relegating them to niche 
market applications. 
• 
More recently, however, basic 3D cameras have become 
available on the consumer market, most notably with the 
XBox Kinect. 
• 
The Kinect is built with two different cameras. 
• 
The first camera acts like a traditional 2D 640×480 webcam. 
• 
The second camera generates a 640×480 depth map, which 
maps the distance between the camera and the object. 
• 
This obviously will not provide a Hollywood style 3D movie, but it 
does provide an additional degree of information that is useful 
for things like feature detection, 3D modeling, and so on. 

Installation 


83 
• 
The Open Kinect project provides free drivers that 
are required to use the Kinect. 
• 
The standard installation on both Mac and Linux 
includes the Freenect drivers, so no additional 
installation should be required. 
• 
For Windows users, however, additional drivers must 
be installed. 
• 
Because the installation requirements from Open 
Kinect may change, please see their website for 
installation requirements at http://openkinect.org.&nbsp;

84 
84 

• 
The overall structure of working with the 2D camera 
is similar to a local camera. However, initializing the 
camera is slightly different: 
from SimpleCV import Kinect 
# Initialize the Kinect 
kin = Kinect()
# Snap a picture with the Kinectimg = kin.getImage()
img.show() 
Kinect() constructor 
(does not take any 
arguments) 
snap a picture 
with the Kinect’s 
2D camera 

85 
85 


• 
Using the Kinect simply as a standard 2D camera is a 
pretty big waste of money. The Kinect is a great tool 
for capturing basic depth information about an object. 
• 
It measures depth as a number between 0 and 1023, with 0 
being the closest to the camera and 1023 being the farthest 
away. 
• 
SimpleCV automatically scales that range down to a 0 to 
255 range. 
• 
Why? Instead of treating the depth map as an array of numbers, 
it is often desirable to display it as a grayscale image. In this 
visualization, nearby objects will appear as dark grays, whereas 
objects in the distance will be light gray or white. 

Depth Information Extraction (2) 


86 
from SimpleCV import Kinect 

  1. Initialize the Kinect 

kin = Kinect()

  1. This works like getImage, but returns depth information

depth = kin.getDepth()
depth.show() 

• 
The Kinect’s depth map 
is scaled so that it can 
fit into a 0 to 255 
grayscale image. 
• 
This reduces the 
granularity of the 
depth map. 
A depth image from the Kinect


87 
87 


• 
It is possible to get the original 0 to 1023 range 
depth map. 
• 
The function getDepthMatrix() returns a NumPy 
matrix with the original full range of depth values. 
• 
This matrix represents the 2×2 grid of each pixel’s 
depth. 
from SimpleCV import Kinect 

  1. Initialize the Kinect 

kin = Kinect()

  1. This returns the 0 to 1023 range depth map

depthMatrix = kin.getDepthMatrix()
print depthMatrix 


88 
88 

from SimpleCV import Kinect 

  1. Initialize the Kinect 

kin = Kinect()

  1. Initialize the display

display = kin.getDepth().show()

  1. Run in a continuous loop forever

while (True):

  1. Snaps a picture, and returns the grayscale depth map

depth = kin.getDepth()

  1. Show the actual image on the screen

depth.save(display) 


Networked Cameras 89 

Introduction 


90 
• 
The previous examples in this lecture have assumed 
that the camera is directly connected to the 
computer. 
• 
However, SimpleCV can also control Internet 
Protocol (IP) Cameras. 
• 
Popular for security applications, IP cameras contain a 
small web server and a camera sensor. 
• 
They stream the images from the camera over a web feed. 
• 
These cameras have recently dropped substantially in 
price. 
• 
Low end cameras can be purchased for as little as for 
a wired camera and for a wireless camera. 

91 
91 


• 
Two-way audio via a single network cable allows 
users to communicate with what they are seeing. 
• 
Flexibility: IP cameras can be moved around 
anywhere on an IP network (including wireless). 
• 
Distributed intelligence: with IP cameras, video 
analytics can be placed in the camera itself 
allowing scalability in analytics solutions. 
• 
Transmission of commands for PTZ (pan, tilt, zoom) 
cameras via a single network cable. 

92 
92 


• 
Encryption & authentication: IP cameras offer secure 
data transmission through encryption and authentication 
methods such as WEP, WPA, WPA2, TKIP, AES. 
• 
Remote accessibility: live video from selected cameras 
can be viewed from any computer, anywhere, and also 
from many mobile smartphones and other devices. 
• 
IP cameras are able to function on a wireless network. 
• 
PoE -Power over ethernet. Modern IP cameras have the 
ability to operate without an additional power supply. 
They can work with the PoE-protocol which gives power 
via the ethernet-cable 

93 
93 


• 
Higher initial cost per camera, except where cheaper 
webcams are used. 
• 
High network bandwidth requirements: a typical CCTV 
camera with resolution of 640x480 pixels and 10 frames 
per second (10 frame/s) in MJPEG mode requires about 3 
Mbit/s. 
• 
As with a CCTV/DVR system, if the video is transmitted over 
the public Internet rather than a private IP LAN, the system 
becomes open to a wider audience of hackers and hoaxers. 
• 
Criminals can hack into a CCTV system to observe security 
measures and personnel, thereby facilitating criminal acts and 
rendering the surveillance counterproductive. 

94 
94 


• 
Most IP cameras support a standard HTTP transport 
mode, and stream video via the Motion JPEG 
(MJPG) format. 
• 
To access a MJPG stream, use the 
JpegStreamCamera library. 
• 
The basic setup is the same as before, except that 
now the constructor must provide 
• 
the address of the camera and 
• 
the name of the MJPG file. 
MJPG: video format in which each video frame or interlaced field of a digital 
video sequence is separately compressed as a JPEG image. 


95 
95 


• 
In general, initializing an IP camera requires the 
following information: 
• 
The IP address or hostname of the camera 
(mycamera) 
• 
The path to the Motion JPEG feed (video.mjpg) 
• 
The username and password, if required. 
from SimpleCV import JpegStreamCamera

  1. Initialize the webcam by providing URL to the camera 

cam = JpegStreamCamera("http://mycamera/video.mjpg")
cam.getImage().show() 


96 
96 

• 
Try loading the URL in a web browser. It should 
show the video stream. 
• 
If the video stream does not appear, it may be that 
the URL is incorrect or that there are other 
configuration issues. 
• 
One possible issue is that the URL requires a login to 
access it. 

Authentication information 


97 
• 
If the video stream requires a username and 
password to access it, then provide that 
authentication information in the URL. 
from SimpleCV import JpegStreamCamera 

  1. Initialize the camera with login info in the URL 

cam = JpegStreamCamera("http://admin:1234@192.168.1.10/video.mjpg") 
cam.getImage().show() 


98 
98 


• 
Many phones and mobile devices 
today include a built-in camera. 
• 
Tablet computers and both the iOSand Android smart phones can be 
used as network cameras with apps 
that stream the camera output to an 
MJPG server. 
• 
To install one of these apps, searchfor “IP Cam” in the app marketplace 
on an iPhone/iPad or search for “IP 
Webcam” on Android devices. 
• 
Some of these apps are for viewing 
feeds from other IP cameras, so make 
sure that the app is designed as a 
server and not a viewer. 
IP Cam Pro 

by Senstic 


IP Webcam 

by Pavel Khlebovich 


99 

Advanced Features 100 

Introduction 


101 
• 
Previous lectures introduced feature detection and 
extraction, but mostly focused on common geometric 
shapes like lines and circles. 
• 
Even blobs assume that there is a contiguous grouping of 
similar pixels. 
• 
This lecture builds on those results by looking for more 
complex objects. Most of this detection is performed by 
searching for a template of a known form in a larger 
image. 
• 
This lecture also provides an overview of optical flow, 
which attempts to identify objects that change between 
two frames. 

102 
102 


• 
Finding instances of template images in a larger 
image Using Haar classifiers, particularly to identify 
faces 
• 
Barcode scanning for 1D and 2D barcodes 
• 
Finding keypoints, which is a more robust form of 
template matching 
• 
Tracking objects that move 

103 
103 


• 
Goal 
• 
learn how to use SimpleCV template matching functions 
to search for matches between an image patch and an 
input image 
• 
What is template matching? 
• 
Template matching is a technique for finding areas of 
an image that match (are similar) to a template image 
(patch). 

How does it work? 


104 
• 
We need two primary components: 
• 
Source image (I): The image in which we expect to find 
a match to the template image 
• 
Template image (T): The patch image which will be 
compared to the template image 
• 
Our goal is to detect the highest matching area: 

105 
105 


• 
To identify the matching area, we have to compare the 
template image against the source image by sliding it: 
• 
By sliding, we mean moving the patch one pixel at a time 
(left to right, up to down). At each location, a metric is 
calculated so it represents how “good” or “bad” the match 
at that location is (or how similar the patch is to that 
particular area of the source image). 

Bitmap Template Matching 


106 
• 
This algorithm works by searching for instances 
where a bitmap template—a small image ofthe object to be found—can be found within a 
larger image. 
• 
For example, if trying to create a vision system to 
play Where’s Waldo, the template image would 
be a picture of Waldo. 
• 
To match his most important feature, his face, the 
Waldo template would be cropped to right 
around his head and include a minimal amount of 
the background. 
• 
A template with his whole body would only match 
instances where his whole body was visible and 
positioned exactly the same way. The other 
component in template matching is the image to 
be searched. 

findTemplate() function 


107 
• 
The template matching 
feature works by calling the 
findTemplate() function on 
the Image object to be 
searched. 
Pieces on a Go board 

• 
Next, pass as a parameter 
the template of the object to 
be found. 
Template of black piece 


108 
108 


from SimpleCV import Image# Get the template and imagegoBoard = Image('go.png')
black = Image('go-black.png')
# Find the matches and draw them 
matches = goBoard.findTemplate(black)
matches.draw()
# Show the board with matches print the numbergoBoard.show()
print str(len(matches)) + " matches found." 
# Should output: 9 matches found. 
Matches for black pieces 


109 
109 


• 
The findTemplate() function also takes two optional 
arguments: 
• 
method 
• 
defines the algorithm to use for the matching 
• 
details about the various available matching algorithms can by found 
by typing help Image.findTemplate in the SimpleCV shell. 
• 
Threshold 
• 
fine-tunes the quality of the matches 
• 
works like thresholds with other feature matching functions 
• 
decreasing the threshold results in more matches, but could also result in 
more false positives. 
• 
These tuning parameters can help the quality of results, but 
template matching is error prone in all but the most 
controlled, consistent environments. 
• 
Keypoint matching, which is described in the next section, is 
a more robust approach. 

110 
110 


• 
Square difference matching method (CV_TM_SQDIFF) 
• 
These methods match the squared difference, so a perfect 
match will be 0 and bad matches will be large: 
• 
Correlation matching methods(CV_TM_CCORR) 
• 
These methods multiplicatively match the template against 
the image, so a perfect match will be large and bad 
matches will be small or 0. 

111 
111 


• 
Correlation coefficient matching methods 
(CV_TM_CCOEFF) 
• 
These methods match a template relative to its mean 
against the image relative to its mean, so a perfect 
match will be 1 and a perfect mismatch will be –1; a 
value of 0 simply means that there is no correlation 
(random alignments). 
where 

Normalized methods 


112 
• 
For each of the three methods just described, there 
are also normalized versions. 
• 
The normalized methods are useful because they 
can help reduce the effects of lighting differences 
between the template and the image. 
• 
In each case, the normalization coefficient is the 
same: 

Values of the method parameter for 
normalized template matching 
113 

Values of the method parameter for 
normalized template matching 
113 
• 
As usual, we obtain more accurate matches (at the cost 
of more computations) as we move from simpler 
measures (square difference) to the more sophisticatedones (correlation coefficient). 

Limitations 


114 
• 
This approach supports searching for a wide variety of 
different objects but: 
• 
The image being searched always needs to be larger than 
the template image for there to be a match. 
• 
It is not scale or rotation invariant. 
• 
The size of the object in the template must equal the size it 
is in the image. 
• 
Lighting conditions also can have an impact because the 
image brightness or the reflection on specular objects can 
stop a match from being found. 
• 
As a result, the matching works best when used in an 
extremely controlled environment. 

115 
115 


• 
Tracking an object as it moves between two images is a 
hard problem for a computer vision system. 
• 
This is a challenge that requires identifying an object in one 
image, finding that same object in the second image, and then 
computing the amount of movement. 
• 
We can detect that something has changed in an image,
using techniques such as subtracting one image from another. 
• 
While the feature detectors we’ve looked at so far are 
useful for identifying objects in a single image, they are 
often sensitive to different degrees of rotation or variable 
lighting conditions. 
• 
As these are both common issues when tracking an object in 
motion, a more robust feature extractor is needed. One 
solution is to use keypoints. 

116 
116 


• 
A keypoint describes an object in terms that are 
independent of position, rotation, and lighting. 
• 
For example, in many environments, detecting corners makegood keypoints. 
• 
As it was described before, a corner is made from two intersecting 
lines at any angle. 
• 
Move a corner from the top-left to the bottom-right of the screen 
and the angle between the two intersecting lines remains the 
same. 
• 
Rotate the image, and the angle remains the same. 
• 
Shine more light on the corner, and it’s still the same angle. 
• 
Scale the image to twice its size, and again, the angle remains the 
same. 
• 
Corners are robust to many environmental conditions and image 
transformations. 

117 
117 


• 
Keypoints can be more than just corners, but corners 
provide an intuitive framework for understudying 
the underpinnings of keypoint detection. 
• 
The keypoints are extracted using the 
findKeypoints() function from the Image library. 
• 
In addition to the typical feature properties, such as 
size and location, a keypoint also has a descriptor() 
function that outputs the actual points that are 
robust to location, rotation, scale, and so on. 

The extracted keypoints 
A common hotel room keycard 
118 
The extracted keypoints 
A common hotel room keycard 
118 
findKeypoints() example 

from SimpleCV import Imagecard = Image('keycard.png')
keys = card.findKeypoints()
keys.draw()

card.show() 

1. 
Find the keypoints on the image using 
the findKeypoints() method. 
2. 
Draw those keypoints on the screen. 
They will appear as green circles at 
each point. The size of the circle 
represents the quality of the keypoint. 

findKeypoints() function example (2) 


119 
• 
The above keypoints are not particularly valuable on 
their own, but the next step is to apply this concept to 
perform template matching. 
• 
As with the findTemplate() approach, a template image 
is used to find keypoint matches. However, unlike the 
previous example, keypoint matching does not work 
directly off the template image. 
• 
Instead, the process extracts the keypoints from the 
template image, and then looks for those keypoints in 
the target image. This results in a much more robust 
matching system. 

120 
120 


• 
Keypoint matching works best when the object has a lot 
of texture with diverse colors and shapes. 
• 
Objects with uniform color and simple shapes do not 
have enough keypoints to find good matches. 
• 
The matching works best with small rotations, usually 
less than 45 degrees. Greater or larger rotations could 
work, but it will be harder for the algorithm to find a 
match. 
• 
Whereas the findTemplate() function looks for multiple 
matches of the template, the findKeypointMatch() 
function returns only the best match for the template. 

121 
121 


• 
template (required) 
• 
The template image to search for in the target image. 
• 
quality 
• 
Configures the quality threshold for matches. 
• 
Default: 500; range for best results: [300:500] 
• 
minDist 
• 
The minimum distance between two feature vectors necessary in 
order to treat them as a match. The lower the value, the better 
the quality. Too low a value, though, prevents good matches from 
being returned. 
• 
Default: 0.2, range for best results: [0.05:0.3] 
• 
minMatch 
• 
The minimum percentage of feature matches that must be found 
to match an object with the template. 
• 
Default: 0.4 (40%); good values typically range: [0.3:0.7] 

122 
122 


• 
Flavor: a string indicating the method to use to 
extract features. 
• 
“SURF” -extract the SURF features and descriptors. If 
you don’t know what to use, use this. 
• 
http://en.wikipedia.org/wiki/SURF&nbsp;
• 
“STAR” -The STAR feature extraction algorithm 
• 
http://pr.willowgarage.com/wiki/Star_Detector&nbsp;
• 
“FAST” -The FAST keypoint extraction algorithm 
• 
http://en.wikipedia.org/wiki/Corner_detection#AST_based&nbsp;
_feature_detectors 

123 
123 


• 
shows a side-by-side image, with lines drawn between 
the two Images to indicate where it finds a match. 
• 
If it draws lots of lines, the matching algorithm should 
work. 

124 
124 


..shows a side-by-side image,
with lines drawn between the 
two Images to indicate where it 
finds a match. 
..If it draws lots of lines, the 
matching algorithm should 
work. 
from SimpleCV import Imagetemplate = Image('ch10-card.png')
img = Image('ch10-card-on-table.png')
match = img.findKeypointMatch(template)
match.draw(width=3)
img.show() 
draw a box around the detected object. 

125 
125 


• 
Note that keypoint matching only finds the single 
best match for the image. If multiple keycards were 
on the table, it would only report the closest match. 
The detected match for the keycard 

http://www.youtube.com/watch?v=CxPeoQDc2-Y&nbsp;


126 
126 


• 
The next logical step beyond template matching is to 
understand how to track objects between frames. 
• 
Optical flow is very similar to template matching in that 
it takes a small area of one image and then scans the 
same region in a second image in an attempt to find a 
match. 
• 
If a match is found, the system indicates the direction of 
travel, as measured in (X, Y) points. 
• 
Only the two-dimensional travel distance is recorded. If 
the object also moved closer to or further away from 
the camera, this distance is not recorded. 

127 
127 


• 
computes the optical flow 
• 
has a single required parameter, 
• 
previous_frame, which is the image used for 
comparison. 
• 
The previous frame must be the same size. 

128 
from SimpleCV import Camera, Color, Displaycam = Camera()
previous = cam.getImage()
disp = Display(previous.size())
while not disp.isDone():
current = cam.getImage()
motion = current.findMotion(previous)
for m in motion: 
m.draw(color=Color.RED,normalize=False)
current.save(disp)
previous = current Draw the little red motion 
lines on the screen to 
indicate where the motion 
occurred. 
128 
from SimpleCV import Camera, Color, Displaycam = Camera()
previous = cam.getImage()
disp = Display(previous.size())
while not disp.isDone():
current = cam.getImage()
motion = current.findMotion(previous)
for m in motion: 
m.draw(color=Color.RED,normalize=False)
current.save(disp)
previous = current Draw the little red motion 
lines on the screen to 
indicate where the motion 
occurred. 

129 
129 


Optical flow example 

• 
The findMotion function supports several different algorithms to detect 
motion. For additional information, type help Image in the SimpleCV shell. 
http://www.youtube.com/watch?v=V4r2HXGA8jw&nbsp;


130 
130 


• 
previous_frame -The last frame as an Image. 
• 
window -The block size for the algorithm. For the the HS and LK 
methods this is the regular sample grid at which we return motion 
samples. For the block matching method this is the matching windowsize. 
• 
method -The algorithm to use as a string. Your choices are: 
• 
‘BM’ -default block matching robust but slow -if you are unsure use this. 
• 
http://en.wikipedia.org/wiki/Block-matching_algorithm&nbsp;
• 
‘LK’ -Lucas-Kanade method 
• 
http://en.wikipedia.org/wiki/Lucas%E2%80%93Kanade_method&nbsp;
• 
‘HS’ -Horn-Schunck method 
• 
http://en.wikipedia.org/wiki/Horn%E2%80%93Schunck_method&nbsp;
• 
aggregate -If aggregate is true, each of our motion features is the 
average of motion around the sample grid defined by window. Ifaggregate is false we just return the value as sampled at the 
window grid interval. For block matching this flag is ignored. 

Haar-like Features 


131 
• 
Unlike template matching, Haar-like Features are 
used to classify more generic objects. 
• 
They are particularly popular for face detection, 
where the system determines whether an object is a 
generic face. 
• 
This is different from face recognition, which tries to 
identify whose face is in the image and is a much more 
complicated process. 
• 
Simply knowing that an object is a face is useful for 
segmenting the image, narrowing down a region of 
interest, or simply doing some other fun tricks. 

132 
132 


• 
Technically, Haar-like features refer to a way of slicing 
and dicing an image to identify the key patterns. 
• 
The template information is stored in a file known as a 
Haar Cascade, usually formatted as an XML file. 
• 
This requires a fair amount of work to train a classifier 
system and generate the cascade file. 
• 
Fortunately, SimpleCV includes some common face detection 
cascades, and additional ones are available on the Internet. 
• 
http://en.wikipedia.org/wiki/Haar-like_features&nbsp;
• 
http://alereimondo.no-ip.org/OpenCV/34&nbsp;

built-in cascades 


133 
• 
The built-in cascades with SimpleCV include: 
• 
Forward-looking faces 
• 
Profile faces 
• 
Eyes 
• 
Noses 
• 
Mouths 
• 
Ears 
• 
Upper and lower bodies 

134 
134 


• 
Haar-like features are detected with the findHaarFeatures() functionof the Image library. 
• 
Parameters: 
• 
cascade 
• 
The path the Haar Cascade XML file. 
• 
scale_factor 
• 
The Haar Cascades are technically sensitive to the image’s scale. To simulate 
scale invariance, the algorithm is run multiple times, scaling the template up 
with each run. The amount that the template should be scaled is controlled bythe scale_factor. Default: 1.2 (it scales up the template 20% each time). 
• 
min_neighbors 
• 
This is like a threshold value in other feature detectors. Technically, it deals 
with the number of adjacent detections needed to classify the object, but 
conceptually it is easier to think of as a threshold parameter. Default: 2. 
• 
use_canny 
• 
The default value is to use Canny pruning to reduce false positives in 
potentially noisy images. This is usually always the best option. 

135 
from SimpleCV import Camera, Displaycam = Camera()
disp = Display(cam.getImage().size())
while disp.isNotDone():
img = cam.getImage()
# Look for a face 
faces = img.findHaarFeatures('face')
if faces is not None: 
# Get the largest face 
faces = faces.sortArea()
bigFace = faces[-1]
# Draw a green box around the face 
bigFace.draw()
img.save(disp) 
loads one of the built-in 
Haar Cascade files 
135 
from SimpleCV import Camera, Displaycam = Camera()
disp = Display(cam.getImage().size())
while disp.isNotDone():
img = cam.getImage()
# Look for a face 
faces = img.findHaarFeatures('face')
if faces is not None: 
# Get the largest face 
faces = faces.sortArea()
bigFace = faces[-1]
# Draw a green box around the face 
bigFace.draw()
img.save(disp) 
loads one of the built-in 
Haar Cascade files 

findHaarFeatures() function example 


(2)
136 
• 
If working with a Haar Cascade that was 
downloaded from the Internet or created manually, 
simply specify the full path to the file. } 
• 
findHaarFeatures("/path/to/haarcascade_frontalface_alt.xml"
• 
The detection works like most other feature 
detectors. Simply pass the cascade file loaded 
earlier and it returns a FeatureSet with the matches. 
http://www.youtube.com/watch?v=c4LobbqeKZc&nbsp;

http://www.youtube.com/watch?v=aTErTqOIkss&nbsp;


Limitations 


137 
• 
Haar Cascades also work best in controlled 
environments. 
• 
The Haar Cascade for detecting forward-looking faces 
will not work well on an image with a face looking to 
the side. 
• 
Size and rotation of the object can also create 
complications, though the underlying algorithms are 
more flexible than with template matching. 

Barcode 


138 
• 
One-dimensional barcodes have become a ubiquitous part of the 
modern shopping experience. 
• 
Two-dimensional barcodes are becoming increasingly popular for 
storing contact information, URL’s, and other small bits of information. 
• 
This growing popularity is helped by the availability of open source 
tools that can be used to extract information from these barcodes. 
• 
One example of these tools is the ZXing library (pronounced “ZebraCrossing”), which is an image processing library for barcodes. 
• 
The SimpleCV framework works with the ZXing library, making it easy to 
process images of one-and two-dimensional barcodes as part of a 
vision system. 
• 
As of SimpleCV version 1.3, ZXing is not automatically installed. If it is 
not installed, trying to use the barcode functions will throw a warning 
and not work. For instructions on installing ZXing, type help 
Image.findBarcode om/p/zxinfrom the SimpleCV shell, or go to 
http://code.google .cg/ 

139 
139 


• 
To extract the information from a barcode, use the 
findBarcode() function. 
• 
It works with both one-dimensional and two-dimensional 
barcodes. 
• 
The image to be scanned can contain more than just a 
barcode, but the detector works best when there is not 
a lot of extra noise around it. 
• 
In addition, while most barcodes are designed to work 
in spite of a partial obstruction of the barcode, the 
detector works best when it has a clear view of the 
barcode. 

140 
140 


from SimpleCV import Image, Barcode# Load a one-dimensional barcode 
img = Image('upc.png')
barcode = img.findBarcode()
print barcode.data# Should output: 987654321098 
Example of a one-
dimensional barcode 


from SimpleCV import Image, Barcode# Load a QR barcode 
img = Image('qr.png')
barcode = img.findBarcode()
print barcode.data.rstrip()
# Should output: http://www.simplecv.org&nbsp;
Example of a QR 
barcode 


Why Learn Computer Vision? 


141 
• 
Computer vision is moving from a niche tool to an 
increasingly common tool for a diverse range ofapplications: 
• 
facial recognition programs or gaming interfaces like the 
Kinect 
• 
automotive safety systems where a car detects when the 
driver starts to drift from the lane, or when is getting drowsy 
• 
point-and-shoot cameras to help detect faces or other central 
objects to focus on 
• 
high tech special effects or basic effects, such as the virtual 
yellow first-and-ten line in football games, or the motion blurs 
on a hockey puck. 
• 
industrial automation, biometrics, medicine, and even 
planetary exploration 
• 
food and agriculture, where it is used to inspect and grade 
fruits and vegetables 
• 
It's a diverse field, with more and more interesting 
applications popping up every day. 

142 
142 


• 
Computer vision is built upon the fields of mathematics,
physics, biology, engineering, and of course, computer 
science. 
• 
There are many fields related to computer vision, such as 
machine learning, signal processing, robotics, and artificial 
intelligence. 
• 
Even though it is a field built on advanced concepts, more 
and more tools are making it accessible to everyone from 
hobbyists to vision engineers to academic researchers. 
• 
It is an exciting time in this field, and there are an endless 
number of possibilities for applications. 
• 
One of the things that makes it exciting is that these days, 
the hardware requirements are inexpensive enough to allow 
more casual developers entry into the field, opening the 
door to many new applications and innovations.