Student, Department of Information Technology
Sri Sairam Engineering College
Student, Department of Information Technology
Sri Sairam Engineering College
Student, Department of Information Technology
Sri Sairam Engineering College
Associate Professor, Department of Information Technology
,Sri Sairam Engineering College
Human Computer Interface system using eye movement is introduced. Traditionally, human computer interface uses mouse or keyboard
input device. The proposed system
presents hands free interface between computer and user .The main objective is
to control the Mouse & Keypad using Eye ball. It also verify the user’s
authentication using Face Recognition. For Face recognition, Violo Jones
Algorithm is used. Camera is connected with the system & mat lab is used
for User Authentication. After successful authentication, camera is continued
to scan User’s Eye ball movement. During this state of action, Our Physical
Keypad & Mouse are freezed in order to stop user’s Key inputs. On-screen
Keyboard & Mouse control is initiated so as to control those through Eye
Ball movements. Mat lab software plays a vital role in controlling the on-screen
Keyboard & Mouse. We will be using Java for freezing the physical keypad
& mouse functionalities. Camera scans the Eye ball of the authenticated
user and control of the mouse is achieved through the Eye ball movement.
Alphabets are selected by Eye ball clicking for effective communication. The
Physical keyboard control is released by Control, Alt & Delete keys. Mat lab is used for Face recognition &
Eye ball control. On-screen Keyboard & Mouse are initiated & freezing
of Physical Mouse & Keypad is achieved by Java software.
Keywords—On-screen keyboard, Face recognition and Eye detection,
Web Camera, MAT lab, Computer, java,
In today’s world
technology gets upgraded to the newest level, majority computers rely on mouse
and keyboard as the major input devices which could not be used by handicapped
people. The proposed system describes a new method for the handicapped people
to communicate using computers with the help of eyes only. Most of the devices
such as computers and laptop prefer touch screen technology, but still the
preferred technology is not cheap enough to be used on desktop systems. The
main aim is to develop an interactive virtual human computer interface.
In our system,
we prefer the usage of Matlab to detect the web camera which is used for taking
images continuously to focus the eye pupil. With the help of various image
processing techniques, face recognition and eye tracking are done. For Face recognition,
Viola Jones Algorithm is used.
Now a days, people use computer by their
hands and touch pad. Traditionally,
human computer interface uses mouse
and keyboard as an input device. An idea to control computer mouse cursor movement with human eyes is
introduced1. Blink actions are introduced to replace the mouse clicks2. Generally for
opening a file, one must click on the file by using physical mouse or touch
pad. Instead a new system is introduced to replace the physical mouse. One can
open a file using the eye movement and blink actions. Both the left and right click is done by blinking
the left eye and right eye.
objective is to control the Mouse & Keyboard using Eye ball for handicapped
people. On-screen keyboard is displayed on the desktop to replace the physical
keyboard. Camera is mounted on the top of the desktop and the user image infront of the computer is captured. The image is
compared and verified with the database for
user’s authentication. After successful
keyboard is displayed on the desktop automatically. Once the On-screen keyboard
is displayed, the physical keyboard and mouse is freezed. The Physical keyboard
and mouse freeze is released by pressing Control, Alt & Delete keys. Camera
is continued to scan for face recognition using Violo Jones Algorithm. Then the
eye is detected successfully to control the mouse and on-screen keyboard. Based
on the blink action, the letter is typed and displayed for
The human eye is
an important organ that senses light. The important parts of human eye related
to eye tracking are described . The transparent coat in front of eyeball is
cornea. The muscle that controls the size of pupil is called iris, which is
like the aperture in a camera to let light goes inside. The iris has colour and
is different from person to person, thus can be used in biometrics. The tough
outer surface of the eyeball is sclera and appears white in the eye image. The
boundary between the sclera and the iris is called limbus. The captured eye
image by digital camera is shown in Figure below.
FIG 1: Structure
video camera (Logitech C170) is mounted above the desktop to observe the user’s
eyes who sits in front of the desktop. The camera captures the video image of
the eye and determines where the user is looking on the screen. No attached is
necessary to the user’s head or body. To “select” any letter on the On-screen
keyboard , the user looks at the letter for a specified period of time and to “press”
any letter, the user just needs to blink the eye. In this system, no
calibration procedure is required. For
this system, input is only eye. External hardware is not attached or required.
system starts with capturing an image of human face and detects an eye from the
face and converts to gray level, remove noises, converts to binary image,
calculate pixel value, detects sclera, divides eye and screen quadrants and
finally performs mouse functions such as mouse move, left click, right click, double
click, selection, drag & drop, and typing using on-screen keyboard according the pixel value. If sum of
white pixel value is zero, mouse click operation is performed, and if sum of
white pixel value of both eyes is one or more, mouse movement is performed.
6. Architecture Diagram
Truth table for
The detailed processing steps
is presented below:
Logitech C170 camera is fixed on the top of the
The camera takes the image of the user who sits
in front of the computer.
The image is compared and verified with the
database for user’s authentication.
After successful user’s authentication,
on-screen keyboard is displayed automatically on the desktop.
Once on-screen keyboard is displayed, the
physical keyboard and mouse are freezed by using java code.
Freezing can be released by pressing control,
alt and delete keys on same time.
Once on-screen keyboard is displayed, Camera
starts scanning the user eye by taking video.
After receiving these streaming videos from the
camera, it then breaks into frames.
9. After receiving frames, it checks for lighting
conditions because the camera requires sufficient lights from external sources
otherwise an error message will display
on the screen.
10. The captured frames that are already in RGB mode are
converted into Black ‘n’ White in order find the edge movement.
11. Images (frames) from the input source focusing the eye
are analyzed for Iris detection (center of eye) using Viola Jones Algorithm.
12. After that, a mid point is calculated by taking the
mean of left and right eye centre point.
13. Then, the mouse will move from one point to another on
the screen and user will perform clicking action by blinking their eyes.
Based on the blink action, the letter
is typed and displayed for effective communication.
8. Face Detection
The camera is
attached with the computer to capture images of the person using the system.
From the captured image, human face is detected and cropped in order to detect
the eyes. Face detection has been researched with a different methods that
often is motivated by a technique of face detector. Such techniques can use
colors, textures, features and templates. The following two techniques are
tried in this proposed system to select the best one.
1) Skin Color Analysis Method
analysis is often used as part of a face detection technique. Various
techniques and colorspaces can be used to divide pixels that belongs to skin
from pixels that are likely to the background.
faces with a big problem as skin colors are usually different over different
some cases skin colors may be similar to
colors with some measures. For example,
having a red
floor covering or a red wooden door in
the human image can
cause to fail the system.
performs set of features at a number of scales and at different locations and
uses them to identify whether an image is a face or not. A simple,
classifier is built by identifying a few
features out of the whole set of the Haar-like
can be generated using the AdaBoost 3 technique. To provide a real time
classifiers that containing set of features
together in a cascaded structure. According to Viola Jones algorithm 4, face
detection is performed using the facts that human eye is darker than upper
cheeks and forehead as presented in Fig.2 (c), and there is a brighter part in
between the two eyes that separates the left eye from the right eye as
presented in Fig.2 (b). The features required by the detection framework
generally performs the sum of image pixels in a rectangular area as presented
in Fig.2 (a). The features used by Viola and Jones algorithm are rely on more
than one rectangular areas. Fig.2 (a) presented the four types of features used
in viola Jones algorithm. Fig.2 (b) presented features that looks similar as
bridge of the nose. Fig.2 (c) presented feature looks similar to the eye is
darker than upper cheeks.
2: Viola Jones algorithm features
The value of a
given feature is the sum of the pixels of the unshaded rectangle subtracted
from the sum of the pixels within shaded rectangle 3. These rectangular
filters are very fast at any scale and location to capture characteristics of
the face. As it is must, the collection of all likely features of the four
types which can be produced on an image window is probably big; applying all of
them could be something intensive and could generate redundant activities. So,
only a small subset of features from the large set of features are used. The
advantage of Viola Jones algorithm is, its robustness with very high detection
rate and real time processing.
3-Processing structure of the proposed
9. Eye Detection
analysis 5, can be used to analyze
eye to cursor integration. Eye pair is
cropped from the cropped face by eliminating other face parts such as mouth,
resultant image is divided into two parts:
left eye and
right eye. The left and right eye images
from RGB to gray scale and then noise
is removed using
image enhancement techniques
and wiener filter).
image is converted into binary image(black and white) using threshold value.
structure of the proposed method is shown
in Fig. 3.
Gray scale images can be the result of measuring the
intensity of light at each pixel according to a particular weighted combination
of frequencies (or wavelengths). Gray scale conversion is done for edge
4- Gray scale image of eye.
2.) Image Enhancement
and improving image quality is used for better accuracy on computer vision.
Gaussian noise, balanced noise and the impulse noise 6. Impulse noise
distributed on the
image as light
and dark noise pixels and corrupts the
information of the image. Therefore, reducing
are key important in computer vision.
In this paper,
two methods of image enhancement
and wiener filter) are used. Those methods are used to remove noises.
3.) Image Binarization Using Threshold Value
In most of
vision systems, it is helpful to be separate
out the parts of
the image that is corresponding to which the image we are interested with, and
of the image
that corresponds to the background.
usually gives an easy and suitable way
to carry out
this segmentation based on different gray
intensities of an image.
A single or
multiple threshold levels could decide for the image to be segmented; for a
single value threshold level every pixel in the image should compare with a
given threshold value and if the pixel of the image intensity value is higher
than the assigned threshold value, the pixel is represented with white in the
output; on the contrary if the pixel of the image intensity value is less than
the assigned threshold value, the pixel is represented with black. For the multiple
threshold level there are groups of intensities to be represented to white
while the image intensities that have out of these groups are represented to
black. Generally thresholding is useful for rapid evaluation on image
segmentation due to its simplicity and fast processing speed 7. Image
binarization is the process of converting a gray level into black and white
image by using some threshold value.
5-Binary image after using single and multiple
Eye and Mouse-Cursor Integration
When both eyes
are opened, the left eye is divided into four quadrants to integrate with
divide the eye into four quadrants, center of the eye is a reference point. Eye
corner location is used to find widths and heights of an eye
which are used
to calculate center of eye. Using x and
that created at the corner of eye, center
of eye is
Fig.6 (a) presented eye quadrants
labeled with 1, 2, 3,
and 4, and Fig.6 (b) presented
quadrants of computer
screen that is
labeled with 1, 2, 3, and 4.
a)Eye quadrant b)Screen quadrant
Eye and Screen quadrants.
This paper is
fully focused on the development of
hands-free PC control- Eyeball movement based cursor and keyboard
control. This system does not require any wearable attachment is the most
unique aspect. The mouse pointer is operated using the user eye movement. This system
has been implemented in Windows 8.1 with 4GB RAM in Matlab environment using
Matlab Image Acquisition Processing Toolbox.
Toolbox is sufficient for the sensitivity of the system and to work in real
time as how normal mouse perform movement task.
keyboard is introduced to perform all keyboard
actions for effective
Our motive is to
create this technology in the lowest
cost possible way and also to create it under a standardized operating system
in more user friendly manner.
1 Craig Hennessey,
Jacob Fiset “Long Range Eye Tracking: Bringing Eye Tracking into the Living
Room”, IEEE, 2016
Patel, V. Prakash ” Autonomous Camera based Eye Controlled Wheelchair system
using Raspberry-Pi”, IEEE
Sponsored 2nd International Conference on Innovations in Information Embedded
and Communication Systems, 13 August 2015
object detection framework (n.d.). In
Retrieved February 22, 2016, from
and A.Geetha, “Cursor Control System Using Facial Exoressions for Human-Computer
Interaction”, ISSN: 0976-1353 vol 8 Issue 1 –April 2014.
5Ziho Kang and
Steven J. Landry, “An Eye Movement Analysis Algorithm for a Multielement
Target Tracking Task: Maximum Transition-Based Agglomerative Hierarchical
Clustering”, IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS,
VOL. 45, NO. 1,FEBRUARY 2015.
Cheng Huang, “An Improved Median Filtering Algorithm for Image Noise
Reduction”, Sciverse ScienceDirect, Elsevier, Physics Procedia 25(2012)
7Moe Win, A. R.
Bushroa, M. A. Hassan, N. M.Hilman, AriIde-Ektessabi, “A Contrast
Adjustment Thresholding Method for Surface Defect Detection Based on
Mesoscopy”,IEEE TRANSACTIONS ON INDUSTRIAL INFORMATICS,VOL. 11, NO.
3, JUNE 2015
Usman Ghani, Sarah Chaudhry, Maryam Sohail and Muhammad Nafees Geelani,
“GazePointer: A Real Time Mouse Pointer Control Implementation Based On
Eye Gaze Tracking”, IEEE, 978-1-4799-3043-2/13, 2013.
9Shrunkhala Satish Wankhede,
Mrs.S.A.Chhabria,Dr.R.V.Dharaska ” Controlling Mouse Cursor Using Eye
Movement”, Special Issue for National Conference On Recent Advances in
Technology and Management for Integrated Growth 2013 (RATMIG 2013)
10Akhil Gupta, Akash Rathi, Dr. Y. Radhika, “HANDS-
FREE PC CONTROL” CONTROLLING OF MOUSE CURSOR USING EYE MOVEMENT, International
Journal of Scientific and Research Publications, Volume 2, Issue 4, April 2012
Naveed, Bushra Sikander, and Malik Sikander Hayat Khiyal “Eye Tracking System
with Blink Detection”,IEEE,2012
Jixu Chen, Yan Tong ,Wayne Grayy ,Qiang Jiz “A Robust 3D Eye Gaze Tracking
Mircea Ionita, Peter Corcoran, “Statistical models of appearance for eye
tracking and eye blink detection and measurement”, IEEE transactions on
consumer electronics, Vol.54 , No.3, pg.
1312?1320 August 2009.
and G.Murugesan, “A Novel Approach for Human Computer Interface Based on
Eye Movements for Disabled People”, IEEE, 978-1-4799-6085-9/15,
15 Ryo Shimata,
Yoshihiro Mitani and Tsumoru Ochiai, “A Study of Pupil Detection and
Tracking by Image Processing Techniques for a Human Eye–Computer Interaction System”,
978-1-4799-8676-7/15/2015 IEEE , SNPD 2015,June 1-3 2015, Takamatsu,
and A.Geetha, “Cursor Control System Using Facial Exoressions for
Human-Computer Interaction”, ISSN: 0976-1353 vol 8 Issue 1 –April 2014.