Recent trends in technology aim to build highly interactive and easy to use applications using non conventional interaction devices such as Web Cam, Projector, Data Gloves etc. Among them Web Cam can be used for development of variety of applications such as Virtual Mouse, Image Grabber etc. these applications are tracking various gestures of human hand by some means, and using them programmatically. The aim of this paper is to present fundamentals of tracking colors from an image and video, so that using those colors as markers, user may be able to track fingers in a real time video.