Mouse control today has become an important tool to interact with computers. It is quite easy for those who are physically fit and mentally sound, but for those who are suffering from physical disabilities, It is quite difficult to use mouse for interacting with computers. Keeping this thing in mind, we in this research paper present a system known as “speech based mouse”. This speech based mouse will allow its users to control mouse pointer both in a discrete and continuous fashion. The application that we have developed will take input both in the form of words and non-vocal sounds. The algorithms that we have used to implement this system are MFCC technique for feature extraction and vector quantization with K-means clustering for speech recognition. Further we have proposed a new technique of phonetic feature extraction to cause continuous mouse motion possible.