In video surveillance, image segmentation in outdoor scenes is a most important and complex task. A novel approach for video object segmentation in outdoor environment is described by using SRM (Statistical Region Merging) algorithm in this paper. Here we are going to identify both structured (e.g. persons, buildings, car, etc.) and unstructured background objects (e.g. sky, road, grass, etc.) which are containing the some characteristic based on color, intensity and texture in sequence. Our main objective of this work is to solve the over segmentation of objects while segmenting images in outdoor environment in a video surveillance system. This work is divided into four modules; preprocessing, bottom up segmentation, boundary detection and region merging. In pre-processing the input image is converted into CIE (Commission Internationale d'eclairage) color space technique. Bottom-up segmentation process is used to capture the structured and unstructured image characteristics. Another process is the Ada boost classifier which is used to classify the background objects in outdoor environment scenes. Then the contour maps are used to detect the boundary energy. Finally the statistical region merging provides the groupings of images to identify the computer vision.