The interaction between man and computer activities of daily life is increasingly becoming an important part of, especially in recent years, with the rapid development of computer technology to study the habits of the new line interpersonal communication interpersonal interaction techniques become very active at the same time also made encouraging progress, these studies include face recognition, facial expression recognition, gesture recognition, etc.. Gesture is a natural, intuitive, easy to learn human-computer interaction means. Manual directly as computer input devices, human-computer communication between the middle of the media will no longer need, the user can simply define an appropriate gesture to the surrounding machine control, therefore, gesture recognition is a human and robot interaction an important tool, but also human-computer interaction, virtual reality, an important component. However, the gesture itself, the diversity, ambiguity and the staff is a complex deformation body and the visual is inherently uncertain, coupled with staff is a complex deformation of their body and visual discomfort qualitative, making vision-based gesture recognition is a very challenging interdisciplinary research topic. This article focuses on the general context of gesture segmentation and the use of BP neural network to recognize the gesture. Gestures are defined in advance, and the input to the neural network was trained, and then collected through the use of color image opencv extract-based skin color, the color images from complex background, hand in the image, further processing and enter the neural network identification.