Semantic Pyramids for Gender and Action Recognition

• Published in: IEEE transactions on image processing Vol. 23; no. 8; pp. 3633 - 3645
• Main Authors: Khan, Fahad Shahbaz, van de Weijer, Joost, Anwer, Rao Muhammad, Felsberg, Michael, Gatta, Carlo
• Format: Journal Article
• Language: English
• Published: New York, NY IEEE 01-08-2014; Institute of Electrical and Electronics Engineers; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Actigraphy - methods; Algorithms; Applied sciences; Artificial Intelligence; Biometry - methods; Body parts; Computer science; control theory; systems; Computer vision; Detection, estimation, filtering, equalization, prediction; Detectors; Exact sciences and technology; Experiments; Face; Face recognition; Feature extraction; Female; Gender; Humans; Image Enhancement - methods; Image Interpretation, Computer-Assisted - methods; Image processing; Image recognition; Information, signal and communications theory; Male; Object recognition; Pattern recognition; Pattern Recognition, Automated - methods; Pattern recognition. Digital image processing. Computational geometry; Recognition; Reproducibility of Results; Semantics; Sensitivity and Specificity; Sex Determination Analysis - methods; Signal and communications theory; Signal processing; Signal, noise; State of the art; Telecommunications and information theory; Whole Body Imaging - methods; Gender recognition; pyramid representation; bag-of-words; action recognition; Biometrics; Performance evaluation; Computer vision; State of the art; Face recognition; Measurement sensor; Sex; Benchmarking; Information extraction; Pattern recognition; Annotation; Semantics; Information processing; Fixed image; Signal processing; Feature extraction; Automatic recognition; Motion analysis
• ISSN: 1057-7149; 1941-0042; 1941-0042
• DOI: 10.1109/TIP.2014.2331759

Person description is a challenging problem in computer vision. We investigated two major aspects of person description: 1) gender and 2) action recognition in still images. Most state-of-the-art approaches for gender and action recognition rely on the description of a single body part, such as face or full-body. However, relying on a single body part is suboptimal due to significant variations in scale, viewpoint, and pose in real-world images. This paper proposes a semantic pyramid approach for pose normalization. Our approach is fully automatic and based on combining information from full-body, upper-body, and face regions for gender and action recognition in still images. The proposed approach does not require any annotations for upper-body and face of a person. Instead, we rely on pretrained state-of-the-art upper-body and face detectors to automatically extract semantic information of a person. Given multiple bounding boxes from each body part detector, we then propose a simple method to select the best candidate bounding box, which is used for feature extraction. Finally, the extracted features from the full-body, upper-body, and face regions are combined into a single representation for classification. To validate the proposed approach for gender recognition, experiments are performed on three large data sets namely: 1) human attribute; 2) head-shoulder; and 3) proxemics. For action recognition, we perform experiments on four data sets most used for benchmarking action recognition in still images: 1) Sports; 2) Willow; 3) PASCAL VOC 2010; and 4) Stanford-40. Our experiments clearly demonstrate that the proposed approach, despite its simplicity, outperforms state-of-the-art methods for gender and action recognition.