基于核心人物和交互关系建模的群组行为识别

全文下载;  202203014.pdf

文章编号： 1672-6987（2022）03009809； DOI： 10.16351/j.1672-6987.2022.03.014

刘继超， 刘云*， 王传旭(青岛科技大学 信息科学技术学院，山东 青岛 266061)

摘要： 提出了一种层级关系网络与关键人物检测相结合的群组行为识别方法。首先从空间CNN和时间CNN提取每个群组成员的时空特征，经LSTM后进一步利用视频的长时序上下文关系，形成组群的时空级联特征；然后利用层级关系网络捕获群组成员交互关系，在每个关系层中构建无向关系图，通过共享多层感知机来捕获交互关系图中边的特征(即成员间的交互关系)，并将节点所有边的特征求和后作为此节点新的特征；为了获得高阶的层级交互关系表征，构建了多个关系层，并学习每个人的层级关系表征。同时，在关键人物检测网络，定义运动特征最强的一个成员为核心人物，依据与核心人物的空间距离和运动特征相关性，定义其他关键人物；再将所有关键人物的特征输入到BiLSTM，学习关键人物之间隐含的交互关系。为了进一步优化识别结果，将通过softmax层获得的群组识别候选标签的概率值输入CRF层，利用二元势函数鼓励外观特征和运动特征相近的群组分配相同标签，纠正由于学习偏差引起的错误，提高了群组识别精度；最后，在公开标准数据集Collective Activity Dataset 和Volleyball Database的平均识别率达到939%和912%，实验对比相同主干网络的最新方法，证明了本方法的有效性。

关键词： 组群行为识别； 核心人物建模； 交互关系建模； CNN； LSTM

中图分类号： TQ 207+.2文献标志码： A

引用格式： 刘继超， 刘云， 王传旭. 基于核心人物和交互关系建模的群组行为识别［J］. 青岛科技大学学报(自然科学版), 2022, 43(3): 98106.

LIU Jichao， LIU Yun， WANG Chuanxu. Group activity recognition based on relationship network and core person modeling［J］. Journal of Qingdao University of Science and Technology（Natural Science Edition), 2022， 43（3）： 98106.

Group Activity Recognition Based on Relationship

Network and Core Person Modeling

LIU Jichao， LIU Yun， WANG Chuanxu

(College of Information Science and Technology, Qingdao University of Science and Technology, Qingdao 266061， China)

Abstract: A group behavior recognition method combining hierarchical relationship network and key person detection is proposed. First, we extract the spatiotemporal features of each group member from the spatial CNN and temporal CNN. The long temporal context of the video is further used via LSTM to form the spatiotemporal cascading features of the group; then the hierarchical relationship network is used to capture the interaction of group members.we construct an undirected relationship graph in each relationship layer, capture the features of the edges in the interactive relationship graph by a sharing multilayer perceptron, and sum the features of all edges of the node as the new features. In order to obtain highlevel representations of hierarchical interaction relationships, we stack multiple relationship layers and learn each person’s hierarchical relationship representation. At the same time, in the key person network, the member with the strongest motion feature is defined as the core person, and other key persons are defined according to the spatial distance from the core person and the correlation of the motion characteristics; then the characteristics of all key persons are input into BiLSTM, to learn the implicit interaction between key persons. In order to further optimize the recognition results, the probability value of the group identification candidate label obtained through the softmax layer is input into the CRF layer, and the pairwise potential function is used to encourage groups with similar appearance characteristics and motion characteristics to assign the same label, and correct errors caused by learning bias, improved the accuracy of group recognition; finally, the average recognition rate of the public standard data set collective activity dataset and volleyball database reached 939% and 912%. Our experimental results show that our method improves the group activity recognition accuracy compared to the stateoftheart methods with the same backbone network.

Key words: group activity recognition；interaction relationship modeling；core person modeling； CNN； LSTM

收稿日期： 20210703

基金项目： 国家自然科学基金项目（61672305).

作者简介： 刘继超(1981—),男,博士研究生.*通信联系人.