SOUNDING OBJECT FOCUSED SEGMENTATION FOR AN AUDIO-VISUAL SCENE
卷宗概要
申请人
International Business Machines Corporation
发明人
Shi Yun Liang; Yuan Yuan Ding; Yong Sun; Zhong Yuan Sun; Jing Zhang
IPC 分类
CPC 分类
An image feature vector for a given video frame is generated from a given video and an audio feature vector for audio of the given video is generated. A textual description of the given video frame is generated and textual feature vectors are generated from the textual description. A first set of audio features of the audio feature vector and visual features of the image feature vector are fused to generate fused audio-visual features. A second set of audio features of the audio feature vector and the textual feature vectors are fused to generate fused audio-text features. A final mask is generated based on the fused audio-visual features and the fused audio-text features.
原文（中文）
An image feature vector for a given video frame is generated from a given video and an audio feature vector for audio of the given video is generated. A textual description of the given video frame is generated and textual feature vectors are generated from the textual description. A first set of audio features of the audio feature vector and visual features of the image feature vector are fused to generate fused audio-visual features. A second set of audio features of the audio feature vector and the textual feature vectors are fused to generate fused audio-text features. A final mask is generated based on the fused audio-visual features and the fused audio-text features.