REGION OF INTEREST PROMPT PROCESSING FOR LARGE MULTIMODAL MODELS
卷宗概要
发明人
Shubham VERMA; Sanjay RAMANUJAN; Rakesh KELKAR; Ashwini KATARIA; Sagar TANEJA
IPC 分类
CPC 分类
A method for processing a multimodal prompt. The method includes receiving a multimodal prompt including a media file and information related to a region of interest (ROI) of the media file. The method further includes determining a ROI of the media file based on the information related to the media file and generating a plurality of media tiles of interest associated with the ROI. The method further includes encoding the plurality of media tiles of interest and using a large multimodal model (LMM) to process the encoded plurality of media tiles of interest according to a natural-language input of the prompt to generate a response.
原文（中文）
A method for processing a multimodal prompt. The method includes receiving a multimodal prompt including a media file and information related to a region of interest (ROI) of the media file. The method further includes determining a ROI of the media file based on the information related to the media file and generating a plurality of media tiles of interest associated with the ROI. The method further includes encoding the plurality of media tiles of interest and using a large multimodal model (LMM) to process the encoded plurality of media tiles of interest according to a natural-language input of the prompt to generate a response.