基于深度生成模型的视觉模式表示与编码

摘要：认为早期智能编码方法的性能受限于手工设计的方案，当前基于神经网络的编码方法可解释性不足，不利于后续面向人机视觉的分析与交互。受生成模型的启发，生成式编码方法通过构建生成模型来实现图像和视频的压缩和合成，获得可解释的紧凑视觉表示并生成符合图像先验分布的高视觉质量内容。其中概念图像编码与概念视频编码利用生成模型强大的样本生成能力与紧凑层次视觉表示模型，实现了编码性能更优的图像与视频编码；跨模态语义编码对图像与文本域进行跨模态转换与编码，保持可解释的同时实现上千倍的超高压缩比与令人满意的重构结果。

关键词：智能视频编码；生成式编码；跨模态压缩；概念编码

Abstract: The performance of early intelligent encoding methods was limited by manually designed solutions, while current neural network-based encoding methods lack interpretability, which hinders subsequent analysis and interaction between humans and machine vision. Inspired by generative models, the generative encoding methods aim to achieve compression and synthesis of images and videos by constructing efficient generative models, obtaining interpretable compact visual representations, and synthesizing high-quality visual content that conforms to the prior distribution of images. Among them, conceptual image encoding and conceptual video encoding leverage the powerful sample generation capability and compact hierarchical visual representation models of generative models, resulting in superior encoding performance for images and videos. Cross-modal semantic coding, on the other hand, enables cross-modal transformation and coding between the image and text domains while maintaining interpretability, achieving ultra-high compression ratios of thousands of times and satisfactory reconstruction results.

Keywords: intelligent video encoding; generative encoding; cross-modal compression; conceptual coding