I2V-Adapter: A General Image-to-Video Adapter for Diffusion Models

SIGGRAPH 2024 (Conference track)

Xun Guo1*‡ Mingwu Zheng2* Liang Hou2 Yuan Gao2 Yufan Deng2 Pengfei Wan2 Di Zhang2
Yufan Liu3 Weiming Hu3 Zhengjun Zha1 Haibin Huang2✝ Chongyang Ma2
✝Corresponding Author. *Equal contribution.
‡Work done as an intern at Kuaishou Technology.
1University of Science and Technology of China 2Kuaishou Technology
3Institute of Automation, Chinese Academy of Sciences


[Paper]      [BibTeX]

Gallery results (Left: input image; Right: output video)

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Ablation study on model design

Input Image
Full model
Input Image
w/o Content-Adapter
Input Image
w/o I2V-Adapter
Input Image
w/o both
Input Image
Full model
Input Image
w/o Content-Adapter
Input Image
w/o I2V-Adapter
Input Image
w/o both
Input Image
Full model
Input Image
w/o Content-Adapter
Input Image
w/o I2V-Adapter
Input Image
w/o both

Ablation study on Frame Similarity Prior

t0=0.6, p=1.0
t0=0.8, p=1.0
t0=1.0, p=1.0
t0=0.6, p=1.0
t0=0.8, p=1.0
t0=1.0, p=1.0
t0=0.6, p=1.0
t0=0.8, p=1.0
t0=1.0, p=1.0
t0=0.6, p=1.0
t0=0.8, p=1.0
t0=1.0, p=1.0
t0=0.6, p=1.0
t0=0.8, p=1.0
t0=1.0, p=1.0

With personalized T2I models

 
 
 
 

 
 
 
 

With ControlNet

Input Image
Pose Sequence
 
Input Image
Pose Sequence
 
Input Image
Pose Sequence
 

BibTeX

@inproceedings{guo2023i2v,
  author = {Xun Guo, Mingwu Zheng, Liang Hou, Yuan Gao, Yufan Deng, Pengfei Wan, Di Zhang, Yufan Liu,
  Weiming Hu, Zhengjun Zha, Haibin Huang, Chongyang Ma},
  title = {I2V-Adapter: A General Image-to-Video Adapter for Diffusion Models},
  year = {2024},
  booktitle = {SIGGRAPH 2024 Conference Papers}
}

Project page template is borrowed from DreamBooth.