Runway just released the paper behind their new Diffusion-based video generation tool!
"Our model is trained on images+videos which exposes explicit control of temporal consistency through a novel guidance method."
π: arxiv.org
π οΈ: research.runwayml.com
1/π§΅
"Our model is trained on images+videos which exposes explicit control of temporal consistency through a novel guidance method."
π: arxiv.org
π οΈ: research.runwayml.com
1/π§΅
Text-guided generative diffusion models have recently been extended to video.
These approaches edit the content of existing footage while retaining structure require expensive re-training for every input or rely on error-prone propagation of image edits across frames.
These approaches edit the content of existing footage while retaining structure require expensive re-training for every input or rely on error-prone propagation of image edits across frames.
This paper presents a structure and content-guided video diffusion model that edits videos based on visual or textual descriptions of the desired output.
Often, conflicts between user-provided edits and structure representations occur due to insufficient disentanglement.
Often, conflicts between user-provided edits and structure representations occur due to insufficient disentanglement.
As a solution, training on monocular depth estimates with varying levels of detail provides control over structure and content fidelity.
This method resulted in successes such as fine-grained control over output characteristics and customization based on a few reference images.
This method resulted in successes such as fine-grained control over output characteristics and customization based on a few reference images.
If you want to stay up to date with the latest breakthroughs in AI check out our weekly summary.
We use ML to identify the top papers, news, and repos. It's read by 50,000+ engineers and researchers.
AlphaSignal.ai
We use ML to identify the top papers, news, and repos. It's read by 50,000+ engineers and researchers.
AlphaSignal.ai
Amazing work by @runwayml team: @pess_r, Johnathan Chiu, Parmida Atighehchian, @jongranskog, @agermanidis
Loading suggestions...