Guiding AI animation with 3D data

This is one of my favorite projects I’ve worked on, after diving into ComfyUI and animate diff I’ve come up with a few workflows that will allow me to feed in 3D generated maps as an input to guide AI generation, being able to create consistent movement and shapes. The great part about this workflow is that the sky is the limit and render times from C4D are marginal as we don’t need beauty passes, so you can quickly iterate looks through AI and still get a consistent outcome.

This workflow works be using 5 exported maps from C4D: A depth pass for general motion guidance, a normal pass to extract more texture detail, a toon shader pass to reinforce the design and shape of the sneaker, a pass with just the swoosh to solidify it shows up consistently on each version and a cryptomatte pass to mask the different areas of the scene, so I can control the design of each area individually

ControlNet setup for the different passes.