Step 3: Creating our Puppet

Step 3: Creating our Puppet

We use Automatic1111 on ‘RunPod’ (https://www.runpod.io/) to create the puppet.

If you don't want to use Automatic1111 locally on your computer, the developer of this kohya-LoRA notebook has just come out with a new 'Automatic1111’ notebook with ControlNet-1 and a new ControlNet-2 as well as the ability to use your newly trained LoRA files.

You can use the Cagliostro Colab UI as it is basically Automatic1111 to generate your images. Or use can use ComPhy or some other as well.

Load your LoRA files into your stable-diffusion-webui/model/LoRA folder .

You can test out all these files here by clicking in on each file. That will then load its tag up into the prompt. Test them all out one at a time.

You can also control the weights of these LoRA files as well by adjusting this figure.

Load your LoRA file with a prompt (as well as a negative prompt) and run it through and see what it looks like.

If you are happy with the overall state of your LoRA file, create an image of the model with a 'bald head'

We will take the 'bald image' and drag and drop that into the 'Headshot’ plugin which then generates a 3D model of the face. We can go in and sculpt this further if you want to. But We will just use this as a ‘Puppet’ or a guide for the AI. It doesn't have to look great. It kind of has to look sort of similar to the model.

Once you are happy with the face and the body shape, add some hair - just drag and drop from a library.

Once I've finished with your character, you will them export it to iClone. These programs work together in a pipeline. iClone is more of an animation program. So this is where you are going to add the facial animation as well as the body idol.

We use a plugin called ‘Motion Live’ for iClone. We just activate 'Motion Live' (you can download the ‘Motion Live’ app on your phone).

All you're doing is recording your facial movements and that's being applied to your 3D model in real time. So we have just dragged and dropped a lighting situation in here and got some Physics on the hair. And you are pretty much ready to render.

Go to the render panel and select PNG sequence. We actually want 768 by 768 in the end. Try and be divisible by 16 if you can. And we come down here and I'm just doing 300 to 500 frames. We are going to export those as PNG files.

We will go back into Automatic1111 and go through the settings you are using.

Use image-to-image and load your LoRA model.

Add a simple prompt because you don't want the prompt to fight the original image – and that's quite important. You want to keep the prompt as brief as possible. Do not put too many details that are not in the image that you want to create.

Enter the negative prompt which is as important as the main prompt.

You will get our rendered image added in here.

Set the 'sampling method' to DPM++ SDE. You can use whatever you want. You get good results with Euler-a, LMS and LMS Karras and k-diffusion.

Sampling Steps. Keeping these relatively low.

Width and Height. Set as 768 by 768 - the same as your input.

De-noising Strength. Keep this low - a strength at 11 is fine.

CFG Scale. Set this down too 6.

We want to pull from this image as much as we can and apply the LoRA over the top without the prompt compromising things too much.

So we have already set our ‘Seed’ because we have tested this out already. When you do this just render a -1 Seed until you get something you like and then lock it in with the Reuse button.

On ControlNet, we have enabled Preprocessor ‘hed’ and we have the Model enabled as well. We have not changed anything from the default settings here.

We have got a secondary model in as well which we have enabled ‘canny’ and enabled the canny model. Again we have not changed any of the default settings. So let's render and see what it creates us.

And it's following the mouth movements as well which we need for lip syncing. We have seen so much AI generation where the mouth is just a blurry mess.

Using this ‘Workflow’ you can now get precise mouth lip syncing. This is why we made the model on a person so you can see that it is working.

We now go back to 'batch render' these frames which will give us a frame by frame animation. To do that, go to 'batch' and put your input to where your 3D files are located on your drive. And then your output where you want them to render to. And then press 'generate'.

Congratulations. We have completed our animation all that is rendering.

You can literally switch your checkpoint file and run that again and look at that. It's created a semi-anime style version with a click of a button. So you can literally change the render type but by using your diffusion checkpoint. I

If you have DaVinci Resolve, use the Dirt removal and the D-flicker times two because that will give you really nice move results to your animation.