---

Posted on Sunday 1 Jan 2023, 20:54 (last modified: 1 Jan 2023, 20:10)

---

There are many ways of turning a (color) sketch of an object or figure into a fully rendered artwork, but in the most simple cases I come across a problem. Here's a quick write-up of my attempt to solve it. If you know a better way, let me know!

Input image is a photo of a random color drawing by a child that was hanging at a local bakery:

The input image I use is a 256x256 crop of the more colorful bird:

The exact prompt and settings don't matter, but I'm basically trying to turn this into a digital rendering of a parrot in outer space. I use Protogen v2.2. Here's a cool result (I used parrot instead of bird in this attempt):

Certainly not bad. But I wanted to be in outer space, not in the clouds. The problem here is, that the white paper around the sketch doesn't give the model a lot of freedom. If I increase the denoising_strength, the result deviate too far from the input bird. In particular, the posture of the bird changes a lot.

inpainting doesn't work for me: If I mask the white paper, the original bird stays unchanged, and the style is far from consistent. You can probably go through various interations by carefully inpainting the background of the result above.

My solution is the following, manually adding noise instead of white paper (used gimp):

For good measure, I added some kind of structure below it's feet so that I can put the bird on a meteorite. Here's the output, exactly what I was going for: