---

Posted on Thursday 29 Dec 2022, 15:30 (last modified: 29 Dec 2022, 15:30)

---

Introduction

A quick, fun experiment with Stable Diffusion: turning a photo of the Ruisdael Observatory into the style of.. Ruisdael.

I am working on publishing a dataset of solar radiation observation taken at a measurement site that's part of the Ruisdael Observatory, a network of atmospheric observatories in the Netherlands. Ruisdael is a famous Dutch painter, known (amongst other things) for his realistic lighting and atmospheres:

The Ruisdael painting my PhD promotor Chiel uses for his project on cloud shadows

At this point (somewhere mid December 2022), I do not have much experience with generative AI models like Stable Diffusion yet, so I'm not expecting much. Here's the photograph I start with:

Photograph taken of the Ruisdael Observatory in Cabauw, with the 213 meter tall tower

Let's see what happens when you throw this into the img2img model of Stable Diffusion (v1.5) with the prompt:

painting, Ruisdael, color

I get results such as this one:

A ship! Obviously the modern type of tower in my picture was not known to 17th century Ruisdael. I try to be a bit more specific with my prompts, to steer the model into the right direction. I now use:

Tall tower, painting, ((Ruisdael)), colorful, clouds,

where the (()) emphasizes that part of the prompts. This gives funny variations like this one:

A medieval tower instead of a ship

The model seems to have too much freedom, the tower is much wider than the input image for example. You have control over this freedom via the denoising strength. Adding less noise to the starting image means the img2img model has less room to deviate. This comes at a risk of not being able to transfer a photograph into a painting style, though. After various attempts, I end up with this nice one:

The Ruisdael Observatory in Cabauw, in Ruisdael style

And finally, upscaled and resized to 1280x1280:

Upscaled result. Obviously not perfect, but not bad either... to be continued