Project Concept:
I knitted a small bunny myself and decided to imagine what it might dream about. I became curious how its dreams would look if it could enter utterly impossible spaces. This gave rise to the concept of the «Dream Archive"—a series of images where the same knitted bunny finds itself at the bottom of the ocean, among pink clouds, on a staircase to the sky, and in a mysterious night-time room.
The main goal of the project is to train a neural network to recognize a specific toy and to test how well it can reproduce it under new conditions while maintaining its shape, color, and knit texture.
Source Images:
As a subject for learning, I chose my crocheted rabbit. It has light, textured yarn, black eyes, a small pink nose, and long ears.
For the dataset, I took 25 photos of the toy from various angles. It was important to me that the neural network would memorize not just the overall silhouette, but also the details that would identify my specific bunny.
Tools Used:Stable Diffusion XL — image generation. DreamBooth LoRA — training the model on photographs of my rabbit. Google Colab — code execution, training, and generation. Hugging Face — storing trained weights and results. SDXL Inpainting — generating the rabbit within already established spaces.
Description of the learning process:
First, I prepared photographs of the toy and uploaded them to Google Colab. Then, I installed the necessary libraries and configured Stable Diffusion XL training using DreamBooth LoRA.I used DreamBooth LoRA for the training. The neural network was trained on pictures of my crocheted bunny, after which I could generate it in new settings while preserving the toy’s distinctive features.During the process, I saved model checkpoints. For the final generations, I used the LoRA from training step 175, which was saved on Hugging Face.
Program Setup
the learning process
trained model
Initial generations:
First, I checked how well the neural network had memorized the toy’s appearance.
The model accurately reproduced the volumetric texture of the knit, the light color, and the main features of the rabbit’s face. However, difficulties arose when attempting to transport the rabbit into a fantastic space.
Sometimes the neural network would generate a recognizable rabbit, but leave it against a standard background similar to the original photos. In other instances, a beautiful, fairy-tale setting would emerge, but the character itself would disappear.
Failed generations
Change in approach:
I decided to divide the image creation into two stages.
First, I generated the space itself without a trained LoRA so that the model could create unusual environments more freely. Then, I used Inpainting and my LoRA to generate the rabbit in a specific part of the image.
Generation Results:
dream «Above the Clouds»
The bed floats above an endless sea of pink clouds. I wanted to create a serene, slightly surreal space where the conventional bedroom literally dissolves its boundaries.The setting came out close to what I envisioned, but placing the rabbit within it proved more challenging. I had to work separately on the character’s presence and its placement on the bed.
Dream: «Stairway to Nowhere»
A rabbit is found on a staircase against a backdrop of purple landscape and a vast sunset sky. The steps become part of an unusual space where it is difficult to determine where reality ends.
«After Midnight» dream
