Photo series on neural network training

The core of this project is an experiment with the capability of a neural network to remember and reproduce the appearance of a specific character. I chose my dog, Audrey, as that character.

I trained the model on a small selection of her photographs to then translate that familiar image into entirely new scenarios that did not exist in reality. As a result, Audrey finds herself in unusual interiors and circumstances while maintaining her recognizable features.

The project explores the boundary between a real photograph and a generated image: the initial material is a real character, and the neural network creates a new, fictional reality for them. The culmination of the experiment is a series of photographs depicting Audrey’s peculiar alternative life.

The first step was preparing a dataset consisting of 22 photographs. I used both pictures I took with a camera and shots taken with an iPhone. When selecting photographs, I prioritized frames where the dog was in various poses. This allowed the model to capture the animal’s unique features more accurately, which could then be reproduced in the generated images.

Before uploading the images to Google Colab, I processed them: all photographs were converted to a square format. Subsequently, a textual description was created for each photograph.

For this experiment, I used Stable Diffusion XL—a generative model capable of creating images from text prompts. The goal of the project was to train the model to reproduce the likeness of a specific character—my dog, Odrich. To adapt the model, I utilized the LoRA (Low-Rank Adaptation) method.

LoRA allows for the fine-tuning of a small portion of the model’s parameters without retraining the entire Stable Diffusion XL model. This resulted in a separate set of LoRA weights that connect to the base model, enabling it to reproduce Odrich’s distinctive physical traits.

The training was conducted in Google Colab using an NVIDIA Tesla T4 GPU. The training was set for 500 steps, a batch size of 1, gradient accumulation over 4 steps, and a learning rate of 0.0001. Mixed precision FP16 was also employed to reduce VRAM consumption.

After training, the LoRA was integrated with Stable Diffusion XL. By using the keyword «odrichdog,» I generated new images, placing Odrich in situations and environments not present in the original dataset. Thus, the final series demonstrates how capable the trained model is of maintaining a recognizable character identity while altering the setting and narrative.

Initially, the plan was to train the model for collaboration with the St. Petersburg coffee shop LUCO, whose visual communication is characterized by using photographs of customer dogs. Unfortunately, at this stage, the model encountered generation difficulties. Changing the setup did not help improve the generated storylines, so I had to abandon this idea.

An attempt was also made to create merchandise for the coffee shop, which was also unsuccessful. The direction for future work was decided to change completely.

After completing the course, I received images where the model utilized my dog’s characteristics to create photographs of it in new settings.

Some of the images turned out a bit «blurry, ” but this is easily explained: some of the photographs in the dataset were taken with a shifted focus, which is part of the artistic style.

The final series is a collection of generated photographs where Odrich is situated in spaces and scenarios absent from the original dataset. The concept underpinning the series was «Odrich’s alternative life»: a real and familiar character is transported by the neural network into a fictional reality. In some images, she sits in a red armchair; in others, she is found on a large bed or amidst interiors that never appeared in the original photographs.

When creating this series, the focus was not simply on generating images of the dog, but on testing whether the model could maintain the distinctive features of a specific character even when the environment was completely changed. To achieve this, a specific keyword, odrichdog, was used in the prompts—a term on which the model had been trained. Furthermore, the description of her appearance was intentionally not overly detailed: the primary goal was to test which of Odrich’s traits the model had absorbed directly from the training dataset.

The model was most successful in conveying the overall combination of characteristic features: coat color, body proportions, head shape, and the general impression of Odrich’s appearance. Because of this, the character remains recognizable, despite the fact that her pose, lighting, and surrounding space change each time. Concurrently, the resemblance is not perfectly precise. In isolated generations, facial details, fur texture, or body proportions vary. This becomes a defining characteristic of the series: the neural network does not copy one of the source photographs but rather reinterprets the information learned about the character anew with every generation.

In this project, I utilized Stable Diffusion XL Base 1.0 augmented with LoRA weights that I trained on my own photographs. The base model was responsible for generating images from text prompts, while the LoRA adapted it to the specific characteristics of the photographs. Training and generation were executed in Google Colab using an NVIDIA Tesla T4 GPU. To create the final series, I constructed text prompts, varying the subjects while maintaining the core stylistic characteristics. In addition to the trained model, I employed ChatGPT as a supplementary tool for developing and refining the prompts. The project concept, the source photographs, the selection of training materials, and the final image curation are all original to my work. Additional image enhancement techniques, such as upscaling, manual inpainting, or separate post-processing, were not applied in this described experiment. The main focus was on training the LoRA and investigating how the output shifts when the text prompts are altered.