Project Idea
I wanted to investigate whether a generative neural network could learn to reproduce a specific graphic style and transfer it to new objects and narratives. To do this, I trained Stable Diffusion on my own collection of 22 images united by a characteristic black-and-white aesthetic: coarse rasterization, high-contrast spots, imitation of screen printing, and a distressed surface.
Image Preparation
A collection of 22 images was curated for the training. It was important to prepare them in such a way that the neural network could identify recurring graphical characteristics rather than focusing solely on the content of individual images.Each image received a textual description. A special trigger word, mystyleicon, was used to denote the style being studied. This word was added to the prompts during generation, helping the network reference the learned visual concept.Thus, the data preparation became a distinct part of the research—the ability of the neural network to assimilate specific style traits depended on the quality and consistency of the source images.
The Learning Process
For the experiment, Stable Diffusion 1.5—a generative model that creates images based on text prompts—was selected. The training was conducted on Kaggle using a Tesla T4 GPU.To adapt the model to a specific visual style, the LoRA (Low-Rank Adaptation) method was employed. This allows training an auxiliary set of parameters responsible for the image’s characteristics without overfitting the base model’s parameters.The training involved several stages:
- Preparation of images and textual descriptions.
- Setting up the environment and loading the base model.
- Training the LoRA on the prepared dataset.
- Saving the resulting weights.
- Integrating the trained LoRA with Stable Diffusion and testing generations using text prompts.The result was a separate LoRA weight file that can be connected to the base model for further experimentation.
1. Image and text description preparation
During the preparation phase, a training dataset of 22 images was assembled, unified by a common visual style. For each image, a separate text file was created describing its content and the characteristic features of the style. The trigger word mystyleicon was used in the descriptions to activate the studied style during image generation. This prepared dataset served as the foundation for subsequently training a LoRA model based on Stable Diffusion 1.5.
2. Environment Setup and Base Model Loading
During the environment setup phase, a development environment was prepared in a Kaggle Notebook to train a LoRA model. The NVIDIA Tesla T4 GPU accelerator and the PyTorch library were used for computation. The foundation of the project was the pre-trained Stable Diffusion 1.5 model, which is designed for generating images from text descriptions. Using the base model allowed us to adapt it to the specified visual style through additional LoRA fine-tuning.
3. Training LoRA on the prepared sample set
To train the model, I utilized the base Stable Diffusion 1.5 model along with a prepared dataset of 22 images, each accompanied by a text description. Before starting the training, I established the key parameters: image resolution set to 512×512 pixels, batch size of 1, learning rate of 0.0001, and LoRA rank of 16.The training was configured for 800 steps. To conserve video memory, I employed gradient checkpointing and mixed precision (fp16). Additionally, I specified parameters for saving intermediate checkpoints, allowing access to the model’s saved states.Figure [insert figure reference here] illustrates the configuration of the LoRA training parameters within the Kaggle Notebook.
After configuring the parameters, I initiated the LoRA training process. The training ran on an NVIDIA Tesla T4 GPU and consisted of 800 steps. During execution, the script outputted information regarding the training progress. Upon completion of all steps, the model successfully saved the trained weights. This allowed me to proceed to the next stage—integrating the LoRA with the base Stable Diffusion 1.5 model and testing image generation.
4. Saving the derived weights
Upon completion of training, the resulting LoRA weights were saved to a separate file, pytorch_lora_weights.safetensors. This file contains the parameters derived from adapting Stable Diffusion 1.5 to the visual style of my dataset.Saving the weights allows the trained LoRA to be re-linked to the base model without needing to repeat the entire training process. This enables the generated result to be utilized for further image generation experiments.Figure: pytorch_lora_weights.safetensors—the trained LoRA weight file.
5. Integrating a trained LoRA with Stable Diffusion and testing generations using text prompts
After completing the training, I attached the saved LoRA weights to the base Stable Diffusion 1.5 model. This allowed me to test how the resulting model reproduces the learned graphic style when generating new images.
For testing, I prepared a text prompt that included the trigger word mystyleicon, which was used during training. The prompt specified the characteristic features of the visual style: black-and-white printing, high contrast, coarse raster, weathered texture, and rough graphic forms.
The generation was performed at a resolution of 512 × 512 pixels. Thirty-five generation steps and a guidance scale of 7 were used to create the image. The resulting output was the first test of the trained model after applying the LoRA.
Conclusion
The project successfully adapted Stable Diffusion to the chosen graphic aesthetic. Generation confirmed that the trained LoRA can be applied to various subjects while preserving the characteristic visual traits of the style.
