«Van Gogh of the 21st Century”—transferring Vincent van Gogh’s painterly style to new scenes using Stable Diffusion XL
Project Concept
The goal is to fine-tune the generative neural network Stable Diffusion XL so that it learns to paint in the recognizable painterly style of Vincent van Gogh: characteristic swirling brushstrokes, thick paint texture, rich color, and expressive manner.
Concept: By training a LoRA adapter on just 35 authentic Vincent van Gogh paintings (which are in the public domain), the adapter will «learn» his style. This style can then be applied to contemporary and arbitrary subjects—subjects that the artist could never have depicted, such as a city street, a portrait, a sports car, or a still life. This tests the core question: can the model separate style from content and transfer van Gogh’s technique to any new scene?
Why this specific choice: Van Gogh’s style is visually highly expressive and recognizable, and his works are in the public domain, which completely meets the dataset requirements (allowing the use of materials under a free license).
Source images for training (examples):
Resulting Image Series
Series concepts (generated from text prompts using the trigger in vanogh style):
- Cat in Van Gogh style;
- City street in Van Gogh style;
- Female portrait in Van Gogh style;
- Mountain landscape in Van Gogh style;
- Still life with fruit in Van Gogh style;
- Sports car in Van Gogh style
What is presented and how does it connect to the idea?
The final series showcases diverse contemporary and neutral subjects rendered in a unified painterly style. This directly reflects the concept: the model does not copy specific paintings, but rather applies the learned style to new content.
What stylistic elements were conveyed?
The model confidently reproduced the key characteristics of Van Gogh’s style: - characteristic swirling, curled brushstrokes and dense vertical background hatching; - thick impasto texture—the sensation of relief in the oil paint; - a recognizable palette: emerald green, chrome yellow, cobalt blue; - an expressive, swirling sky in the landscape—a direct parallel to «Starry Night.»
There are references to specific works: the cat is resolved as in his self-portraits (the figure in the «suit» against a swirling background), the street scene resembles «Night Terrace of a Cafe», and the female portrait features a green background with white flowers, similar to «Blooming Almond».
Alignment with the original concept
The main hypothesis was confirmed: the model successfully disentangled style from content and applied the painterly manner to subjects the artist could not have possibly depicted. The style rendered most convincingly on organic/natural subjects (landscape, still life with fruit, portrait)—where the brushwork and palette are maximally persuasive. On a technical object (a sports car), the style was also applied, but the subject resisted it more strongly: the composition felt less natural, although the texture and color remained «Van Gogh-esque.» All 6 frames maintain a consistent manner and palette—which means the LoRA learned the style itself, rather than merely memorizing individual paintings.
Generation Details
- base model: Stable Diffusion XL base 1.0 + VAE sdxl-vae-fp16-fix;
— method: DreamBooth + LoRA (rank 16), style trigger token — vangogh;
— captions for training images generated by the BLIP model and prefixed with in vangogh style,
(a technique from the course template—linking style to token);
— optimizations for free T4: 8-bit Adam, gradient checkpointing, fp16, SNR-gamma = 5;
— training: [500 / 800] steps. Generation: DPM-sampler, 30 steps, guidance 7.0
Visual Analysis (How the frames differ)
- **Strongest style transfer: ** Landscape (swirling sky, impasto) and portrait of a woman (floral background, planar decorative quality)—almost indistinguishable from Van Gogh’s late style;
— **Strong: ** Cat and street scene—recognizable composition and palette, good texture;
— **Weaker: ** Sports car—the style is applied, but the object’s geometry «argues» with the painterly manner;
— **Pattern: ** The more organic the subject, the more naturally the style settles; clearly man-made objects require more training steps or a lower lora_scale;
— There are virtually no artifacts (distorted faces/hands) in the series; the palette is stable across all shots
Curriculum and Learning Process
How the education was conducted:
1. Dataset: 35 Van Gogh images (public domain) resized to 1024×1024.
2. Auto-captions generated by BLIP → metadata.jsonl (using the prompt column, prefixed with «in vanogh style, „).
3. Training LoRA on SDXL using the script train_dreambooth_lora_sdxl.py from diffusers (~[30–60] min, T4).
4. Inference: LoRA weights are loaded onto the base SDXL model, generating a series based on the style prompt set.
5. Tuning: [If changed—number of steps, lora_scale when fuse_lora]
Generative Model Application Description
The project utilized generative neural networks: - Stable Diffusion XL base 1.0 (https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0), fine-tuned using the DreamBooth-LoRA method on Van Gogh’s paintings and applied to generate the final series; - BLIP (Salesforce)—for automatically captioning training images.
Purpose of Application: To train the model in a painterly style and generate a final series of images
