Project Concept
In the fourth module of my second year, I worked on an animated longread, for which I needed to create a series of illustrations in a cohesive style. All the images were hand-drawn, and this process took a significant amount of time. Looking back, I realize I would like to find a way to speed up this work while preserving the character and individuality of the drawings.
Therefore, I decided to return to my illustrations and use them as a dataset to train the generative neural network Stable Diffusion.
Project Goal
To train a neural network to replicate the characteristics of my graphic style: uneven lines, simplified forms, and deliberate carelessness. Crucially, the model must be capable of generating images not only on marine themes—which were the subject of the initial sketches—but also on entirely different subjects.
Dataset
Original illustrations were used; no additional rights clearances from copyright holders were required.
It consists of 26 original drawings: separate objects, diagrams, and composite illustrations. The images are prepared in a 1:1 square format, and the source files are sized at 1500×1500 pixels.
Graphic characteristics: uneven black outline, simple recognizable shapes, absence of realistic chiaroscuro, a small degree of intentional inaccuracy, and a white background.
Coding and Learning Stages
- A ZIP archive containing 26 original illustrations is loaded into Colab. The code checks the file format and prepares the images for the neural network’s subsequent training.
- Each image is manually assigned a short description in English that identifies the subject matter. This helps the model differentiate the content of the artwork and link it to the author’s unique style.
- Training utilizes Stable Diffusion XL with the DreamBooth LoRA method. During the process, LoRA parameters are modified while the base model remains untouched by full retraining.
- After training, I test the model on six subjects absent from the original dataset. An additional negative prompt constrains the level of detail, realism, shadows, and background elements.
To evaluate the learning outcomes, three versions of each image are created: the original SDXL output, a generation with the trained LoRA, and an additionally processed version. Threshold processing converts the image to black and white, removing the gray background. This allows for a comparison of the training results and an assessment of whether subsequent processing is necessary.
Model evaluation after training
In test prompts unrelated to maritime themes, standard SDXL was compared against the model with an attached LoRA. After 20 steps, the images remained overly detailed and resembled typical sketches. After 400 steps, simplified silhouettes and uneven black outlines became more noticeable. However, the model did not always adhere to the specified object count or the requirement for a completely white background.
The first attempt at the final series proved less successful; there were areas with gray backgrounds, realistic textures, and excessive hatching. Further refinement of prompts and verification of the correct loading of the final LoRA weights improved the resemblance to the original graphics.
Final Generation Task
To test the transference of the author’s style, I chose a new context—a hike in the forest. The series of eight images features camping gear, camp details, and forest objects. This intentionally diverges from the marine theme of the original dataset, allowing me to test whether the model transfers the manner of painting rather than simply memorizing familiar objects.
The base SDXL 1.0 is used, connected via trained weights using pipe.load_lora_weights (…). The special phrase «in lzsketch style» activates the trained style. Generation was performed at a resolution of 768×768 px, 28 steps, and a guidance scale of 6.0. Different seeds and LoRA strengths were tested; the final selected variations were achieved with weights of 0.9 or 1.0.
The negative prompt excludes gray backgrounds, shadows, cross-hatching, photorealism, complex compositions, and extraneous objects. For a strictly white background, thresholded black-and-white copies were saved separately. Since this operation could alter or remove delicate gray lines, the unprocessed originals have been preserved separately.
Results
Washed:
1) One small triangular camping tent, featuring a simple open front flap—just one tent.
2) One hiking backpack with two shoulder straps, one front pocket, and a simple top flap.
3) One small campfire, composed of three crossed wooden logs and large, simple flames—no smoke.
4) Two separate, bare trees supporting one empty small hammock strung between them.
5) A small cluster of exactly three forest mushrooms, each with a different cap shape and height.
6) One small, solitary deciduous tree with a slightly wobbly trunk and one large, irregular rounded leafy crown, rendered without individual leaves.
7) One friendly, chunky bear standing on all four legs, shown in side profile, with small rounded ears.
8) One simple, old-fashioned campfire cooking pot with a slightly curved body, a flat lid, and one arched carrying handle—rendered as a single object.
Introduction
The neural network could not fully replace hand-drawn illustration, but it has become a tool capable of accelerating the creation of illustrations within a consistent style.
As a result of the experiment, I successfully trained Stable Diffusion to reproduce the core characteristics of an illustrative style and apply them to entirely new narratives. Although the initial dataset primarily consisted of maritime-themed images, the neural network managed to create a series of illustrations for a hiking expedition, preserving the uneven contours, simplified forms, and characteristic looseness of the drawings.
However, the outcome was not perfect; the model did not always adhere precisely to the text prompts, frequently adding extraneous details, complicating compositions, and altering the character of the lines. Achieving the desired results required additional experimentation with prompts, generation parameters, manual image selection, and background refinement.
Returning to the original project idea, I can say that such a tool could truly ease my work on the animated longform piece for my second year.
Description of Generative Model Application
1. Stable Diffusion XL 1.0 — image generation
2. DreamBooth LoRA (diffusers library) — fine-tuning on original artwork.
3. ChatGPT (OpenAI, GPT-6) was utilized for adapting the Colab code, debugging, error correction, and prompt composition. The selection of the initial dataset and final illustrations was performed by the project author.
