Wallpaper for a panel building
Twelve seamless patterns using the Japanese katagami stencil technique. Instead of cranes and chrysanthemums, these patterns feature crows on wires, cast-iron radiators, satellite dishes, and cats on windowsills. The patterns were rendered by Stable Diffusion XL: it was trained on 188 fragments of 19th-century museum stencils.
Where it all began
The Metropolitan Museum of Art’s collection holds nearly half a thousand Japanese fabric stencils, known as katagami. These are several layers of paper made from mulberry fiber, glued with the juice of unripe persimmons. This process hardens the paper, giving it a brown hue that is impervious to water. The artisan carves a pattern into it with a small knife, sometimes creating thousands of holes on a single sheet. The stencil is then placed onto the fabric, rice paste is applied through the perforations, and after dyeing, a white design remains exactly where the paste was applied.
Two things capture my interest in katagami. First: the pattern repeat. The stencil is slid across the fabric cut piece step by step, and the design must meet seamlessly at the joint without a visible seam. Second: the motifs. This isn’t courtly painting; it’s what the artist sees every day. Waves, bamboo, ivy, dragonflies, birds on a branch, fans, fishing nets, rain. This led to an idea. If a katagami master lived in a residential neighborhood, what would they cut? Crows on power lines instead of cranes. Radiator grates instead of bamboo screens. Rows of balconies with laundry instead of waves. Dandelions pushing through asphalt instead of chrysanthemums. And because of the pattern repeat, it can be printed on wallpaper, fabric, and tile. This resulted in a small collection for a typical apartment.
Where do the stencils come from?
All stencils are sourced from the Metropolitan Museum’s Open Access collection. The museum releases photographs of items in the public domain under the CC0 license—meaning they can be used freely, including for training models.
How they were selected:
The museum’s «stencil» API query returned 970 items;
from these, Japanese stencils in the public domain with a photograph were selected;
descriptions could be read for 693 items (the museum’s server limits query frequency), of which 498 were suitable;
previews were downloaded for 384, which is more than sufficient;
106 stencils were manually chosen from the contact lists.
Items excluded from the selection: Stencil patterns with lettering and hieroglyphs in the design; Very small patterns, resembling gray fabric, which cannot be discerned in the photos; Figurative images: landscape scenes with boats, large birds covering the entire sheet; Stencils photographed against a black background: the background changes the meaning, and the black becomes the cutout.
Moments from the model’s studies, with captions
Fragments. The stencil was photographed in its entirety: a white background, wide brown borders around the design, with labels and numbers on the borders. The model cannot be trained on this image, otherwise, it will start drawing the labels. Therefore, the script automatically finds the design area in two steps: 1. It separates the sheet from the white background; 2. Within the sheet, it separates the design from the borders: the white background shows through the cutouts in the design, but not in the borders. Two squares are cut from the elongated stencil—one from an almost square shape. Each square is scaled down to 1024×1024. A total of 188 fragments.
Captions. Manually labeling every fragment is time-consuming, so the captions were generated automatically. CLIP compares each fragment to a vocabulary of 32 traditional Katagami motifs (waves, bamboo, chrysanthemums, ivy, dragonflies, fans, diamonds, grids, etc.) and selects one or two of the most similar matches. The caption appears as follows:
TOK katagami stencil, pattern of wisteria and small birds
The generated captions are not perfectly accurate. CLIP frequently mistakes thin parallel lines in any pattern for «bamboo»: bamboo appeared in 64 out of 188 captions, even though it is visibly less frequent in the actual sample. This is not a critical issue for this method. The caption is necessary so that the model can separate the motif from the style, and the style ultimately gets absorbed into new tokens anyway.
How the model learned to cut stencils
The model here is Stable Diffusion XL 1.0. The method is called pivotal tuning, which is a combination of two techniques.
A new word. Two new tokens, and, are added to the model’s vocabulary. Initially, they have no meaning. During training, the model assigns meaning to them—a meaning such that the phrase katagami stencil results in precisely this paper, this knife cut, this brown tint. This is textual inversion.
LoRA. Small trainable matrices are simultaneously added to the attention layers of the UNet, as is customary.
Why both at once? SDXL knows the word «katagami, ” but it understands it as „something Japanese“ and renders either an engraving or a kimono. New tokens give style its name, and LoRA refines the rendering to the desired accuracy. In the first half of training, both tokens and LoRA learn. Afterward, the tokens are frozen (this is the „pivot“), and only LoRA continues to learn. This allows the LoRA to adapt to the already established word instead of overriding the style entirely.
The script is official, taken from the diffusers examples: train_dreambooth_lora_sdxl_advanced.py. There are two fixes in it:
the original script deleted the tokens with a single file during every intermediate save, but the fix saves them alongside the weights for each stage;
after training, the original script converted the entire UNet to fp32 directly on the GPU, which caused out-of-memory errors on a T4 after all 1200 steps. The fix moves the UNet into RAM.
Reflection is included here, unlike photographs: the pattern has no right or left side.
Prodigy instead of the usual AdamW: for pivotal tuning, this is the standard choice. Prodigy itself assesses the appropriate learning rate, so a single setting (rate 1.0) works immediately for both tokens and LoRA. Furthermore, they have vastly different scales: a token is one vector, while LoRA involves millions of numbers.
The loss function is fluctuating around 0.1 throughout the process; there is no noticeable drop. This is typical when fine-tuning a diffusion model: the error primarily depends on the random noise level at each step. The dashed line on the graph is at step 600, exactly in the middle. In reality, the tokens freeze slightly earlier, at step 564: the script calculates the proportion based on full passes over the sample.
The Stitch That Isn’t There
The wallpaper, fabric, and tile are printed with a repeat: one square tiles seamlessly in all directions. A standard diffusion model draws an image with edges, and when tiled, a seam becomes visible. The technique to fix this is very simple. Each convolutional layer in the UNet and VAE has padding around the edges: to process the boundary pixel, the layer takes neighbors that do not exist beyond the edge and substitutes them with zeros. If you replace the zeros with a circular padding (padding_mode="circular"), the layer will take pixels from the opposite edge instead of emptiness. For the model, the image becomes a torus: the right edge continues the left, and the bottom continues the top. The model draws a pattern that closes on itself.
The number under the images indicates how many times greater the color jump at the seam is compared to the normal jump between adjacent pixels within the pattern. With a standard offset of 32.5, the seam is immediately visible. With a circular offset of 1.1, the seam is almost indistinguishable from any other area.
Three Attempts
Attempt 1: Everything turned into chrysanthemums
The first collection was generated on the final weigh-in, after 1,200 steps. The texture turned out excellent: brown oiled paper, white cutouts, a slightly uneven knife. But the motifs disappeared. The radiators became clouds, satellite dishes became flowers, and scooters became leaves. The model had learned the catalogs too well—it responded to any word with what it had seen in the museum.
