Helen Frankenthaler Through the Eyes of Artificial Intelligence

Creating an AI model to generate paintings in the style of the renowned Expressionist Helen Frankenthaler.

Frankenthaler’s abstractions seem alive with an invisible energy. What was new and truly remarkable was her application of an oil medium—which she thinned to the consistency of watercolor—allowing it to soak into and color the canvas rather than pooling on its surface. She used turpentine to thin the oil paint. After achieving the desired consistency, she would gently pour it onto the canvas. This thinning method had a softening effect on the colors. Furthermore, since her canvases were not sealed with gesso, they absorbed the colors, creating unusual hues. On a more theoretical level, Helen’s technique represented an important step for the modernist project as a whole. The theme of modernism is the tension between the inherent flatness of the canvas and the illusion of depth in painting. The flatness of the image was broken down by subsequent, increasingly abstract gestures that readily acknowledged their own flatness. Frankenthaler described her method of beginning a painting as, «starting with something, and then seeing what emerges throughout the continuous act of painting…».

This is common practice among contemporary artists, especially abstract expressionists. It is a highly instinctive way of working, differing from the traditional method of producing a series of preparatory sketches and compositional studies for paintings. This more spontaneous modern approach allows for unexpected directions, fortunate accidents, and self-discovery in art—a more «existential» mindset that questions biases and dogma, emphasizing the importance of individual intent over collective conformity. This spontaneity does not imply a lack of substance. However, it is often difficult for people to articulate why a particular abstract work appears harmonious and beautiful.

To train a neural network on an artist’s work with the goal of creating an image generation tool in a specific style. Such a neural network can be useful for understanding how much and which of an artist’s works are required to train the network to the point where it can generate voluminous, colored images.

Technically, the instructor’s code was used—specifically, a program designed to fine-tune the Stable Diffusion XL (SDXL) model using DreamBooth and LoRA (Low-Rank Adaptation) methods on a free Google Colab notebook equipped with a T4 GPU. DreamBooth is a method for personalizing diffusion models by training them on new concepts (such as a specific artistic style) using a small set of images. Instead of fine-tuning all the weights of the massive SDXL model, LoRA freezes the pre-trained weights and introduces trainable low-rank matrices into the model’s layers. This reduces memory usage and computational costs while preserving the model’s original knowledge. The model processes the input images and captions to generate predictions. The difference between the predicted and actual outputs (based on the captions) is calculated using a loss function (e.g., SNR weighted loss).

Gradients are calculated, and LoRA matrices are updated using an 8-bit Adam optimizer. Gradients are accumulated across several batches to simulate a larger batch size, which increases stability. The goal is to teach the model a specific artistic style—in this case, Helen Frankenthaler’s abstract expressionist style—by training it on a carefully curated dataset of her paintings. The instructor’s code was slightly modified: a small script was written to extract images of Helen Frankenthaler’s paintings from the specified URL, filter them for square proportions, and save them locally:

BLIP services were then utilized to generate captions for the artwork. This model learns to associate a unique identifier («Helen Frankenthaler») with the style depicted in the training images. During the process, the necessary libraries—bitsandbytes, transformers, accelerate, peft, and the latest diffusers from the Hugging Face repository—are installed in a Google Colab notebook, and the training script train_dreambooth_lora_sdxl.py is loaded from the Hugging Face GitHub repository. LoRA is applied to the model for efficient fine-tuning by updating only low-rank matrices, which significantly reduces the number of trainable parameters. The training configuration is set up to use the 8-bit Adam optimizer and mixed-precision training (fp16) to minimize memory usage; gradient checkpointing and gradient accumulation are also employed to handle the large model size, and the learning rate is set to 1e-4 with a constant scheduler and no warm-up steps. Training proceeds for 500 steps, with checkpoints saved every 250 steps. The instance prompt is set to «Helen Frankenthaler» to link the style to her name.

The script launches the training process using an acceleration library that optimizes distributed training. The model is trained on a curated dataset with custom captions, learning to associate artistic styles with the provided prompts. The trained LoRA weights are saved locally and uploaded to the Hugging Face Hub for sharing and reuse. A fine-tuned model is then loaded, and image samples are generated using prompts that describe the desired style (e.g., «Helen Frankenthaler, abstract color field painting…»). The trained model was saved at: https://huggingface.co/Katrun/Frankenthaler_style_sd1_LoRA

In the final series of AI-generated images, the AI was prompted to render paintings in the style of Helen Frankenthaler. In the first three cases, the AI was asked to generate images based on a description of the colors, shapes, and placement of various elements in an abstract painting:

In the subsequent three instances, the AI was prompted to generate real-world objects, such as mountains, the sea, a man, and a woman. We were interested to see how the AI would translate these real-world subjects through abstract art. The resulting series of images is as follows:

To improve presentation, several experiments were conducted with prompts—for example, by emphasizing specific colors, noting particular feelings evoked by the artwork, or highlighting specific techniques. You can iterate by starting with one of the more general prompts, reviewing the result, and then adding or modifying the prompt to move closer to the desired outcome. For instance, the initial prompt used was:

The resulting image looks like this (Fig. 7):

If we add the paint application technique and surface texture, we get the following image (Fig. 8):

If we focus on happy emotions, the following image results (Fig. 9):