Generation of realistic children’s book images based on diffusion models

Abstract

This paper describes a model for generating realistic images for children’s books using diffusion models and explains each step of the proposed model. The focus is on reviewing the existing area of research on controlled diffusion models. Specifically, ControlNet, a neural network model for controlling stable diffusion models, is used to add controls to guide image generation. Edges (Canny, Sobel, and Prewitt) were used as controls. Additionally, a bank of images extracted from children’s books was designed to generate new versions of books with realistic images. Tests were conducted using the books "The Little Prince" and "The Seagull and the Penguin." Results highlight that ControlNet is a potent tool for generating visually attractive, diverse, and high-quality images, affirming its relevance for illustrators and developers working on children’s books. Using edges as control improves the detail level in the generated images.