Processing Visual Content Using Spatial scene Techniques for LLM/RAG in Generative AI
Open Access DepositedThis study explores the application of Retrieval-Augmented Generation (RAG) and image preprocessing methods to enhance “image-to-image generation” consistency using Spatial-Scene Techniques, including Image Segmentation, Segmented Depth information, and “3D Positional Encoding”. The image segmentation model used is a pretrained model utilized as a frozen layer model. The Segmented Depth information involves a depth estimation model to retrieve the average depth values of each segment and involves a custom method of clustering neighboring segments together, resulting in the formation of different scene objects for future RAG indexing and querying processes.The study investigates the impact of different initial input image pre-processing, including White-Image, Black-Image, Gaussian Noise, Simple-RAG-Queried, Dark-Pixel-Replacement, Pixel-Averaging, and Patches Fixing on image generation performance with an image-to-image model. This study makes use of the MSCOCO dataset and evaluates performance using Fréchet Inception Distance (FID) and Contrastive Language-Image Pre-Training (CLIP) metrics. The results show that RAG similarity search with Spatial Scene Techniques improves FID and CLIP scores on the image generation tasks. Additionally, the results show that Patch Fixing from previously generated output shows promising results in certain scenarios. On the other hand, the results show that increasing the sample size in document indexing may degrade FID scores. Specifically, the Simple-RAG-Queried method achieves FID scores of 1.88 and CLIP scores of 0.30, outperforming other state-of-the-art methods using a similar MSCOCO dataset listed in Chapter 4 in demonstrating the improvement of image generation with RAG and Spatial Scene Techniques in “image-to-image generation” systems.
- All rights reserved
Notice to Authors
If you are the author of this work and you have any questions about the information on this page, please use the Contact form to get in touch with us.