Building Smarter Visual Recommendations with Gemini Multimodal Embeddings

Building a recommendation system that truly understands aesthetics is challenging. While Convolutional Neural Networks (CNNs) capture basic visual features like edges and colors, they often miss the big picture.

As illustrated in a , ResNet50 struggled with semantic relevance. To mitigate this issue, I had to implement an additional filtering layer using Filtered k-NN in Elasticsearch.

In another story, I experimented with SigLIP for embedding extraction, and it helped produce better recommendations without needing to apply Filtered k-NN.

The goal of this story is to experiment with an API-based embedding model and evaluate its results against the ResNet and SigLIP models used previously. Concretely, this story leverages the Gemini Embedding API through the Google AI Studio and the GenAI Python SDK.

Below are the other stories related to building a visual recommendation system.

1. Building a Basic Image-Based Recommendation System,
2. Building an…