使用GPT-4o mini为图像添加标签和描述：多模态AI实践指南

技术背景

在电商、推荐系统和内容管理中，图像数据往往缺乏结构化文本信息，导致搜索和检索困难。传统方法依赖人工标注，成本高且难以扩展。多模态大模型（如GPT-4o mini）能够同时理解图像和文本，自动生成标签和描述，从而将非结构化图像转化为可搜索的文本表示。本教程解决的核心问题是如何利用这些模型高效地为图像生成高质量标签和标题，并基于嵌入实现语义搜索。适用场景包括电商产品搜索、图像分类、内容推荐、RAG应用等。主要限制包括API成本、处理速度、模型对图像的理解偏差，以及需要处理生成文本的准确性和一致性。

核心原理

核心原理是利用GPT-4o mini的视觉-语言联合理解能力。模型接收图像和文本上下文（如产品标题），通过提示工程引导其输出结构化标签或描述。标签提取采用零样本方法，模型根据系统提示中的规则（如物品类型、材质、风格、颜色）生成关键词。为避免同义词重复，使用嵌入模型（如text-embedding-3-large）将关键词向量化，并通过余弦相似度与已有关键词比较，超过阈值则替换为已有词。描述生成使用较低温度（0.2）以保持稳定性，标题生成则采用少样本示例，让模型学习从描述到简洁标题的映射。搜索阶段，将关键词和标题拼接后嵌入，与用户查询（文本或图像生成的标题）的嵌入进行余弦相似度比较，返回最相似结果。数据流为：图像+标题 -> 标签/描述 -> 标题 -> 嵌入 -> 相似度检索。

架构分析

系统架构分为四个模块：1) 数据准备层：加载数据集（CSV），包含图像URL、标题等字段。2) 图像理解层：调用GPT-4o mini API，分别实现标签提取（analyze_image）和描述生成（describe_image），均使用系统提示和用户消息（图像+标题）。3) 后处理层：标签去重（基于嵌入和余弦相似度），标题生成（caption_image）使用少样本示例。4) 搜索层：将关键词和标题拼接后嵌入，存储为DataFrame，查询时计算相似度。设计决策包括：使用零样本标签提取以快速启动，但通过嵌入去重提高质量；描述生成采用较低温度保证一致性；标题生成使用少样本而非微调，以平衡成本和效果；搜索结合关键词和标题，提高检索准确性。可修改点包括：更换嵌入模型、调整相似度阈值、增加微调步骤、优化提示词。

完整实战教程

本笔记本探讨如何利用GPT-4*模型（例如gpt-4o、gpt-4o-mini或gpt-4-turbo）的视觉能力为图像添加标签和描述。

我们可以利用这些模型的多模态能力，输入图像以及关于图像所代表内容的额外上下文，并提示模型输出标签或图像描述。然后，可以使用语言模型（在本笔记本中，我们将使用gpt-4o-mini）进一步优化图像描述以生成标题。

从图像生成文本内容可用于多种用例，尤其是涉及搜索的用例。

我们将在本笔记本中通过使用生成的关键词和产品描述来搜索产品（无论是从文本输入还是图像输入）来说明搜索用例。

作为示例，我们将使用亚马逊家具数据集，为其添加相关关键词标签，并生成简短、描述性的标题。

设置

# Install dependencies if needed
%pip install openai
%pip install scikit-learn

from IPython.display import Image, display
import pandas as pd
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np
from openai import OpenAI

# Initializing OpenAI client - see https://platform.openai.com/docs/quickstart?context=python
client = OpenAI()

# Loading dataset
dataset_path =  "data/amazon_furniture_dataset.csv"
df = pd.read_csv(dataset_path)
df.head()

图像打标签

在本节中，我们将使用 GPT-4o mini 为我们的产品生成相关标签。

我们将采用简单的零样本方法来提取关键词，并使用嵌入对这些关键词进行去重，以避免出现多个过于相似的关键词。

我们将结合图像和产品标题，以避免提取图像中描绘的其他物品的关键词——有时场景中会出现多个物品，而我们只想专注于我们想要标记的那个。

提取关键词

system_prompt = '''
    You are an agent specialized in tagging images of furniture items, decorative items, or furnishings with relevant keywords that could be used to search for these items on a marketplace.
    
    You will be provided with an image and the title of the item that is depicted in the image, and your goal is to extract keywords for only the item specified. 
    
    Keywords should be concise and in lower case. 
    
    Keywords can describe things like:
    - Item type e.g. 'sofa bed', 'chair', 'desk', 'plant'
    - Item material e.g. 'wood', 'metal', 'fabric'
    - Item style e.g. 'scandinavian', 'vintage', 'industrial'
    - Item color e.g. 'red', 'blue', 'white'
    
    Only deduce material, style or color keywords when it is obvious that they make the item depicted in the image stand out.

    Return keywords in the format of an array of strings, like this:
    ['desk', 'industrial', 'metal']
    
'''

def analyze_image(img_url, title):
    response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {
            "role": "system",
            "content": system_prompt
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": img_url,
                    }
                },
            ],
        },
        {
            "role": "user",
            "content": title
        }
    ],
        max_tokens=300,
        top_p=0.1
    )

    return response.choices[0].message.content

用几个示例进行测试

examples = df.iloc[:5]

for index, ex in examples.iterrows():
    url = ex['primary_image']
    img = Image(url=url)
    display(img)
    result = analyze_image(url, ex['title'])
    print(result)
    print("\n\n")

查找现有关键词

利用嵌入来避免重复（同义词）和/或匹配预定义的关键词

# Feel free to change the embedding model here
def get_embedding(value, model="text-embedding-3-large"): 
    embeddings = client.embeddings.create(
      model=model,
      input=value,
      encoding_format="float"
    )
    return embeddings.data[0].embedding

使用示例关键词进行测试

# Existing keywords
keywords_list = ['industrial', 'metal', 'wood', 'vintage', 'bed']

df_keywords = pd.DataFrame(keywords_list, columns=['keyword'])
df_keywords['embedding'] = df_keywords['keyword'].apply(lambda x: get_embedding(x))
df_keywords

def compare_keyword(keyword):
    embedded_value = get_embedding(keyword)
    df_keywords['similarity'] = df_keywords['embedding'].apply(lambda x: cosine_similarity(np.array(x).reshape(1,-1), np.array(embedded_value).reshape(1, -1)))
    most_similar = df_keywords.sort_values('similarity', ascending=False).iloc[0]
    return most_similar

def replace_keyword(keyword, threshold = 0.6):
    most_similar = compare_keyword(keyword)
    if most_similar['similarity'] > threshold:
        print(f"Replacing '{keyword}' with existing keyword: '{most_similar['keyword']}'")
        return most_similar['keyword']
    return keyword

# Example keywords to compare to our list of existing keywords
example_keywords = ['bed frame', 'wooden', 'vintage', 'old school', 'desk', 'table', 'old', 'metal', 'metallic', 'woody']
final_keywords = []

for k in example_keywords:
    final_keywords.append(replace_keyword(k))
    
final_keywords = set(final_keywords)
print(f"Final keywords: {final_keywords}")

生成标题

在本节中，我们将使用 GPT-4o mini 生成图像描述，然后使用 GPT-4-turbo 的少样本示例方法从图像生成标题。

如果少样本示例不足以满足您的用例，请考虑微调模型，以使生成的标题符合您所针对的风格和语气。

# Cleaning up dataset columns
selected_columns = ['title', 'primary_image', 'style', 'material', 'color', 'url']
df = df[selected_columns].copy()
df.head()

使用 GPT-4o mini 描述图像

describe_system_prompt = '''
    You are a system generating descriptions for furniture items, decorative items, or furnishings on an e-commerce website.
    Provided with an image and a title, you will describe the main item that you see in the image, giving details but staying concise.
    You can describe unambiguously what the item is and its material, color, and style if clearly identifiable.
    If there are multiple items depicted, refer to the title to understand which item you should describe.
    '''

def describe_image(img_url, title):
    response = client.chat.completions.create(
    model="gpt-4o-mini",
    temperature=0.2,
    messages=[
        {
            "role": "system",
            "content": describe_system_prompt
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": img_url,
                    }
                },
            ],
        },
        {
            "role": "user",
            "content": title
        }
    ],
    max_tokens=300,
    )

    return response.choices[0].message.content

在几个示例上进行测试

for index, row in examples.iterrows():
    print(f"{row['title'][:50]}{'...' if len(row['title']) > 50 else ''} - {row['url']} :\n")
    img_description = describe_image(row['primary_image'], row['title'])
    print(f"{img_description}\n--------------------------\n")

将描述转化为标题

使用少样本示例方法，将长描述转化为简短的图像标题

caption_system_prompt = '''
Your goal is to generate short, descriptive captions for images of furniture items, decorative items, or furnishings based on an image description.
You will be provided with a description of an item image and you will output a caption that captures the most important information about the item.
Your generated caption should be short (1 sentence), and include the most relevant information about the item.
The most important information could be: the type of the item, the style (if mentioned), the material if especially relevant and any distinctive features.
'''

few_shot_examples = [
    {
        "description": "This is a multi-layer metal shoe rack featuring a free-standing design. It has a clean, white finish that gives it a modern and versatile look, suitable for various home decors. The rack includes several horizontal shelves dedicated to organizing shoes, providing ample space for multiple pairs. Above the shoe storage area, there are 8 double hooks arranged in two rows, offering additional functionality for hanging items such as hats, scarves, or bags. The overall structure is sleek and space-saving, making it an ideal choice for placement in living rooms, bathrooms, hallways, or entryways where efficient use of space is essential.",
        "caption": "White metal free-standing shoe rack"
    },
    {
        "description": "The image shows a set of two dining chairs in black. These chairs are upholstered in a leather-like material, giving them a sleek and sophisticated appearance. The design features straight lines with a slight curve at the top of the high backrest, which adds a touch of elegance. The chairs have a simple, vertical stitching detail on the backrest, providing a subtle decorative element. The legs are also black, creating a uniform look that would complement a contemporary dining room setting. The chairs appear to be designed for comfort and style, suitable for both casual and formal dining environments.",
        "caption": "Set of 2 modern black leather dining chairs"
    },
    {
        "description": "This is a square plant repotting mat designed for indoor gardening tasks such as transplanting and changing soil for plants. It measures 26.8 inches by 26.8 inches and is made from a waterproof material, which appears to be a durable, easy-to-clean fabric in a vibrant green color. The edges of the mat are raised with integrated corner loops, likely to keep soil and water contained during gardening activities. The mat is foldable, enhancing its portability, and can be used as a protective surface for various gardening projects, including working with succulents. It's a practical accessory for garden enthusiasts and makes for a thoughtful gift for those who enjoy indoor plant care.",
        "caption": "Waterproof square plant repotting mat"
    }
]

formatted_examples = [[{
    "role": "user",
    "content": ex['description']
},
{
    "role": "assistant", 
    "content": ex['caption']
}]
    for ex in few_shot_examples
]

formatted_examples = [i for ex in formatted_examples for i in ex]

def caption_image(description, model="gpt-4o-mini"):
    messages = formatted_examples
    messages.insert(0, 
        {
            "role": "system",
            "content": caption_system_prompt
        })
    messages.append(
        {
            "role": "user",
            "content": description
        })
    response = client.chat.completions.create(
    model=model,
    temperature=0.2,
    messages=messages
    )

    return response.choices[0].message.content

在几个示例上进行测试

examples = df.iloc[5:8]

for index, row in examples.iterrows():
    print(f"{row['title'][:50]}{'...' if len(row['title']) > 50 else ''} - {row['url']} :\n")
    img_description = describe_image(row['primary_image'], row['title'])
    print(f"{img_description}\n--------------------------\n")
    img_caption = caption_image(img_description)
    print(f"{img_caption}\n--------------------------\n")

图像搜索

在本节中，我们将使用生成的关键词和标题来搜索与给定输入（文本或图像）匹配的项目。

我们将利用嵌入模型为关键词和标题生成嵌入，并将它们与输入文本或输入图像生成的标题进行比较。

# Df we'll use to compare keywords
df_keywords = pd.DataFrame(columns=['keyword', 'embedding'])
df['keywords'] = ''
df['img_description'] = ''
df['caption'] = ''

# Function to replace a keyword with an existing keyword if it's too similar
def get_keyword(keyword, df_keywords, threshold = 0.6):
    embedded_value = get_embedding(keyword)
    df_keywords['similarity'] = df_keywords['embedding'].apply(lambda x: cosine_similarity(np.array(x).reshape(1,-1), np.array(embedded_value).reshape(1, -1)))
    sorted_keywords = df_keywords.copy().sort_values('similarity', ascending=False)
    if len(sorted_keywords) > 0 :
        most_similar = sorted_keywords.iloc[0]
        if most_similar['similarity'] > threshold:
            print(f"Replacing '{keyword}' with existing keyword: '{most_similar['keyword']}'")
            return most_similar['keyword']
    new_keyword = {
        'keyword': keyword,
        'embedding': embedded_value
    }
    df_keywords = pd.concat([df_keywords, pd.DataFrame([new_keyword])], ignore_index=True)
    return keyword

准备数据集

import ast

def tag_and_caption(row):
    keywords = analyze_image(row['primary_image'], row['title'])
    try:
        keywords = ast.literal_eval(keywords)
        mapped_keywords = [get_keyword(k, df_keywords) for k in keywords]
    except Exception as e:
        print(f"Error parsing keywords: {keywords}")
        mapped_keywords = []
    img_description = describe_image(row['primary_image'], row['title'])
    caption = caption_image(img_description)
    return {
        'keywords': mapped_keywords,
        'img_description': img_description,
        'caption': caption
    }

df.shape

处理整个数据集的312行将需要一段时间。

为了测试这个想法，我们只在前50行上运行：这大约需要20分钟。

可以随意跳过这一步，直接加载已经处理好的数据集（见下文）。

# Running on first 50 lines
for index, row in df[:50].iterrows():
    print(f"{index} - {row['title'][:50]}{'...' if len(row['title']) > 50 else ''}")
    updates = tag_and_caption(row)
    df.loc[index, updates.keys()] = updates.values()

df.head()

data_path = "data/items_tagged_and_captioned.csv"

# Saving locally for later - optional: do not execute if you prefer to use the provided file
df.to_csv(data_path, index=False)

# Optional: load data from saved file if you haven't processed the whole dataset
df = pd.read_csv(data_path)

嵌入标题和关键词

现在我们可以使用生成的标题和关键词，将相关内容与输入文本查询或标题进行匹配。

为此，我们将嵌入关键词+标题的组合。

注意：创建嵌入大约需要3分钟运行。可以随意加载预处理的数据集（见下文）。

df_search = df.copy()

def embed_tags_caption(x):
    if x['caption'] != '':
        try:
            keywords_string = ",".join(k for k in x['keywords']) + '\n'
            content = keywords_string + x['caption']
            embedding = get_embedding(content)
            return embedding
        except Exception as e:
            print(f"Error creating embedding for {x}: {e}")

df_search['embedding'] = df_search.apply(lambda x: embed_tags_caption(x), axis=1)

df_search.head()

# Keep only the lines where we have embeddings
df_search = df_search.dropna(subset=['embedding'])
print(df_search.shape)

data_embeddings_path = "data/items_tagged_and_captioned_embeddings.csv"

# Saving locally for later - optional: do not execute if you prefer to use the provided file
df_search.to_csv(data_embeddings_path, index=False)

# Optional: load data from saved file if you haven't processed the whole dataset
from ast import literal_eval
df_search = pd.read_csv(data_embeddings_path)
df_search["embedding"] = df_search.embedding.apply(literal_eval).apply(np.array)

从输入文本中搜索

我们可以将用户的输入文本直接与我们刚刚创建的嵌入进行比较。

# Searching for N most similar results
def search_from_input_text(query, n = 2):
    embedded_value = get_embedding(query)
    df_search['similarity'] = df_search['embedding'].apply(lambda x: cosine_similarity(np.array(x).reshape(1,-1), np.array(embedded_value).reshape(1, -1)))
    most_similar = df_search.sort_values('similarity', ascending=False).iloc[:n]
    return most_similar

user_inputs = ['shoe storage', 'black metal side table', 'doormat', 'step bookshelf', 'ottoman']

for i in user_inputs:
    print(f"Input: {i}\n")
    res = search_from_input_text(i)
    for index, row in res.iterrows():
        similarity_score = row['similarity']
        if isinstance(similarity_score, np.ndarray):
            similarity_score = similarity_score[0][0]
        print(f"{row['title'][:50]}{'...' if len(row['title']) > 50 else ''} ({row['url']}) - Similarity: {similarity_score:.2f}")
        img = Image(url=row['primary_image'])
        display(img)
        print("\n\n")

从图像搜索

如果输入是图像，我们可以先通过将图像转换为描述文字，再将这些描述文字嵌入并与已创建的嵌入进行比较，从而找到相似的图像。

# We'll take a mix of images: some we haven't seen and some that are already in the dataset
example_images = df.iloc[306:]['primary_image'].to_list() + df.iloc[5:10]['primary_image'].to_list()

for i in example_images:
    img_description = describe_image(i, '')
    caption = caption_image(img_description)
    img = Image(url=i)
    print('Input: \n')
    display(img)
    res = search_from_input_text(caption, 1).iloc[0]
    similarity_score = res['similarity']
    if isinstance(similarity_score, np.ndarray):
        similarity_score = similarity_score[0][0]
    print(f"{res['title'][:50]}{'...' if len(res['title']) > 50 else ''} ({res['url']}) - Similarity: {similarity_score:.2f}")
    img_res = Image(url=res['primary_image'])
    display(img_res)
    print("\n\n")

总结

在本笔记本中，我们探讨了如何利用 gpt-4o-mini 的多模态能力来为图像添加标签和生成说明文字。通过向模型提供图像以及上下文信息，我们能够生成标签和描述，这些标签和描述可以进一步细化以创建说明文字。这个过程在各种场景中都有实际应用，尤其是在增强搜索功能方面。

所展示的搜索用例可以直接应用于推荐系统等应用，但本笔记本中涵盖的技术可以扩展到物品搜索之外，并用于多种用例，例如利用非结构化图像数据的 RAG 应用。

作为下一步，您可以探索将基于规则的过滤（结合关键词）与基于嵌入的搜索（结合说明文字）相结合，以检索更相关的结果。

常见问题

常见错误：1) API调用失败：检查OpenAI客户端初始化，确保API密钥有效，网络连接正常。2) 关键词解析错误：模型可能返回非列表格式，使用ast.literal_eval时需捕获异常，并设置默认值。3) 嵌入维度不匹配：确保所有嵌入使用相同模型，且形状一致。4) 相似度计算错误：余弦相似度输入需为二维数组，使用reshape(1,-1)。5) 处理速度慢：批量处理时考虑并发或使用异步，或减少数据量。6) 生成内容不准确：调整提示词，增加示例，或使用更高温度。排查方法：打印中间结果，检查API响应格式，使用try-except捕获异常，逐步调试。生产优化：使用缓存减少重复API调用，采用批处理提高吞吐量，使用更高效的嵌入模型（如text-embedding-3-small），对模型输出进行后处理校验，考虑微调模型以适应特定领域。

企业应用扩展

本教程可扩展至多种场景：1) 推荐系统：利用生成的标签和标题构建用户画像和物品画像，实现个性化推荐。2) RAG应用：将图像描述作为文档，结合向量数据库实现基于图像的检索增强生成。3) 多模态搜索：支持文本、图像、甚至混合查询。4) 自动化内容生成：为社交媒体、广告等自动生成描述。5) 质量控制：结合规则过滤和嵌入搜索，提高结果相关性。6) 模型微调：针对特定领域（如家具、服装）微调GPT-4o mini，提升标签和标题质量。7) 性能优化：使用异步API调用、分布式处理、模型蒸馏等技术降低延迟和成本。

总结

本教程面向有编程基础的开发者，深入讲解如何利用GPT-4o mini的多模态能力为图像生成标签和描述，并应用于产品搜索。内容涵盖环境搭建、零样本关键词提取、基于嵌入的关键词去重、图像描述生成、少样本标题生成，以及基于文本和图像的搜索实现。教程强调工程实践，包括代码解析、架构设计、常见错误排查和生产优化建议，帮助读者掌握多模态AI应用的完整流程。

• --

内容来源与验证

• 内容等级：PREMIUM_A
• 质量等级：A
• 难度：中级
• 验证等级：V1
• 原文作者：katiagg
• 原文日期：2024-07-18
• 来源许可：MIT
• 来源版本：03d3e2c0b712bd50d7a35f74f1472c3b3c761689
• 查看原文