Building a Multimodal Application with VectorCypherRetriever

Recap and Introduction

Previously, you’ve explored various retrieval strategies using Neo4j, including VectorRetriever, VectorCypherRetriever, HybridRetriever, HybridCypherRetriever, and Text2CypherRetriever to fetch semantically relevant data. Now, we’ll enhance your GraphRAG applications with a multimodal approach using the VectorCypherRetriever. This retriever integrates both textual and visual data, enabling more powerful and accurate queries.

The VectorCypherRetriever allows for multimodal integration, where both text and visual data are used to perform advanced searches within your Neo4j recommendations database. Follow these steps to set up the components.

Try this Cypher command in sandbox:

MATCH (m:Movie {title: "Homeward Bound: The Incredible Journey"})
MATCH (actor:Actor)-[:ACTED_IN]->(m)
RETURN m, collect(actor) AS actors;

Open the 2-neo4j-graphrag\multimodal_app.py file in your code editor.

1. Initialize the Embedder

Create an image embedder using the "clip-ViT-B-32" model to extract visual features from movie posters:

IMAGE_EMBEDDING_MODEL = "clip-ViT-B-32"
embedder = SentenceTransformerEmbeddings(IMAGE_EMBEDDING_MODEL)

2. Initialize the VectorCypherRetriever

Set up the VectorCypherRetriever to integrate both visual and text data for more comprehensive retrieval:

POSTER_INDEX_NAME = "moviePosters"
retrieval_query = "RETURN node.title as title, node.plot as plot, node.poster as posterUrl, score"

def format_record_function(record: neo4j.Record) -> RetrieverResultItem:
    return RetrieverResultItem(
        content=f"Movie title: {record.get('title')}, movie plot: {record.get('plot')}",
        metadata={
            "title": record.get("title"),
            "plot": record.get("plot"),
            "poster": record.get("posterUrl"),
            "score": record.get("score"),
        },
    )

retriever = VectorCypherRetriever(
    driver,
    index_name=POSTER_INDEX_NAME,
    retrieval_query=retrieval_query,
    result_formatter=format_record_function,
    embedder=embedder,
)

query_text = ("Find a movie where in the poster there are only animals without people")
top_k = 3

result = retriever.search(query_text=query_text, top_k=top_k)

for r in result.items:
    print(r.content, r.metadata.get("score"))
    print(r.metadata["poster"])

Tips for Effective Use

Continue

When you are ready, you can move on to the next task.