Vector Indexes

Vectorizing Movie Plots

GraphAcademy created a Movie Recommendation Sandbox when you enrolled in this course. The sandbox database contains over 9000 movies, 15000 actors, and over 100000 user ratings.

Each movie has a .plot property.

MATCH (m:Movie {title: "Toy Story"})
RETURN m.title AS title, m.plot AS plot

"A cowboy doll is profoundly threatened and jealous when a new spaceman figure supplants him as top toy in a boy's room."

You can use the vector index to find the most similar movies by converting the plots into vector embeddings and comparing them.

You will use a pre-created CSV file of 1000 movie plot vector embeddings in this lesson.

The CSV file contains:

movieId, embedding
1, [-0.0271058, -0.0242211, 0.0060390322, -0.02437703, ...]
2, [-0.001596838, -0.022397375, 0.0046575777, 0.0019427929, ...]

Loading Embeddings

The embeddings will be stored as a .embedding property on the (:Movie) node.

You will use the LOAD CSV command to load the embeddings into the Neo4j Sandbox instance.

The following Cypher loads the embeddings CSV file, performs a MATCH query to find the (:Movie) node with the corresponding movieId property, and then sets the .embedding property on that node.

Review this Cypher statement before running it.

LOAD CSV WITH HEADERS
FROM 'https://data.neo4j.com/llm-fundamentals/openai-embeddings.csv'
AS row
MATCH (m:Movie {movieId: row.movieId})
CALL db.create.setNodeVectorProperty(m, 'embedding', apoc.convert.fromJsonList(row.embedding))
RETURN count(*)

The statement:

Run the statement to create the Movie embeddings.

Once complete, you can query the database to see the .embedding property on the (:Movie) nodes.

MATCH (m:Movie {title: "Toy Story"})
RETURN m.title AS title, m.plot AS plot, m.embedding

Creating the Vector Index

You will need to create a vector index to search across these embeddings.

You will use the CREATE VECTOR INDEX Cypher statement to create the index:

CREATE VECTOR INDEX [index_name] [IF NOT EXISTS]
FOR (n:LabelName)
ON (n.propertyName)
OPTIONS "{" option: value[, ...] "}"

CREATE VECTOR INDEX expects the following parameters:

Review and run the following Cypher to create the vector index:

CREATE VECTOR INDEX moviePlots IF NOT EXISTS
FOR (m:Movie)
ON m.embedding
OPTIONS {indexConfig: {
 `vector.dimensions`: 1536,
 `vector.similarity_function`: 'cosine'
}}

Note that the index is called moviePlots, it is against the Movie label, and it is on the .embedding property. The vector.dimensions is 1536 (as used by OpenAI) and the vector.similarity_function is cosine. The IF NOT EXISTS clause ensures that the statement only creates the index if it does not already exist.

Run the statement to create the index.

SHOW INDEXES  YIELD id, name, type, state, populationPercent WHERE type = "VECTOR"

Querying Vector Indexes

You can query the index using the db.index.vector.queryNodes() procedure.

The procedure returns the requested number of approximate nearest neighbor nodes and their similarity score, ordered by the score.

CALL db.index.vector.queryNodes(
    indexName :: STRING,
    numberOfNearestNeighbours :: INTEGER,
    query :: LIST<FLOAT>
) YIELD node, score

The procedure accepts three parameters:

The procedure yields two arguments:

You can use this procedure to find the closest embedding value to a given embedding.

For example, find movies with a similar plot to another.

Review this Cypher before running it.

MATCH (m:Movie {title: 'Toy Story'})

CALL db.index.vector.queryNodes('moviePlots', 6, m.embedding)
YIELD node, score

RETURN node.title AS title, node.plot AS plot, score

The query finds the Toy Story Movie node and uses the .embedding property to find the most similar plots. The db.index.vector.queryNodes() procedure uses the moviePlots vector index to find similar embeddings.

Run the query. The procedure returns the requested number of approximate nearest neighbor nodes and their similarity score, ordered by the score.

The similarity score is between 0.0 and 1.0, with 1.0 being the most similar. Note how the most similar plot is that of the Toy Story movie itself!

Considerations

As you can see, this approach is relatively straightforward and can quickly yield results. The downside to this approach is that it relies heavily on the embeddings and similarity function to produce valid results.

This approach is also a black box - with 1536 dimensions, and it would be impossible to determine how the vectors are structured and how they influenced the similarity score.

The movies returned look similar, but without reading and comparing them, you would have no way of verifying that the results are correct.

Check your understanding