Now you have all the components needed to retrieve data from Neo4j with Cypher based on a user input. It is time to combine them.
To complete this challenge, you must create a Runnable instance that:
-
Generates and evaluates a Cypher statement
-
Use the Cypher statement to retrieve data from the database
-
Extract the element IDs and convert the results to a
stringfor use in the context prompt -
Generate an answer using the answer generation chain
-
Save the response to the database along with the Cypher statement
-
Return the LLM response
{lab-folder}/cypher-retrieval.chain.ts →
Cypher Generation and Evaluation
To generate and evaluate a new Cypher statement, you’ll need to create a function that generates a Cypher statement.
The {lab-folder}/cypher-retrieval.chain.ts file already has a placeholder function called recursivelyEvaluate() to perform this task.
Unresolved directive in lesson.adoc - include::https://raw.githubusercontent.com/neo4j-graphacademy/genai-workshop/main/src/modules/agent/tools/cypher/cypher-retrieval.chain.ts[tag="recursive"]In this function, first use the initCypherGenerationChain function from Cypher Generation Chain lesson and
initCypherEvaluationChain function from the Cypher Evaluation Chain lesson to create the generation and evaluation chains.
Unresolved directive in lesson.adoc - include::https://raw.githubusercontent.com/neo4j-graphacademy/genai-workshop/main/src/solutions/modules/agent/tools/cypher/cypher-retrieval.chain.ts[tag="chains",indent=0]Next, invoke the generationChain to generate an initial Cypher statement.
Unresolved directive in lesson.adoc - include::https://raw.githubusercontent.com/neo4j-graphacademy/genai-workshop/main/src/solutions/modules/agent/tools/cypher/cypher-retrieval.chain.ts[tag="initialcypher",indent=0]Now, use a while loop to recursively evaluate the Cypher statement up to five times until the number of errors the evaluation chain returns is 0.
Unresolved directive in lesson.adoc - include::https://raw.githubusercontent.com/neo4j-graphacademy/genai-workshop/main/src/solutions/modules/agent/tools/cypher/cypher-retrieval.chain.ts[tag="evaluateloop",indent=0]Finally, return the cypher statement.
Unresolved directive in lesson.adoc - include::https://raw.githubusercontent.com/neo4j-graphacademy/genai-workshop/main/src/solutions/modules/agent/tools/cypher/cypher-retrieval.chain.ts[tag="evaluatereturn",indent=0]id() to elementId() replacement
The first line of this code contains a fix that converts id({variable}) to elementId({variable}).
No matter what we try in the prompt, the GPT-3.5 Turbo and GPT-4 models use the deprecated id() method over the elementId().
Eventually, the models will recognize that the id() method is deprecated. This problem suggests training a model specifically to generate valid Cypher statements might be necessary.
View full recursivelyEvaluate function
Unresolved directive in lesson.adoc - include::https://raw.githubusercontent.com/neo4j-graphacademy/genai-workshop/main/src/solutions/modules/agent/tools/cypher/cypher-retrieval.chain.ts[tag="recursive",indent=0]Handling errors
The LLM will generate a correct Cypher statement most of the time. But, as we’ve found in testing, depending on the instructions provided to the prompt, the loop of Cypher generation and evaluation can be flaky.
You can execute your Cypher statement with an additional evaluation loop to make the application more robust. If the database throws an error, you can analyze the error message using the same evaluation chain and rewrite the statement accordingly.
Find the getResults() function in {lab-folder}/cypher-retrieval.chain.ts.
Unresolved directive in lesson.adoc - include::https://raw.githubusercontent.com/neo4j-graphacademy/genai-workshop/main/src/modules/agent/tools/cypher/cypher-retrieval.chain.ts[tag="results"]Replace the // TODO comment with code that will attempt to execute the Cypher statement and retry if the graph.query() method throws an error.
Start by defining a results variable and an attempts variable to hold the maximum number of attempts.
Define a mutable cypher statement to hold the Cypher statement.
Then, call the initCypherEvaluationChain() function to create an instance of the evaluation chain.
Unresolved directive in lesson.adoc - include::https://raw.githubusercontent.com/neo4j-graphacademy/genai-workshop/main/src/solutions/modules/agent/tools/cypher/cypher-retrieval.chain.ts[tag="resultvars",indent=0]Next, create a while loop that will iterate a maximum of five times.
Inside use try/catch to attempt to execute the Cypher statement.
If an error is thrown, pass the .message property along with the Cypher statement, question, and schema to the evaluation chain.
Assign the output of the evaluation chain to the cypher statement.
Unresolved directive in lesson.adoc - include::https://raw.githubusercontent.com/neo4j-graphacademy/genai-workshop/main/src/solutions/modules/agent/tools/cypher/cypher-retrieval.chain.ts[tag="resultloop"]Finally, return the results.
Building the Chain
This section will take place in the initCypherRetrievalChain() function.
Unresolved directive in lesson.adoc - include::https://raw.githubusercontent.com/neo4j-graphacademy/genai-workshop/main/src/modules/agent/tools/cypher/cypher-retrieval.chain.ts[tag="function"]Since an agent will call this chain, it will receive a structured input containing both an input and a rephrasedQuestion.
Unresolved directive in lesson.adoc - include::https://raw.githubusercontent.com/neo4j-graphacademy/genai-workshop/main/src/modules/agent/agent.types.ts[tag=agenttoolinput,indent=0]Initialize Chains
You must use the Generate Authoritative Answer Chain from the previous lesson to generate an answer.
Use the initGenerateAuthoritativeAnswerChain() function
Unresolved directive in lesson.adoc - include::https://raw.githubusercontent.com/neo4j-graphacademy/genai-workshop/main/src/solutions/modules/agent/tools/cypher/cypher-retrieval.chain.ts[tag="answerchain",indent=0]Generate a Cypher Statement
Now, define the output.
As with the Vector Retrieval tool, you can return a Runnable using RunnablePassthrough.assign().
The first step is to call the recursivelyEvaluate() function, assigning the output to the cypher key.
Unresolved directive in lesson.adoc - include::https://raw.githubusercontent.com/neo4j-graphacademy/genai-workshop/main/src/solutions/modules/agent/tools/cypher/cypher-retrieval.chain.ts[tag="cypher",indent=0]Get Results
Use the getResults() function to get the results from the database.
Unresolved directive in lesson.adoc - include::https://raw.githubusercontent.com/neo4j-graphacademy/genai-workshop/main/src/solutions/modules/agent/tools/cypher/cypher-retrieval.chain.ts[tag="getresults",indent=0]Manipulate Results
You will need to extract any element IDs from the results to save the context to the database.
The utils.ts file exports an extractIds() function that recursively iterates through the results to find any objects with a key of _id.
View the extractIds() function
Unresolved directive in lesson.adoc - include::https://raw.githubusercontent.com/neo4j-graphacademy/genai-workshop/main/src/utils.ts[tag="extractids",indent=0]The result obtained in the previous step must also be converted to a string.
If there is only one result, use JSON.stringify() to convert the first object to a JSON string, otherwise return a string representing the entire array.
Unresolved directive in lesson.adoc - include::https://raw.githubusercontent.com/neo4j-graphacademy/genai-workshop/main/src/solutions/modules/agent/tools/cypher/cypher-retrieval.chain.ts[tag="extract",indent=0]Generate Output
The input and context can then be passed to the Authoritative Answer Generation chain to generate an answer.
Unresolved directive in lesson.adoc - include::https://raw.githubusercontent.com/neo4j-graphacademy/genai-workshop/main/src/solutions/modules/agent/tools/cypher/cypher-retrieval.chain.ts[tag="answer",indent=0]Save response to database
Next, use the saveHistory() function built in module 3 to save the details of the response to the database.
Unresolved directive in lesson.adoc - include::https://raw.githubusercontent.com/neo4j-graphacademy/genai-workshop/main/src/solutions/modules/agent/tools/cypher/cypher-retrieval.chain.ts[tag="save",indent=0]Return the output
Finally, the pick() function returns the output key.
Unresolved directive in lesson.adoc - include::https://raw.githubusercontent.com/neo4j-graphacademy/genai-workshop/main/src/solutions/modules/agent/tools/cypher/cypher-retrieval.chain.ts[tag="output",indent=0]Final Function
If you have followed the instructions correctly, your code should resemble the following:
Unresolved directive in lesson.adoc - include::https://raw.githubusercontent.com/neo4j-graphacademy/genai-workshop/main/src/solutions/modules/agent/tools/cypher/cypher-retrieval.chain.ts[tag=function]Testing your changes
If you have followed the instructions, you should be able to run the following unit test to verify the response using the npm run test command.
npm run test cypher-retrieval.chain.test.tsView Unit Test
Unresolved directive in ../../../../includes/test.adoc - include::https://raw.githubusercontent.com/neo4j-graphacademy/genai-workshop/main/src/modules/agent/tools/cypher/cypher-retrieval.chain.test.ts[]Randomized responses
LLMs are probabilistic models, meaning they generate different responses with each call.
Given this variability, you might find that not all tests pass whenever testing this function with multiple tests. Therefore, running the test several times may be necessary to achieve consistent results.
Verifying the Test
If every test in the test suite has passed, a new (:Session) node with a .id property of cypher-retriever-3 will have been created in your database.
The session should have atleast one (:Response) node, linked with a :CONTEXT relationship to a movie with the title Neo4j - Into the Graph.
Click the Check Database button below to verify the tests have succeeded.
Hint
You can compare your code with the solution in src/solutions/modules/agent/tools/cypher/cypher-retrieval.chain.ts and double-check that the conditions have been met in the test suite.
Solution
You can compare your code with the solution in src/solutions/modules/agent/tools/cypher/cypher-retrieval.chain.ts and double-check that the conditions have been met in the test suite.
You can also run the following Cypher statement to double-check that the index has been created in your database.
MATCH (s:Session {id: 'cypher-retrieval-3'})
RETURN s, [
(s)-[:HAS_RESPONSE]->(r) | [r,
[ (r) -[:CONTEXT]->(c) | c ]
]
]Once you have verified your code and re-ran the tests, click Try again…* to complete the challenge.
Summary
In this lesson, you combined the components built during this module to create a chain that will generate a Cypher statement that answers the user’s question, execute the Cypher statement, and generate a response.
In the next module, you will build an agent that combines this chain with the Vector Retrieval Chain to create an agent that uses an LLM to choose the correct tool to answer the user’s question.