Have you ever heard of Cunningham’s Law? Cunningham’s Law states:
The best way to get the correct answer on the internet is not to ask a question; it’s to post the wrong answer.
It also seems that this is true when it comes to LLMs!
While LLMs are good at generating Cypher statements, they really excel at correcting the statements they have written.
To complete this challenge, you must modify the initCypherEvaluationChain() function in modules/agent/tools/cypher/cypher-evaluation.chain.ts to return a chain that validates the provided Cypher statement for accuracy and corrects where necessary.
-
Create a prompt instructing the LLM to analyze a Cypher statement and return a list of errors.
-
Create a chain that replaces placeholders in the prompt for
schema,question,cypher, anderrors. -
Pass the formatted prompt to the LLM
-
Parse the output as a JSON object.
This chain will recursively correct the Cypher statement generated by the LLM.
Opencypher-evaluation.chain.ts →
Prompt Template
In the initCypherEvaluationChain() function, use the PromptTemplate.fromTemplate() method to create a new prompt template with the following prompt.
Unresolved directive in lesson.adoc - include::https://raw.githubusercontent.com/neo4j-graphacademy/app-python/main/prompts/cypher-evaluation.txt[]Output Instructions
This prompt instructs the LLM to output a JSON object containing keys for cypher and errors.
This differs from the chains you have built before because they all return a string. To parse the output as a string, you will use the JsonOutputParser class to interpret the response and coerce it into an object.
Your code should resemble the following:
Unresolved directive in lesson.adoc - include::https://raw.githubusercontent.com/neo4j-graphacademy/app-python/main/src/solutions/modules/agent/tools/cypher/cypher-evaluation.chain.ts[tag=prompt, indent=0]Braces in prompts
Use double braces ({{ and }}) to escape braces that are not text placeholders.
Return a Runnable Sequence
Use the RunnableSequence.from() method to create a new chain.
return RunnableSequence.from([
// ...
])Initial Inputs
The chain will recursively verify using the output described in the prompt, which includes an array of errors.
The prompt will need these in string format, so as the first step, use the RunnablePassthrough.assign() method to convert the array of errors into a single string.
Unresolved directive in lesson.adoc - include::https://raw.githubusercontent.com/neo4j-graphacademy/app-python/main/src/solutions/modules/agent/tools/cypher/cypher-evaluation.chain.ts[tag=startsequence, indent=0]
Unresolved directive in lesson.adoc - include::https://raw.githubusercontent.com/neo4j-graphacademy/app-python/main/src/solutions/modules/agent/tools/cypher/cypher-evaluation.chain.ts[tag=assign, indent=2]
// ...
Unresolved directive in lesson.adoc - include::https://raw.githubusercontent.com/neo4j-graphacademy/app-python/main/src/solutions/modules/agent/tools/cypher/cypher-evaluation.chain.ts[tag=endsequence, indent=0]Format Prompt and Process
Now that you have the inputs that the prompt expects, update the chain to format the prompt, pass it to the LLM to process and parse the output.
Unresolved directive in lesson.adoc - include::https://raw.githubusercontent.com/neo4j-graphacademy/app-python/main/src/solutions/modules/agent/tools/cypher/cypher-evaluation.chain.ts[tags="runnable", indent=0]Completed Sequence
If you have followed the steps correctly, your code should resemble the following:
Unresolved directive in lesson.adoc - include::https://raw.githubusercontent.com/neo4j-graphacademy/app-python/main/src/solutions/modules/agent/tools/cypher/cypher-evaluation.chain.ts[tag=function, indent=0]Testing your changes
If you have followed the instructions, you should be able to run the following unit test to verify the response using the npm run test command.
npm run test cypher-evaluation.chain.test.tsView Unit Test
Unresolved directive in ../../../../includes/test.adoc - include::https://raw.githubusercontent.com/neo4j-graphacademy/app-python/main/src/modules/agent/tools/cypher/cypher-evaluation.chain.test.ts[]It works!
If all the tests have passed, you will have a chain that evaluates a Cypher statement and provides hints if any errors are detected.
Hit the button below to mark the challenge as complete.
Summary
In this lesson, you built a chain that evaluates the Cypher statement generated in the Cypher Generation chain.
In the next lesson, you will create a chain that will generate an authoritative answer to a question based on the context provided.