Lessons from Prompt Engineering

Sharing lessons on Prompt Engineering as we build an Insights product that analyzes copious amounts of unstructured data:

Set up for success by sharing context - Imagine your product is a teacher's helper for scoring tests for elementary school children. When using the prompt, mention what grade is being assessed along with the type of test: "You are a 4th grade Language Arts teacher responsible for analyzing the test results of 4th grade students. The focus of the test is on writing and grammar."

Make your prompt super specific - Avoid leaving the answer to the question you are asking to the AI's judgment. More specificity leads to higher accuracy. Building on the example above, specify how the test needs to be assessed: "Assess the student's writing on creativity, grammar, and coherence. Give specific examples of areas to improve."

Less is more - Focus on asking a couple of questions in your prompt. When analyzing large amounts of unstructured data, it's tempting to include all the attributes being assessed in the same prompt. This approach often leads to less accurate results. Time and again, we have seen better accuracy by limiting the total outputs.

Order of instructions in the prompt - We know that LLM output is non-deterministic from a language standpoint. However, there was a specific instance where we were getting varied results for the same prompt across multiple runs, even though the response should have been definitive. Upon further investigation, we found that we were passing a parameter that belonged to the User prompt in the System prompt. Once we adjusted the order, our results aligned with our expectations.

Format of input and output in the prompt - We initially started with plaintext output; however, after further research and experimentation, we are seeing better results using JSON and XML for our inputs and outputs. Stay tuned for more details on which format outperforms the other.

What are some of your key lessons with using LLMs in your applications? If you are curious about the insights you can gain with our product, feel free to reach out.

#promptengineering

#customizedinsights

#genai

#gottapromptrightway

‍