I m trying to teach openai s text

ttilleul

I'm trying to teach openai's text-davinci-003 a new programming language based on BASIC. I'm feeding llama_index with a dozen text files explaining variables types, if/then/else blocks, PRINT/INPUT, how to comment code, and a few other things like numeric functions (min, max, abs, sgn, etc). The files are divided by topic.

I'm using a simple vector reading the documents stored in a folder as explained in the very first basic examples of llama index. There's also a service context with temp set to 0.7 and a prompt helper to to slightly tweak the initial settings.

When I ask davinci to explain a particular instruction or function, the result is usually good.
When I ask the AI to write code that asks the user his age and then write a funny comment about his age category. The code returned is actually rather good:

Plain Text

INPUT "What is your age?": Age

IF Age < 18 THEN
   PRINT "You are a kid!"
ELSE IF Age >= 18 AND Age < 30 THEN
   PRINT "You are young and wild!"
ELSE IF Age >= 30 AND Age < 50 THEN
   PRINT "You are a grown-up!"
ELSE
   PRINT "You are wise!"
END IF

But if ask the AI to do the same AND comment the code (I just add " and comment your code" at the end of the query, it will return something completely wrong either in JS or in C.

Any idea why a single additional simple order is enough to break a process that seemed to work ?

11 comments

ttilleul

Ok I modified/refined the prompt and it's more or less back to normal ...

LLogan M

Are you using custom prompt templates as well?

LLogan M

Might help to reinforce the idea that it should be writing BASIC

LLogan M

Could also lower the temperature, 0.7 is rather high

ttilleul

for now I'm using a QuestionAnswerPrompt inspired of the default one but with some information ...

Plain Text

QA_PROMPT_TMPL = (
    "You are an expert in AOZ programming language. \n"
    "When asked to write code, you write exclusively AOZ code, follow the AOZ syntax and use only valid AOZ instructions and functions. \n"
    "Remember that AOZ is syntactically more like BASIC than JS, C or Python. \n"
    "Context information is below. \n"
    "---------------------\n"
    "{context_str}"
    "\n---------------------\n"
    "Given the context information and not prior knowledge, "
    "answer the question: {query_str}\n"
)

The line about the fact that the language is more BASIC than JS/Python helps the AI to write better code (as it is already able to kind of write BASIC code) ... although sometimes it will just invent silly syntax ...

LLogan M

Ah, you should also setup a refine template too

ttilleul

For now my corpus documentation is made of dozen of TXT files (in the end there should be hundred or even more) ... it seems that to build the embeddings, all formatting is removed but when the prompt is sent to openAI, one of the chunk is sent as well (I suppose the one that llama_index thinks is the most relevant) ... since I know that chatGPT likes markdown to better understand the structure of info it is given, I've used simple md for the headings in my TXT files ...

ttilleul

refine template ... how do I do that ?

LLogan M

Here's an example of a refine template for gpt-3.5

If you aren't using gpt-3.5, let me know lol

Plain Text

from langchain.prompts.chat import (
    AIMessagePromptTemplate,
    ChatPromptTemplate,
    HumanMessagePromptTemplate,
)

from llama_index.prompts.prompts import RefinePrompt

# Refine Prompt
CHAT_REFINE_PROMPT_TMPL_MSGS = [
    HumanMessagePromptTemplate.from_template("{query_str}"),
    AIMessagePromptTemplate.from_template("{existing_answer}"),
    HumanMessagePromptTemplate.from_template(
        "I have more context below which can be used "
        "(only if needed) to update your previous answer.\n"
        "------------\n"
        "{context_msg}\n"
        "------------\n"
        "Given the new context, update the previous answer to better "
        "answer my previous query."
        "If the previous answer remains the same, repeat it verbatim. "
        "Never reference the new context or my previous query directly.",
    ),
]


CHAT_REFINE_PROMPT_LC = ChatPromptTemplate.from_messages(CHAT_REFINE_PROMPT_TMPL_MSGS)
CHAT_REFINE_PROMPT = RefinePrompt.from_langchain_prompt(CHAT_REFINE_PROMPT_LC)
...
index.query("my query", similarity_top_k=3, refine_template=CHAT_REFINE_PROMPT)

ttilleul

will context_msg work ? shouldn't it be context_str ?

LLogan M

It's an old legacy thing, it's different for the refine prompt lol

Add a reply

Find answers from the community

I m trying to teach openai s text