very interesting

very cool. The start and end make the biggest impact

Yep! I remembered I asked you about claude from anthropic with the 100k context window

so even if the context length is bigger, the performance gets slower and still can't enjoy it well at all

nice challenge to be solved

Very cool - ty for share. I have a sneaky suspicion positional bias is also a thing in prompt design as well, not just document retrieval

It think so too

Our refine prompt, which tells the model to update or repeat an existing answer based on new context, worked really badly with gpt-3.5

The fix was moving the existing answer to the end of the prompt, so that it repeated/updated it properly

This is very interesting!