Find answers from the community

Updated 2 weeks ago

The Multi Concierge Flow Issue

I've been digging into an issue with the multi concierge flow. I have 2 agents. Each can add or multiply 2 numbers. I ask the system to 'add 10 to 10 and then multiply the result by 20'.

My expectation is that the first agent says "10 + 10 is 20", then transfers. I see this from the tools and chat history:
tools input:
['transfer_to_agent']

chat history:
Plain Text
[ChatMessage(role=<MessageRole.SYSTEM: 'system'>, additional_kwargs={}, blocks=[TextBlock(block_type='text', text="You are on orchestration agent.\nYour job is to decide which agent to run based on the current state of the user and what they've asked to do.\nYou do not need to figure out dependencies between agents; the agents will handle that themselves.\nHere are the agents you can choose from:\nAddition Agent: Used to add two numbers together.\nMultiply Agent: Used to multiply two numbers together.\n\n\nHere is the current user state:\n\n\nPlease assist the user and transfer them as needed.\nDo not make up information or provide information that is not in the user state or agent context.\n")]), ChatMessage(role=<MessageRole.USER: 'user'>, additional_kwargs={}, blocks=[TextBlock(block_type='text', text='Add 10 to 10 and then multiply the result by 22')]), ChatMessage(role=<MessageRole.ASSISTANT: 'assistant'>, additional_kwargs={'tool_calls': [ChatCompletionMessageToolCall(id='call_SOOg9k2TNckhXV3MAITRo4Vs', function=Function(arguments='{"a":10,"b":10}', name='add_two_numbers'), type='function')]}, blocks=[]), ChatMessage(role=<MessageRole.TOOL: 'tool'>, additional_kwargs={'tool_call_id': 'call_SOOg9k2TNckhXV3MAITRo4Vs', 'name': 'add_two_numbers'}, blocks=[TextBlock(block_type='text', text='20')])]


Results in the following tool selection:
Plain Text
[ToolSelection(tool_id='call_34DqgTGSX5VhWVMmCiTgsFhQ', tool_name='multiply_two_numbers', tool_kwargs={'a': 20, 'b': 22})]


Despite not giving it a "multiply_two_numbers" tool, it says that's the next one it should run, so it fails on 'agent_name' missing.
L
O
18 comments
The chat history you logged there has a tool call already in it

Plain Text
ChatMessage(role=<MessageRole.ASSISTANT: 'assistant'>, additional_kwargs={'tool_calls': [ChatCompletionMessageToolCall(id='call_SOOg9k2TNckhXV3MAITRo4Vs', function=Function(arguments='{"a":10,"b":10}', name='add_two_numbers'), type='function')]}, blocks=[]), ChatMessage(role=<MessageRole.TOOL: 'tool'>, additional_kwargs={'tool_call_id': 'call_SOOg9k2TNckhXV3MAITRo4Vs', 'name': 'add_two_numbers'}, blocks=[TextBlock(block_type='text', text='20')])])


So the message at the end of this chat history is the assistant picking a tool. Where in the code did you log this?
It's not picking that tool though. I logged this in the orchestrator.
The tool in the chat history is add_two_numbers. It's picking multiply_two_numbers My understanding is that the orchestrator should only be picking transfer_to_agent and return an agent_name
Is there possibly some kind of message history maintained by AzureOpenAI? It almost feels like it's getting some kind of memory from somewhere other than locally.
Otherwise, it's hallucinating the multiply_two_numbers tool_name
Azure does not keep memory
Where in the orcehsrator was this logged? The start? After an llm call?
Seems like the chat history has a tool call for add_two_number, but not a tool result (which is odd)
I feel like I've ran this exact example and it worked lol
This is really hard to help you debug if you don't share something reproducible. Let me put this into a colab notebook
Giving enough code is hard with the text limit. What's the best way to give it? Free google colab notebooks?
My colleague said he is pretty sure that he can get the multi agent python notebook to do it if he simply says "add 10 + 10 and then multiply times 2 and then tell me the capital of France" will do it.
NVM. That is working in the notebook. I can try to create a notebook that works tomorrow that I can share
I have a notebook I just made, ez pz
https://colab.research.google.com/drive/1RNA0DqjsU5RJELFO7nSJPX6YZQQ3KFQe?usp=sharing

LLMs are too smart for simple math lol, a single agent keeps answering instead of passing between. Which in this hyper simplisitic case is fine imo
Thanks! I'll update this with an example.
So, I used yours and it repeatedly worked in all cases I tried to force it to. Which led me to investigate the model deployments and settings. I updated the model to 11-20-2024 and fixed the api_version it was reference (it was a old date), and now all is working as expected.
glad my example helped narrow down the issue
Add a reply
Sign up and join the conversation on Discord