Find answers from the community

Updated last month

Counting tokens in llm instances without a query engine

At a glance

The community member is asking how to count tokens when using LLM instances in a workflow without using a query engine, and whether the TokenCountingHandler needs to be set globally in a callback manager. In the comments, another community member suggests skipping the token counting handler and creating a custom counter using newer instrumentation, providing a link to a Colab notebook. The second comment indicates that this will track every LLM call made. There is no explicitly marked answer in the post and comments.

Useful resources
Hello, how can i count tokens when using LLM instances in a workflow and not using a query engine as shown in the example in the docs?

Also does the TokenCountingHandler have to be set globally in a callback manager?
L
2 comments
Skip the token counting handler, create your own counter using the newer instrumentation
https://colab.research.google.com/drive/1QV01kCEncYZ0Ym6o6reHPcffizSVxsQg?usp=sharing

I really should put this in the docs somewhere
This will track every LLM call made
Add a reply
Sign up and join the conversation on Discord