Hi, I have a question using Llama Index

SSan Nguyen

Hi, I have a question using Llama Index with Guidance AI but for Ollama. Currently I think guidance only support OpenAI, llama.cpp, or transformer. Is it possible to use langchain model as well. Langchain has OllamaModel support so it will bridge the gap?

13 comments

LLogan M

yea you can use langchain LLMs, just throw it into the service context and it should figure it out

ServiceContext.from_defaults(llm=langchain_llm)

SSan Nguyen

Hi @Logan M , thanks a lot for your reply. Do you mean in this example, I can just replace guidance_llm=OpenAI() with langchain llm?
https://docs.llamaindex.ai/en/stable/examples/output_parsing/guidance_pydantic_program.html

Plain Text

program = GuidancePydanticProgram(
    output_cls=Album,
    prompt_template_str=(
        "Generate an example album, with an artist and a list of songs. Using"
        " the movie {{movie_name}} as inspiration"
    ),
    guidance_llm=OpenAI("text-davinci-003"),
    verbose=True,
)

SSan Nguyen

Or should I set ServiceContext.from_defaults(llm=langchain_llm) and remove the guidance_llm arg?

LLogan M

ohhhh no. I think you can only put LLMs from the guidance package here

LLogan M

More details on guidance models in their docs
https://github.com/guidance-ai/guidance#loading-models

SSan Nguyen

yeah, I think so too. Too bad

LLogan M

You can use llm-format-enforcer instead of guidance to get access to more LLMs though
https://docs.llamaindex.ai/en/stable/examples/output_parsing/lmformatenforcer_pydantic_program.html#lm-format-enforcer-pydantic-program

Plain Text

from llama_index.llms import LangChainLLM

llm = LangChainLLM(lc_llm)


program = LMFormatEnforcerPydanticProgram(
    output_cls=Album,
    prompt_template_str=(
        "Your response should be according to the following json schema: \n"
        "{json_schema}\n"
        "Generate an example album, with an artist and a list of songs. Using"
        " the movie {movie_name} as inspiration. "
    ),
    llm=llm,
    verbose=True,
)

LLogan M

(note: Have not tried this package yet lol but seems basically the same)

SSan Nguyen

oh, nice. thank you very much. I will give it a try.

SSan Nguyen

@Logan M Is there a way to integrate PydanticProgram to Agent? I would like the ReAct agent to use this LMFormatEnforcer.

LLogan M

Hmm... Not easily right now 🤔

LLogan M

But, a good idea. Agents + Structured Outputs going onto the roadmap

SSan Nguyen

@Logan M yeah I think Agent can use structured outputs to decide the next step or calling the function/tools more precisely. My hope is that can produce a more consistent behavior.

Add a reply

Find answers from the community

Hi, I have a question using Llama Index