openai_fn_spec = to_openai_tool(self._output_cls) but not sooooo helpful.DEFAULT_TEXT_QA_PROMPT doesn't use the SelectorPromptTemplate like refine does?service_context= ServiceContext.from_defaults(prompt_helper=prompt_helper, node_parser=node_parser, chunk_size=1024, llm=OpenAI(
temperature=0.0, model="gpt-4", max_tokens=output_tokens))this is a chat model and not supported in the v1/completions endpoint. Did you mean to use v1/chat/completions?index = VectorStoreIndex.from_vector_store(vector_store=vector_store, service_context=service_context, storage_context=storage_context) filters = MetadataFilters(filters=[ExactMatchFilter(key="name", value="Council Liquor License Review Committee")]) retriever = index.as_retriever(filters=filters, top_k_similarity=1000000) all_nodes = retriever.retrieve() summary_index = SummaryIndex.build_index_from_nodes(all_nodes)
prompt_helper = PromptHelper(
context_window=context_window,
num_output=output_tokens,
chunk_overlap_ratio=0.1,
chunk_size_limit=chunk_size)
node_parser = SimpleNodeParser.from_defaults(text_splitter=text_splitter)
service_context= ServiceContext.from_defaults(prompt_helper=prompt_helper, node_parser=node_parser)
vector_store = WeaviateVectorStore(weaviate_client=client, index_name="Minutes")
storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(documents, llm=OpenAI(
temperature=0.0, model="gpt-4", max_tokens=output_tokens),
service_context=service_context,
storage_context=storage_context) "_node_content": "{\"id_\": \"b1d58d70-4a66-4236-b3f1-be6b2c80c592\", \"embedding\": null, \"metadata\": {\"name\": \"City Council\", \"uuid\": \"5D95AE16-32E0-4256-A9A9-1F9D311ABF33\", \"date\": \"9/5/2023\"}, \"excluded_embed_metadata_keys\": [], \"excluded_llm_metadata_keys\": [], \"relationships\": {\"1\": {\"node_id\": \"ee9091d2-736d-4c0d-91eb-7bd5d17db2b7\", \"node_type\": null, \"metadata\": {\"name\": \"City Council\", \"uuid\": \"5D95AE16-32E0-4256-A9A9-1F9D311ABF33\", \"date\": \"9/5/2023\"}, \"hash\": \"01b122ab1f744ceca5694e65ddf8cbb7bdc03561c2af3cc47b0439763b1ddb21\"}}, \"hash\": \"01b122ab1f744ceca5694e65ddf8cbb7bdc03561c2af3cc47b0439763b1ddb21\", \"text\": \"\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\"}",I'm sorry, but I cannot create a machine learning algorithm or model based on the given data. As an expert Q&A system, I can provide information and answer questions based on the provided context, but I do not have the capability to create or train machine learning models.langchainapi-langchain-1 | 17-Jun-23 19:28:00 - > Building index from nodes: 1 chunks langchainapi-langchain-1 | > Building index from nodes: 1 chunks langchainapi-langchain-1 | 17-Jun-23 19:28:05 - message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=4881 request_id=64146b50553b7ef6b43bc8e7e21a30ac response_code=200 langchainapi-langchain-1 | message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=4881 request_id=64146b50553b7ef6b43bc8e7e21a30ac response_code=200 langchainapi-langchain-1 | 17-Jun-23 19:28:06 - message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=5480 request_id=935cbe4f2158adcb864e902a03a424d1 response_code=200 langchainapi-langchain-1 | message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=5480 request_id=935cbe4f2158adcb864e902a03a424d1 response_code=200 langchainapi-langchain-1 | 17-Jun-23 19:28:11 - > [get_response] Total LLM token usage: 508 tokens langchainapi-langchain-1 | > [get_response] Total LLM token usage: 508 tokens langchainapi-langchain-1 | 17-Jun-23 19:28:11 - > [get_response] Total embedding token usage: 0 tokens langchainapi-langchain-1 | > [get_response] Total embedding token usage: 0 tokens langchainapi-langchain-1 | 17-Jun-23 19:28:11 - > [get_response] Total LLM token usage: 6311 tokens langchainapi-langchain-1 | > [get_response] Total LLM token usage: 6311 tokens langchainapi-langchain-1 | 17-Jun-23 19:28:11 - > [get_response] Total embedding token usage: 0 tokens langchainapi-langchain-1 | > [get_response] Total embedding token usage: 0 tokens
ValueError: Invalid template: Context information is below.
---------------------
{context_str} prompt = f"""Write three unique titles of the text. {extra_prompt} \n Write it as an exciting podcast description. Act as an Copywriter. Try to include all topics. No longer than 200 tokens \n Return the format in a JSON Object {{"titles": ["Title 1", "Title 2", "Title 3"]}}\n return only valid JSON:"""
response = self.index.query(prompt, similarity_top_k=5, response_mode="tree_summarize") prompt = self._prompt(municipality_dto.name)
prompt = re.sub(
r"\s{3,}",
" ",
prompt,
)
response_synthesizer = CompactAndAccumulate(
text_qa_template=CUSTOM_TEXT_QA_PROMPT,
output_cls=ContactList,
)
query_engine_contact = index.as_query_engine(
response_synthesizer=response_synthesizer
)
contact_list: PydanticResponse = query_engine_contact.query(prompt) # type: ignore File "/Users/bmax/src/pursuit/ai/lib/python3.8/site-packages/typing_extensions.py", line 2562, in wrapper
return __arg(*args, **kwargs)
File "/Users/bmax/src/pursuit/ai/lib/python3.8/site-packages/pydantic/main.py", line 1026, in parse_raw
raise pydantic_core.ValidationError.from_exception_data(cls.__name__, [error])
pydantic_core._pydantic_core.ValidationError: 1 validation error for ContactList
__root__
Unterminated string starting at: line 232 column 15 (char 4738) [type=value_error.jsondecode, input_value='{\n "contacts": [\n ... Clogged Weekdays 6 a.m', input_type=str]INFO:openai:error_code=rate_limit_exceeded error_message='Rate limit reached for gpt-4 in organization org-4TyujAPAwHUpADHfmELtxusc on tokens per min. Limit: 10000 / min. Please try again in 6ms. Contact us through our help center at help.openai.com if you continue to have issues.' error_param=None error_type=tokens message='OpenAI API error received' stream_error=FalseINFO:openai:error_code=None error_message="'$.input' is invalid. Please check the API reference: https://platform.openai.com/docs/api-reference." error_param=None error_type=invalid_request_error message='OpenAI API error received' stream_error=False error_code=None error_message="'$.input' is invalid. Please check the API reference: https://platform.openai.com/docs/api-reference." error_param=None error_type=invalid_request_error message='OpenAI API error received' stream_error=False INFO:openai:error_code=None error_message="'$.input' is invalid. Please check the API reference: https://platform.openai.com/docs/api-reference." error_param=None error_type=invalid_request_error message='OpenAI API error received' stream_error=False error_code=None error_message="'$.input' is invalid. Please check the API reference: https://platform.openai.com/docs/api-reference." error_param=None error_type=invalid_request_error message='OpenAI API error received' stream_error=False
RssReader().load_data([url]) to load an xml file. Currently it's creating a document per line item and each get_content() of the document is only like 169 characters. Is this the optimal way? especially because I'm going to be loading a lot more xml's and expect them to be searchable (also, i want to change what metadata they're automatically using).text_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
chunk_size=chunk_size,
chunk_overlap=300)
reader = UnstructuredXMLLoader('./data/agenda.xml')
documents_unstructured = reader.load_and_split(text_splitter=text_splitter)
documents = RssReader().load_data(['https://a2gov.legistar.com/Feed.ashx?M=CalendarDetail&ID=1062179&GUID=72A10A68-6E3E-4A2D-9C1B-15FF554DC60F&Title=City+of+Ann+Arbor+-+Meeting+of+City+Council+on+8%2f21%2f2023+at+7%3a00+PM'])