LlamaIndex
Talk to us
Talk to us

© 2024 LlamaIndex

Laurie Voss • Nov 14, 2023

LlamaIndex Newsletter 2023–11–14

Hello Llama Friends 🦙

LlamaIndex is 1 year old this week! 🎉 To celebrate, we’re taking a stroll down memory lane on our blog with twelve milestones from our first year. Be sure to check it out.

Last week we had a blast with all the new things from OpenAI Dev day to learn and explore at LlamaIndex. There was a special edition newsletter with the things we released the same day as the conference, but this week’s newsletter is full of follow-up releases and explorations — don’t miss our slide deck summing up all the new features!

As always, if you’ve got a cool project or a video to share we’d love to see it! Just drop us a line at news@llamaindex.ai.

🤩 First, the highlights:

  1. Multi-Modal RAG Stack: we unveiled Multi-Modal RAG ****for complex Q&A on documents and images, with new text/image queries and retrieval solutions. Notebook, Tweet, Blog post.
  2. OpenAIAssistantAgent Abstractions: we released new abstractions to connect OpenAI Assistant API with any vector database. Docs, Tweet.
  3. Parallel Function Calling: we enhanced our data extraction and tool execution using OpenAI’s parallel function calling. Tweet.
  4. MechGPT Project: Prof. Markus J. Buehler’s work merges LLM fine-tuning with knowledge graphs for scientific discovery. Tweet, Paper.
  5. Feature Slide Deck: Released a slide deck with 10+ new features and guides post-OpenAI updates.

✨ Feature Releases and Enhancements:

  • We introduced a multi-modal RAG stack for complex document and image QA, featuring text/image queries, joint text/ image embeddings, and versatile storage and retrieval options. Notebook, Tweet, Blog post.
  • We now offer experimental GPT-4-vision support in chat.llamaindex.ai . Users can now upload images for enhanced chatbot interactions. Tweet.
  • We integrated OpenAI’s parallel function calling for efficient extraction of structured data from unstructured text and improving tool execution with agents. Tweet.
  • We introduced OpenAIAssistantAgent abstractions for seamless connection of OpenAI Assistants API with your chosen vector database. Docs, Tweet.
  • We introduced a new agent leveraging OpenAI Assistants API with features like in-house code interpretation, file retrieval, and function calling for external tools integration. Notebook, Tweet.

🎥 Demos:

  • MechGPT by Professor Markus J. Buehler showcases the integration of LLM fine-tuning and knowledge graph creation with LlamaIndex, leading to interesting insights in cross-disciplinary scientific research and hypothesis generation. Tweet, Paper.

🗺️ Guides:

  • We released a concise slide deck that aggregates over 10+ newly shipped features, guides, and analyses, complete with links to accompanying notebooks for developer use based on OpenAI’s recent updates.
  • We also released a full cookbook showing how you can build advanced RAG with the Assistants API — beyond just using the in-house Retrieval tool.
  • We produced a guide on evaluating the OpenAI Assistant API vs RAG with LlamaIndex.
  • Here’s a guide on evaluating How well long-context LLMs (gpt-4-turbo, claude-2) recall specifics in BIG documents? (>= 250k tokens).
  • Here’s another guide that highlights how function calling simplifies structured data extraction, while JSON mode ensures format correctness without schema enforcement.
  • Finally, we released a guide to craft a GPT Builder, enabling an agent to programmatically construct another task-specific agent. This builder streamlines the creation of systems for specific functions. Notebook, Tweet.

✍️ Tutorials:

🎥 Webinars:

  • Check out our webinar with Dan Shipper, CEO of every to talk about the implications of OpenAI’s release updates.
  • A second webinar with Victoria Lin, author of the RA-DIT paper on Fine-tuning + RAG.
  • Last but not least, Mayo Oshin’s webinar with Jerry Liu on How to Analyze Tables In Large Financial Reports Using GPT-4.

Related articles