Hello Llama Friends 🦙
LlamaIndex is 1 year old this week! 🎉 To celebrate, we’re taking a stroll down memory lane on our blog with twelve milestones from our first year. Be sure to check it out.
Last week we had a blast with all the new things from OpenAI Dev day to learn and explore at LlamaIndex. There was a special edition newsletter with the things we released the same day as the conference, but this week’s newsletter is full of follow-up releases and explorations — don’t miss our slide deck summing up all the new features!
As always, if you’ve got a cool project or a video to share we’d love to see it! Just drop us a line at news@llamaindex.ai.
🤩 First, the highlights:
- Multi-Modal RAG Stack: we unveiled Multi-Modal RAG ****for complex Q&A on documents and images, with new text/image queries and retrieval solutions. Notebook, Tweet, Blog post.
- OpenAIAssistantAgent Abstractions: we released new abstractions to connect OpenAI Assistant API with any vector database. Docs, Tweet.
- Parallel Function Calling: we enhanced our data extraction and tool execution using OpenAI’s parallel function calling. Tweet.
- MechGPT Project: Prof. Markus J. Buehler’s work merges LLM fine-tuning with knowledge graphs for scientific discovery. Tweet, Paper.
- Feature Slide Deck: Released a slide deck with 10+ new features and guides post-OpenAI updates.
✨ Feature Releases and Enhancements:
- We introduced a multi-modal RAG stack for complex document and image QA, featuring text/image queries, joint text/ image embeddings, and versatile storage and retrieval options. Notebook, Tweet, Blog post.
- We now offer experimental GPT-4-vision support in chat.llamaindex.ai . Users can now upload images for enhanced chatbot interactions. Tweet.
- We integrated OpenAI’s parallel function calling for efficient extraction of structured data from unstructured text and improving tool execution with agents. Tweet.
- We introduced
OpenAIAssistantAgentabstractions for seamless connection of OpenAI Assistants API with your chosen vector database. Docs, Tweet. - We introduced a new agent leveraging OpenAI Assistants API with features like in-house code interpretation, file retrieval, and function calling for external tools integration. Notebook, Tweet.
🎥 Demos:
- MechGPT by Professor Markus J. Buehler showcases the integration of LLM fine-tuning and knowledge graph creation with LlamaIndex, leading to interesting insights in cross-disciplinary scientific research and hypothesis generation. Tweet, Paper.
🗺️ Guides:
- We released a concise slide deck that aggregates over 10+ newly shipped features, guides, and analyses, complete with links to accompanying notebooks for developer use based on OpenAI’s recent updates.
- We also released a full cookbook showing how you can build advanced RAG with the Assistants API — beyond just using the in-house Retrieval tool.
- We produced a guide on evaluating the OpenAI Assistant API vs RAG with LlamaIndex.
- Here’s a guide on evaluating How well long-context LLMs (gpt-4-turbo, claude-2) recall specifics in BIG documents? (>= 250k tokens).
- Here’s another guide that highlights how function calling simplifies structured data extraction, while JSON mode ensures format correctness without schema enforcement.
- Finally, we released a guide to craft a GPT Builder, enabling an agent to programmatically construct another task-specific agent. This builder streamlines the creation of systems for specific functions. Notebook, Tweet.
✍️ Tutorials:
- Bhavesh Bhat gave us a tutorial on How to Chat with YouTube Videos Using LlamaIndex.
- David Garnitz’s tutorial blog explores the use of VectorFlow alongside ArizePhoenix, Weaviate, and LlamaIndex to manage large data sets.
- Harshad Suryawanshi’s tutorial covers Building My Own ChatGPT Vision with PaLM, KOSMOS-2 and LlamaIndex.
- Sudarshan Koirala’s made a tutorial on Creating OpenAI Assistant Agent with LlamaIndex.
- Our own Ravi Theja released his tutorial on Boosting RAG with Embeddings & Rerankers.
🎥 Webinars:
- Check out our webinar with Dan Shipper, CEO of every to talk about the implications of OpenAI’s release updates.
- A second webinar with Victoria Lin, author of the RA-DIT paper on Fine-tuning + RAG.
- Last but not least, Mayo Oshin’s webinar with Jerry Liu on How to Analyze Tables In Large Financial Reports Using GPT-4.
