LLM batch processing with MyMagic AI and LlamaIndex

This is a guest post from MyMagic AI.

MyMagic AI allows processing and analyzing large datasets with AI, via a powerful API for batch inference that brings various open-source Large Language Models (LLMs) such as LLAMA 70B, Mistral 7B, Mixtral 8x7B, CodeLlama70b, and advanced Embedding models to its users. Our framework is designed to perform data extraction, summarization, categorization, sentiment analysis, training data generation, and embedding.

LlamaIndex Integration 🦙.

How It Works:

1. Setup:
   1. Organize Your Data in AWS S3 Bucket:
      1. Create a folder using your user ID assigned to you upon registration.
      2. Inside that folder, create another folder (called a "session") to store all the files you need for your tasks.
   2. Purpose of the 'Session' Folder:
      1. This "session" folder keeps your files separate from others, making sure that your tasks run on the right set of files. You can name your session subfolder anything you like.
   3. Granting Access to MyMagic AI:
      1. To allow MyMagic AI to securely access your files in the cloud, follow the setup instructions provided in the MyMagic AI documentation.
2. Install: Install both MyMagic AI’s API integration and LlamaIndex library: pip install llama-index and pip install llama-index-llms-mymagic
3. API Request: The llamaIndex library is a wrapper around MyMagic AI’s API. What it does under the hood is simple: it sends a POST request to the MyMagic AI API while specifying the model, storage provider, bucket name, session name, and other necessary details.

import asyncio

from llama_index.llms.mymagic import MyMagicAI

llm = MyMagicAI(

api_key="user_...", # provided by MyMagic AI upon sign-up

storage_provider="s3",

bucket_name="batch-bucket", # you may name anything

session="my-session",

role_arn="arn:aws:iam::<your account id>:role/mymagic-role",

system_prompt="You are an AI assistant that helps to summarize the documents without essential loss of information", # default prompt at https://docs.mymagic.ai/api-reference/endpoint/create

region="eu-west-2",

)

We have designed the integration to allow the user to set up the bucket and data together with the system prompt when instantiating the llm object. Other inputs, e.g. question (i.e. your prompt), model and max_tokens are dynamic requirements when submitting complete and acomplete requests.

resp = llm.complete(

question="Summarise this in one sentence.",

model="mixtral8x7",

max_tokens=20, # default is 10

)

print(resp)

async def main():

aresp = await llm.acomplete(

question="Summarize this in one sentence.",

model="llama7b",

max_tokens=20,

)

print(aresp)

asyncio.run(main())

This dynamic entry allows developers to experiment with different prompts and models in their workflow while also controlling for model output to cap their spending limit. MyMagic AI’s backend supports both synchronous requests (complete) and asynchronous requests (acomplete). It is advisable, however, to use our async endpoints as much as possible as batch jobs are inherently asynchronous with potentially long processing times (depending on the size of your data).

Currently, we do not support chat or achat methods as our API is not designed for real-time interactive experience. However, we are planning to add those methods in the future that will function in a “batch way”. The user queries will be aggregated and appended as one prompt (to give the chat context) and sent to all files at once.

Use Cases

While there are myriads of use cases, here we provide a few to motivate our users to embed our API in their workflows that are a good fit for batch processing.

1. Batch Extraction

Imagine needing to extract specific information from millions of files stored in a bucket. Information from all files will be extracted with one API call instead of a million sequential ones.

2. Batch Classification

For businesses looking to classify customer reviews such as positive, neutral, and negative. With one request you can start processing the requests over the weekend and get them ready by Monday morning.

3. Batch Embedding

Embedding text files for further machine learning applications is another powerful use case of MyMagic AI's API. You will be ready for your vector db in a matter of days not weeks.

4. Batch Training Data Generation

Imagine generating thousands of synthetic data for your fine-tuning tasks. With MyMagic AI’s API, you can reduce the generation time by a factor of 5-10x compared to GPT-3.5.

5. Batch Transcription

MyMagic AI’s API supports different types of files, so it is also easy to batch transcribe many mp3 or mp4 files in your bucket.

Explore some of our use cases on our YouTube channel to see the API in action.

"Get started with MyMagic AI’s API!