Temporal is a durable queueing and work execution layer that has the promise of helping developers “shift their focus to business logic rather than infrastructure concerns and create applications that are inherently scalable and maintainable.” It has become a core component of day to day development and operations at LlamaIndex, with users ranging from our Parse and Extract product teams to our billing and support teams relying on Temporal to drive our business-critical operations at scale. In this blog post, we’ll cover what Temporal offers to our product and developers, the problems that we were experiencing operating complex pipelines at scale with traditional queuing systems (in our case RabbitMQ), and how Temporal allowed us to scale to tens of millions of pages processed per day to power our most recent Batch API release.

An introduction to document parsing

LlamaParse handles 130+ file types and turns all of that into markdown, text, or JSON. Behind the tidy API is a distributed system juggling CPUs, GPUs, model providers, agentic orchestrators, and media storage.

Standardizing a document should be straightforward. You send a PDF, parse it, and get back markdown or JSON. Easy. Except a document is almost never one unit of work. One file can be thousands of pages, and every page is its own little problem: clean digital text here, a scanned table there, handwriting, a chart, dense schematics. Some pages only need text extraction. Others need OCR, or a vision model, or a few passes before the structure makes sense.

The legacy workflow system

At first, RabbitMQ was used to buffer and distribute worker capacity and control message priority. When we began, the workflow looked something like this

This is nice and simple, but it starts to fall over once it meets the real complexity involved in generically processing any document, and over time our system started to resemble a workflow system stapled together out of databases, queues, and wherever else we could cache or hold intermediate state. If a 1200-page dense manufacturing catalog shows up, should it be one giant task? What happens if the machine cycles while processing? The reality of document processing is that not all documents are created equal, and there is complex and resource intensive work around identifying that work, splitting it up, and assigning compute resources in such a way that the work can be done on time. Our pipelines updated to meet the real demands of our core competency (document processing), but also simultaneously began to evolve bespoke workflows out of the tools we had immediately available.

More and more of our code became dedicated to managing this problem ad-hoc. Things like:

1. Without durable state, we built status-tracking and fairness primitives over queue messages, locks, retry counters, and status columns.
2. Adversarial documents may occasionally OOM machines, leading to silently killed jobs on our queue - forcing more advanced heartbeat mechanism management with RabbitMQ to track and report retries to users.
3. Fairness and backpressure so no single user can consume provider rate limits or system resources, and to pause or slow work when those limits have been reached

We introduced a lot of layered complexity into our RabbitMQ orchestration system in an effort to keep parsing execution reliable. Without the right primitives to easily orchestrate this, we often had to over-provision our system to make sure that work could be done quickly without stalling out the system. Rather than continue trying to build on technical debt, we began to look elsewhere for our maintainable workflow orchestration requirements, which is where Temporal came in.

Enter Temporal

Temporal is an open source runtime for durable function execution. You organize business logic into a deterministic series of steps called Workflows. Workflows and their steps (Activities) run in processes called Workers, and the Temporal server durably stores execution state, so a Workflow that fails can be retried from where it left off. This is great, but the reality is that it is not a silver bullet. It will not magically solve your problems by hooking up to it, but it does provide the primitives and tools to build durable coordinated distributed systems elegantly.

Here's how we solved the critical problems that were so hard to build on top of our legacy system: A) Durability (most out of the box, still relies on external storage), B) fairness and concurrency, and C) efficient resource usage across our fleet

Every running Workflow execution has a globally unique identifier and can send signals to other Workflows, or wait for them. That is enough to coordinate access to a shared resource, and Temporal documents the pattern in its guide to locking shared resources. Here's a small example Workflow in Python that hands out permits.

@dataclass
class PermitSlotInput:
    resource: str
    slot: str
    lease_seconds: float

@workflow.defn(name="PermitSlotWorkflow")
class PermitSlotWorkflow:
    def __init__(self) -> None:
        self._released = False

    @workflow.signal(name="release")
    def release(self) -> None:
        self._released = True

    @workflow.run
    async def run(self, input: PermitSlotInput) -> str:
        try:
            await workflow.wait_condition(
                lambda: self._released,
                timeout=timedelta(seconds=input.lease_seconds),
            )
            return "released"
        except TimeoutError:
            return "lease_expired"

A caller acquires by starting it as a child Workflow with an ID like permit:gpu-pool:2 , and releases by sending it a release  Signal.

Note that in this Workflow:

• The Workflow doesn't set its own ID. The caller sets it at start time, which is what makes the ID a lock rather than a name. permit:tenant-migration:acct-1234:0  protects one tenant's migration; permit:gpu-pool:2  is one of several slots in a pool.
• Acquiring is a single atomic operation against Temporal's state. The start either succeeds, in which case you hold the permit, or it fails with WorkflowAlreadyStartedError , in which case someone else does. There's no read-then-write race to lose.
• The lock and the work it gates run on the same platform. No database, no cache layer, no central limiter service.
• Nothing has to sweep up after a crashed holder. If the parent Workflow closes for any reason, ParentClosePolicy.TERMINATE  kills the permit immediately. If the parent is alive but has gone silent, the lease timeout fires on its own.
• The two exit paths return different values. A permit that ends as lease_expired  instead of released  is a recovery, and you can see which happened by looking at the Workflow in the Temporal UI.
• Holding a lock is a durable wait, not a held connection or a TTL you keep renewing. A Worker restart doesn't drop it.

This maps onto our own concurrency problems, so we built a small library on top of the primitive, with one change: the permits live inside a single coordinator Workflow per resource rather than one Workflow per slot.

Parallelism with Temporal

Today every concurrency-limited job runs behind a slot it got from a Temporal Workflow. Here's what that looks like.

• The semaphore is a Workflow, and its ID is the name of the resource it protects. SemaphoreWorkflow  is a long-lived coordinator keyed by that name:

def semaphore_workflow_id(key: str, project_id: str) -> str:
    """Build a deterministic workflow ID for the semaphore."""
    return f"semaphore:{key}:{project_id}"

semaphore:job_concurrency:<project>  limits a project's in-flight jobs. Callers never look the semaphore up and never create it explicitly. They signal-with-start it, and Temporal's ID uniqueness collapses every concurrent starter onto the same execution:

await client.start_workflow(
    WORKFLOW_NAME,
    arg=None,
    id=semaphore_workflow_id(key, project_id),
    task_queue=task_queue,
    id_conflict_policy=WorkflowIDConflictPolicy.USE_EXISTING,
    start_signal=ACQUIRE_SIGNAL_NAME,
    start_signal_args=[acquire_signal],
)

One call means "create the semaphore if nobody has yet, and enqueue me on it either way." There is no bootstrap path to get wrong and nothing to provision ahead of first use.

• Acquiring is a signal out and a signal back. The caller sends its own Workflow ID in the acquire signal, then parks on a wait_condition  until the semaphore signals it back.


@workflow_decorator(name=JOB_WORKFLOW_NAME, queue=TaskQueueTypeValues.JOBS)
class GatedJobWorkflow(JobWorkflow[JobInput, None]):
    @workflow.signal(name=GRANTED_SIGNAL_NAME)
    async def on_semaphore_granted(self, signal: GrantedSignal) -> None:
        """Handle grant notification from the semaphore workflow."""
        self._semaphore_granted= True

    async def run(self, job: JobInput) -> None:
        acquired, grant_timeout= await acquire_semaphore_compat(
            JOB_CONCURRENCY_KEY, job.project_id, "job-acquire-owns-settings"
        )
        wait_duration: float | None = None
        try:
            if acquired:
                wait_duration= await wait_for_semaphore_grant(
                    lambda: self._semaphore_granted, job.id, job.project_id, grant_timeout
                )
            return await super().run(job)
        finally:
            if acquired:
                await release_semaphore(JOB_CONCURRENCY_KEY, job.project_id, wait_duration)

• Signaling runs in the caller's own Worker. The semaphore Workflows get a dedicated concurrency-limiter  Task Queue and Worker pool, because permits are cheap and latency-sensitive, and putting them behind application backpressure creates the obvious feedback loop: delayed acquires slow jobs, which extend slot holds, which deepen the backlog.

Conclusion

Distributed systems are still hard, and having the right primitives in place saves engineering teams from maintaining technical debt, and helps roll new products out faster. Temporal is a good coherent foundation for building on, and it allowed us to focus more on document processing.

After switching to Temporal, we were able to consolidate and remove caching layers, additional microservices, split queues, and message metadata that had been built up to support scaling:

1. per-worker limits
2. provider rate limits
3. separate capacity per model
4. workload priorities
5. backpressure
6. batching
7. fairness between small and large documents
8. fairness between smaller and larger customers

Interested in learning more or working with us?

• sales@runllama.ai
• https://www.llamaindex.ai/careers