Featured image: AI-generated conceptual illustration. Sources checked 26 October 2026.
When an agent feels slow, changing the model is an obvious response. It may also be the wrong one. The delay could come from a queue, a document search, a series of dependent API calls or a human approval. Measure the path from the user’s request to the verified result before deciding what to optimize.
Latency is a property of the workflow. Model generation is one component, and the user’s experience includes waiting before and after it. A useful performance review separates those components so the team can improve speed without weakening checks or producing less reliable work.
Draw a timeline for one real task
Consider a hypothetical service assistant that prepares a return authorization. Its timeline includes request intake, customer verification, order lookup, policy retrieval, eligibility interpretation, approval and creation of the return record. Some steps are software operations; others depend on an external system or person.
Instrument start and finish events at each boundary. Record queue time separately from execution time. If a tool takes a long time, determine whether the delay occurs in the connector, the remote application or repeated retries. A single total-duration number cannot answer those questions.
The OpenAI Agents SDK tracing guide describes spans that can help represent execution steps. Use that mechanism to build an understandable timeline, while minimizing sensitive payloads in the trace. Timing information often remains useful without storing the full customer conversation.
Find serial work that does not need to be serial
An order lookup may have to finish before eligibility can be checked. A general policy retrieval and a warehouse-location lookup may be independent once the relevant identifiers are known. Draw those dependencies explicitly rather than assuming that every step must wait for the previous one.
Parallelize only work whose inputs and authority are already established. Launching several writes at once can create race conditions or make cancellation difficult. A faster run is not an improvement if it changes records in an unsafe order or duplicates actions.
The OpenAI latency-optimization guide includes fewer requests, selective parallelism and reducing unnecessary generation among its principles. Treat those as options to test against the work graph, not instructions to remove every sequential check.
Remove calls that do not add a decision
An agent may ask a model to classify a request, then ask another model to restate the classification, then ask a third to choose the obvious tool. Sometimes those boundaries serve a real purpose. Sometimes they are leftovers from a prototype. Inspect what each call contributes.
If a deterministic rule can select the next step from validated data, consider using it. If two model operations share the same evidence and can be evaluated together, test whether combining them preserves quality. Avoid changing several components at once, or it becomes difficult to identify what helped and what broke.
Tool design can remove hidden latency too. A search tool that returns stable identifiers and the needed summary fields can avoid repeated follow-up calls. A vague error that triggers several speculative retries can make a small integration problem look like slow model reasoning.
Make the output fit the task
A user requesting a short eligibility answer may not need a long essay before the application proceeds. Define the output format around the decision and supporting evidence. For internal intermediate steps, avoid generating verbose prose that another component immediately parses back into a few fields.
Do not shorten outputs by removing essential qualifications. A return decision may need the policy exception or missing-evidence note to remain correct. Measure whether a compact format preserves the information reviewers need rather than optimizing word count in isolation.
If the workflow produces a document, distinguish the time to an initial useful update from the time to a complete verified artifact. Streaming or progress messages can improve the experience, but they should not claim that a record changed before the destination confirms it.
Treat human waiting as its own category
Approval time may dominate an end-to-end process. A faster model will not resolve an absent reviewer. Improve the review card, routing and notification process, and make the pending state clear. Measure active system work separately from time awaiting a person.
Do not remove required approval merely to improve a latency chart. If the review burden is excessive, revisit the risk classification and permission design with the responsible owners. A policy change is a governance decision, not a performance tweak an engineer should make silently.
Test a model change against the same workload
A different model can change speed, tool selection, output length and error rates. Run the same representative cases and inspect both latency and correctness. Include difficult cases and failures, not only the quick requests that make the new configuration look attractive.
The OpenAI evaluation guide emphasizes task-specific evaluation. Apply that discipline to performance changes: the system should still meet its acceptance criteria after an optimization. A lower average duration can conceal a higher rate of incomplete or unsafe outcomes.
- Compare end-to-end duration and the duration of each major stage.
- Separate active execution from queue and human-approval waiting.
- Inspect repeated calls, retries and unnecessary output.
- Check slower tail cases rather than relying only on the average.
- Confirm that authorization and verification steps remain intact.
- Record quality regressions alongside any speed improvement.
Publish an honest service expectation
Different tasks may deserve different expectations. A simple status lookup should not be described like an open-ended investigation. Let the interface communicate when work is immediate, queued or waiting on approval, and give the user a way to cancel or check status.
The best optimization is the one that removes an actual bottleneck while preserving the service promise. Start with the timeline, make one reasoned change and retest. That approach is more reliable than blaming the model for every delay or trading away essential controls for a faster demonstration.
Sources and further reading
OpenAI Agents SDK: Tracing. Execution spans and timing observation mechanisms; checked October 2026.
OpenAI: Latency optimization. Primary workflow optimization principles; checked October 2026.
OpenAI: Evaluation best practices. Task-specific regression evaluation; checked October 2026.
Leave a Reply