- 20 September 2026
- Mario Florea
Edge AI gets more interesting when a model does more than answer prompts. It plans, calls tools, and keeps working toward a goal. That is the appeal of agentic workloads, and it is why we explored running OpenCode through llama.cpp with Qwen 3.5 on Imagination E-Series.
This blog explains the approach we took, why the software path was practical, what we optimized along the way, and why E-Series GPUs are a good fit for this class of workload.
Headline result: through this profiling-led optimization process, we achieved a 20x performance multiplier in 6 weeks, moving from initial bring-up to a much more responsive OpenCode + llama.cpp + Qwen 3.5 path on E-Series.
Why this combination made sense
OpenCode gives us the agent loop. llama.cpp gives us a practical inference runtime with an established ecosystem, flexible model support, and a pathway to efficient local execution. Qwen 3.5 is a modern model family that is relevant for coding and assistant-style use cases. E-Series provides a programmable GPU architecture with matrix acceleration that can handle graphics, AI, and compute workloads on the same platform.
That combination matters because agentic AI at the edge is not just about raw TOPS. It is about getting real software stacks to work together, then making them responsive enough to feel useful. In our case, the goal was to run a popular model through a realistic runtime and connect it to an agent-style workflow rather than optimize a synthetic benchmark in isolation.
Our focus was practical deployment: OpenCode on top, llama.cpp underneath, Qwen 3.5 as the model, and E-Series as the compute engine.
Why our software stack made this straightforward
One of the biggest advantages of a programmable GPU approach is software flexibility. We did not need a custom, one-off demo stack. Instead, we built on familiar components and connected them through standard interfaces.
|
What helped
|
Why it matters
|
- Run Qwen 3.5 through llama.cpp using the OpenCL backend.
- Connect llama.cpp into OpenCode to exercise an agentic workflow.
- Profile execution to understand where time was going.
- Optimize the highest-impact kernels and data movement patterns.
- Remeasure and iterate.
This is exactly the kind of workflow a mature GPU software stack should enable. You can start from open software, use familiar APIs, and then progressively add architecture-aware optimizations where they matter most.
What we learned during bring-up
Getting an LLM running is only the first step. Making it efficient requires understanding the model, the runtime, and the hardware together.
Early on, we looked at the topology and behavior of Qwen 3.5 in llama.cpp, including quantization choices and where execution was falling back to less optimized paths. We also isolated specific parts of the workload, such as decode layers, so we could iterate more quickly without always running a full end-to-end test.
That helped us separate three different questions:
- Is the model running correctly?
- Which operations dominate time in prefill and decode?
- Which of those operations are worth optimizing first?
The answer was clear: matrix-heavy work dominates the important paths, with attention and supporting operations also contributing depending on phase and shape. That is exactly where E-Series can help most.
Key observation: prefill and decode behave differently. Prefill tends to be dominated by larger matrix operations, while decode often becomes highly sensitive to GEMV-style work, memory behavior, and runtime overheads.
Our optimization process
We treated optimization as an engineering loop, not a guessing exercise. Profiling results were mapped against utilisation expectations so we could prioritize work with the highest likely impact.
To find the right ideas, we used a mix of application-level timing, runtime instrumentation, and PowerVR-specific profiling. PVRTune - Imagination Developers was especially useful because it let us look beyond a single tokens-per-second number and understand how the GPU was being used while llama.cpp moved through prefill and decode. By correlating command submissions, kernel durations, hardware utilisation, memory activity, and idle gaps, we could tell whether a candidate bottleneck was genuinely compute-bound, limited by bandwidth, or simply waiting on orchestration between kernels.
That workflow helped turn profiling into a shortlist of concrete experiments. PVRTune was primarily used to identify runtime overhead and CPU-side gaps, highlighting cases where launch and scheduling overhead were becoming significant. We paired these findings with kernel-level counters and a working target model, which provided more detailed visibility into the behaviour and performance of individual kernels.
The key was using the tools together. PVRTune gave us the timeline and utilisation view, while lower-level kernel analysis and controlled llama.cpp runs helped confirm root cause. That prevented us from over-optimizing visually obvious but low-impact kernels, and kept the effort focused on the paths that mattered most for user-visible responsiveness.
The main areas we focused on included:
- Matmul improvements: bringing in better CMM-enabled and tuned matrix kernels for relevant shapes.
- Flash attention work: integrating and improving OpenCL flash-attention paths, including ongoing work around softmax and buffering behavior.
- Conversion overhead: reducing the cost of f32↔f16 style convert paths that were contributing more than expected.
- Conv and normalization kernels: improving kernels such as conv1D and RMS norm where they showed measurable impact.
- Memory/layout handling: experimenting with transposes, image-based paths, sub-buffers, and tiling choices to improve effective throughput.
- Batching experiments: testing micro-batch choices to better match the workload to the hardware.
Some changes delivered quick wins. Others taught us what not to prioritize. That is just as important in a time-bounded optimization effort.
Performance gain: the most important outcome was the multiplier: by iterating quickly on the real application path, we were able to deliver roughly a 20x gain in 6 weeks, rather than just isolated microbenchmark improvements.
Examples of optimisation wins we saw include:
- conv1D tuning produced a measurable token-rate uplift and saved cycles in the relevant path.
- Flash-attention work significantly reduced kernel cycle counts in internal measurements as implementation matured.
- Additional work on convert and GEMV paths showed promising improvements, especially for decode-oriented behavior.
- Several low-impact ideas were deliberately deprioritized so effort could stay focused on hotspots.
Why E-Series MMA is a strong fit for agentic AI
Agentic workloads are a great example of why flexible matrix acceleration matters. These systems are not just running one giant offline batch. They are interactive, often latency-sensitive, and made up of repeated matrix operations mixed with control flow, runtime orchestration, and tool use.
E-Series is well suited to that environment for three reasons.
1. Matrix acceleration where LLMs need it most
Large language models spend the vast majority of execution time and memory bandwidth in dense matrix math. E-Series includes dedicated MMA capabilities designed to accelerate these compute patterns natively across both half-precision (FP16, BF16) and reduced-precision formats (INT8, FP8). Operating on low-precision data types brings a fundamental low-power advantage to edge deployments: shrinking weight and activation footprints drastically cuts memory traffic across the cache and memory hierarchy, the primary driver of energy consumption in edge inference. For continuous, multi-step agentic loops, this efficiency ensures high token throughput and sustained performance without exceeding tight thermal and power budgets.
2. A programmable path instead of a narrow one
Agentic applications evolve quickly. Models change, quantization schemes change, runtimes change, and workflows change. A programmable GPU plus OpenCL software stack gives developers room to adapt without waiting for a fixed-function pipeline to catch up.
3. Shared graphics, AI, and compute foundation
At the edge, developers often need more than one workload on the same device. E-Series is designed around a unified architecture for graphics, AI, and compute, which makes it a natural platform for systems that need local intelligence alongside other interactive workloads.
Why this matters for agents: agentic workloads benefit from a platform that can deliver good matrix performance, low-friction software portability, and the flexibility to support changing models and toolchains over time.
What this means for developers
From a developer point of view, the bigger story is not just that we ran OpenCode with Qwen 3.5 on E-Series. It is that we did it using an accessible stack:
- an open agent framework,
- a widely used local inference runtime,
- standard GPU compute APIs, and
- targeted architecture-aware optimization where it mattered.
That lowers the barrier to experimentation. Teams can start with known software, bring up a real model quickly, and then tune performance progressively instead of needing a fully bespoke stack on day one.
Where optimization effort went furthes
Our profiling reinforced a useful rule for LLM deployment on GPUs: focus on the dominant math first, then remove the avoidable overhead around it.
In practice, that meant:
- improving matrix kernels before chasing minor operators,
- treating attention as a major opportunity once baseline support was in place,
- watching conversion and transpose costs closely, and
- measuring end-to-end impact rather than trusting isolated kernel gains alone.
That last point matters. A kernel can look faster in isolation but still fail to improve user-visible tokens per second if memory behavior or orchestration costs erase the benefit. The work on OpenCode through llama.cpp was valuable precisely because it kept us grounded in the full application path.
Closing thought
Running OpenCode through llama.cpp with Qwen 3.5 on E-Series was a useful demonstration of what a flexible GPU AI software stack should look like in practice. We were able to combine open software components, bring the workload up on E-Series through OpenCL, profile real bottlenecks, and then improve performance through targeted optimization of the hottest kernels and data paths.
For externally facing audiences, the takeaway is simple: E-Series is not just capable of running modern local AI workloads. It is well matched to them. Its programmable software path makes integration approachable, its MMA-backed compute architecture makes optimization worthwhile for the matrix-heavy core of agentic inference, and this work showed that focused tuning can produce a 20x performance multiplier in a short time.
As edge AI moves from single-shot prompts to persistent, tool-using assistants, that combination of flexibility and acceleration becomes increasingly important. We also plan to upstream the optimisations we develop back into llama.cpp, so the wider community can benefit from the performance work and build on it across platforms.
