Horizon Performance on Android: Benchmarked Across 60+ Devices

Download the full benchmark report (PDF)

What we tested

We ran Horizon in two quantisation configurations — Int4-Q4 and Int8-Q8 — across 60 devices spanning High, Mid, and Low tier hardware. The device list covers phones and tablets from Google, Samsung, OnePlus, Sony, Motorola, and others, released between 2020 and 2025.

For each device and configuration, we measured:

• Decode speed: how fast the model generates tokens (the main driver of perceived responsiveness.
• Time to first token (TTFT): how long before the model starts responding
• Prefill speed: how fast the model processes the input prompt
• Peak memory usage: RAM consumed at peak
• Initialization time: cold start and steady-state load time

Int4 vs Int8: the core trade-off

The headline finding is straightforward: Int4-Q4 is the right configuration for on-device deployment in almost every scenario.

Across all 60 devices, Q4 delivers 1.62× faster decode speed on average — the metric users feel most directly. It also uses 34% less memory (621 MB saved on average) and loads 40% faster on cold start. The only areas where Q8 has an edge are prefill throughput (18% faster) and time to first token (12% faster), which matters more for batch or offline use cases than interactive ones.

The most striking finding: no Q8 device in our test set reaches 40 tokens per second. Q4 hits that mark on 25% of tested devices. If you need responsive generation on-device, Q4 is the only viable path.

Performance by device tier

Not all Android hardware is equal, and the benchmark reflects that clearly.

High-tier devices (2022–2024 flagships) deliver the best experience by a significant margin. Q4 decode averages 42.9 tok/s on this tier, with the top performers — Samsung Galaxy S24, Z Fold6, and Z Flip6, all on Snapdragon 8 Gen 3 — exceeding 63 tok/s. Time to first token on these devices drops below 3 seconds, making Horizon feel fast and responsive.

Mid-tier devices are capable but noticeably slower, averaging 21.2 tok/s on Q4. Devices like the Pixel 6, Motorola Edge 30, and Samsung A-series land in this range — usable for many applications, but generation speed will be perceptible to users.

Low-tier devices average 12.7 tok/s on Q4. Several devices in this group — particularly those with Unisoc chips or older Snapdragon 660-class processors — fall below 10 tok/s, which is marginal for interactive use. We'd recommend setting Q4 as a minimum baseline and gating the feature on devices below this threshold.

The processor story

The chip inside a device matters more than the brand or price tier. Qualcomm's Snapdragon 8 Gen 2 and Gen 3 are in a class of their own, averaging 59.5 tok/s on Q4 — nearly 52% faster than the next group. Google's Tensor G3 and G4 chips punch well above their tier at ~39 tok/s, competitive with Snapdragon 8 Gen 1. ARM Mali-based chips (Exynos, MediaTek) trail behind, with MediaTek Dimensity averaging 16.5 tok/s and Exynos 20.7 tok/s.

The GPU also matters: Qualcomm Adreno GPUs average 36.7 tok/s versus ARM Mali at 26.3 tok/s — a 39% gap driven entirely by the memory bandwidth and compute architecture difference.

How performance has evolved over time

Looking at average Q4 decode speed by device release year tells an encouraging story: hardware has gotten 3.8× faster between 2020 and 2024. A 2020 device averages 10.2 tok/s; a 2024 device averages 38.5 tok/s. Memory footprint has also stayed flat or slightly decreased since 2023, suggesting on-device model efficiency is improving alongside raw hardware gains.

The practical implication: 2022 is roughly the cutoff for a comfortable experience. Devices released before that year struggle to sustain 20 tok/s on Q4, and the experience degrades noticeably.

Deployment recommendations

Based on these results, here's how we think about Horizon deployment on Android:

• Use Int4-Q4 by default. The decode speed and memory advantages are decisive for interactive use.
• Minimum viable hardware: 2022+ mid-range devices with Snapdragon 7xx, Tensor G2, or Exynos 1380. These reliably hit ~20 tok/s on Q4.
• Best experience: 2023+ devices with Snapdragon 8 Gen 2 or Tensor G3+. These deliver 50+ tok/s and sub-3s TTFT.
• Gate or degrade gracefully on devices below ~10 tok/s (Unisoc, Snapdragon 662/665/680, pre-2021 budget hardware).
• RAM floor: Q4 requires approximately 960 MB–1.2 GB peak. Devices with less than 4 GB total RAM need careful testing.
  

What's next

This benchmark covers the current Horizon model. We're actively working on further quantisation and pruning techniques to push performance on mid and low-tier devices. We'll publish updated results as those improvements land.
In the meantime, the full benchmark data — including per-device breakdowns, memory analysis, and processor family comparisons — is available in the report linked below.

Download the full benchmark report (PDF)