Benchmarks
The time-to-first-word benchmark
How fast Natively answers, measured from the keypress to the first word on screen: in the app end to end, at the provider with reasoning off and on, and on screenshots. Every number on this page is copied from a result file you can download here or open in the repository.
What time to first word means
Natively is used during a live call. The person on the other side has just asked something, and the user needs the first few words now; the rest of the answer can keep arriving while they speak. So the figure that matters is not how long the whole answer takes but how long until its first word is on screen. This page calls that time to first word; the result files call it ttft_ms or the first answer token.
Two product decisions follow from it, and both can be read in the code. Answers stream: the overlay shows the first chunk as soon as it arrives and keeps writing. And the generator's reasoning mode is off: DeepSeek's current models reason by default, and reasoning tokens stream before the first answer token, so the user would wait through them. Natively sends thinking: { type: 'disabled' } on every DeepSeek request (electron/LLMHelper.ts, DEEPSEEK_NO_THINKING; electron/llm/deepseekModels.ts). The second table below puts a number on what that setting saves.
- Median time to the first word, in the app
- 889 ms
- 1,038 questions across nine modes, DeepSeek, 30 September 2026. p95: 1371 ms.
- Median first answer token at the provider, reasoning off
- 1.17 s
- 431 prompts through AgentRouter, 2 October 2026. The same prompts with low-effort reasoning: 2.49 s.
- Median first token on a screenshot, DeepSeek
- 832 ms
- 18 streamed calls, 17 September 2026. Gemini 3.8 Flash on the same calls: 1985 ms.
In the app, end to end
The answer-quality harness drives a real Natively instance over the Chrome DevTools Protocol. For a heard question, the other party's line is placed in the transcript and the hotkey path runs exactly as Cmd+Enter does; for a typed question, the overlay's own input path runs. The clock starts in the renderer immediately before the request and stops at the first model-generated chunk delivered back to it, so the figure includes the IPC hop, context assembly and retrieval, prompt building and the provider's own time to first token. The whole answer is timed to the final event, after any post-processing. Transcription is not included: the line is already in the transcript when the clock starts.
Natively 2.9.1 at commit bb41b031, a development instance (npm run dev:agent) on macOS (Darwin 27.0.0 arm64), Node v25.9.0, one DeepSeek key in the profile and the app's own routing choosing deepseek-flash. Three warm-up requests per surface are excluded.
| Run | Questions | Modes | First word p50 / p95 | Whole answer p50 / p95 | Model | When (UTC) |
|---|---|---|---|---|---|---|
| Final partitionaq-final-fix2 | 1,038 | 9 | 889 / 1371 ms | 1279 / 2046 ms | deepseek-flash | 30 Sep 2026, 05:08 to 06:14 |
| Holdout partitionaq-holdout-fix2 | 270 | 9 | 964 / 1648 ms | 1438 / 2607 ms | deepseek-flash | 30 Sep 2026, 05:24 to 06:10 |
Ten of the 1,038 final-partition rows carry no timing because no model call was made for them (served_by_llm is false in the row); the percentiles cover the rest. Source: the ALL rows of benchmarks/natively-answer-quality/astra/out/latency/existing-runs.md; run conditions from each run's run.json.
The final partition by mode
| Mode | Questions | First word p50 | First word p95 | Whole answer p50 / p95 |
|---|---|---|---|---|
| Call centercall-center | 116 | 807 ms | 1307 ms | 1062 / 1555 ms |
| Generalgeneral | 110 | 809 ms | 1303 ms | 1164 / 1620 ms |
| Lecturelecture | 116 | 824 ms | 1215 ms | 1382 / 1950 ms |
| Looking for worklooking-for-work | 116 | 1025 ms | 1588 ms | 1487 / 2019 ms |
| Recruitingrecruiting | 116 | 869 ms | 1270 ms | 1202 / 1867 ms |
| Salessales | 116 | 891 ms | 1396 ms | 1237 / 1935 ms |
| Seminarseminar | 116 | 997 ms | 1471 ms | 1475 / 2700 ms |
| Team meetingteam-meet | 116 | 882 ms | 1338 ms | 1204 / 1891 ms |
| Technical interviewtechnical-interview | 116 | 798 ms | 1649 ms | 1344 / 2354 ms |
Source: the aq-final-fix2 rows of benchmarks/natively-answer-quality/astra/out/latency/existing-runs.md. The raw rows, with each answer's text, are in benchmarks/natively-answer-quality/results/aq-final-fix2/natively_benchmark_full.jsonl.
At the provider: reasoning off and on
The recorded prompts of three earlier benchmark runs (the dev, holdout and final partitions, nine modes, three difficulty levels) were sent again, outside the app, to DeepSeek through AgentRouter: deepseek-v4-flash on the /v1/messages route, streaming, the route Natively's AgentRouter provider uses. Each prompt went once per arm, back to back in an order drawn per prompt: reasoning off (what the app sends), low effort, and the provider's default effort. The clock runs from the request to the first answer token; with reasoning on, reasoning tokens stream first and the user sees nothing until they stop.
432 prompts, 1,296 calls, one failed (a default-arm call with no answer text in its stream), so the tables compare the 431 prompts every arm answered. Measured 2 October 2026, 13:30 to 13:48 UTC. The file holds timings and token counts only; no answer text was stored.
| Reasoning | First answer token p50 / p90 / p95 | Whole answer p50 / p90 / p95 | First word later than 2 s / 3 s / 5 s | Output tokens, mean |
|---|---|---|---|---|
| Off (what the app sends)thinking: disabled | 1.17 / 1.84 / 2.23 s | 1.81 / 2.68 / 3.02 s | 8% / 1% / 0% | 95 |
| Low efforteffort: low | 2.49 / 4.75 / 5.76 s | 2.96 / 5.39 / 6.31 s | 69% / 38% / 8% | 405 |
| Default effortthinking: enabled | 3.69 / 6.32 / 8.04 s | 4.05 / 6.88 / 8.77 s | 92% / 66% / 24% | 660 |
With reasoning off, heard and typed questions behave alike: 1.17 s median for the 275 heard prompts, 1.13 s for the 156 typed ones. Source: the "All prompts" and "By how the question arrives" tables of reason-ar-text.report.md, computed from reason-ar-text.jsonl.
By difficulty
| Difficulty | Prompts | Off p50 / p95 | Low p50 / p95 | Default p50 / p95 |
|---|---|---|---|---|
| Easy | 144 | 1.14 / 2.09 s | 2.01 / 4.50 s | 2.99 / 5.67 s |
| Normal | 144 | 1.12 / 1.94 s | 2.65 / 5.25 s | 3.87 / 7.77 s |
| Hard | 143 | 1.24 / 2.54 s | 3.12 / 7.24 s | 4.46 / 9.84 s |
Source: the "By difficulty" table of the same report.
The same prompt, with reasoning minus without
| Reasoning | Prompts | First answer token later by p50 / p90 / p95 | Whole answer later by p50 / p90 | Output tokens, times |
|---|---|---|---|---|
| Low effort | 431 | 1.31 / 3.59 / 4.82 s | 1.15 / 3.45 s | 4.3 |
| Default effort | 431 | 2.42 / 5.17 / 6.81 s | 2.15 / 4.90 s | 6.9 |
Each prompt was answered by all three arms, so these are differences on the same prompt, not differences of medians. Source: the paired table of the same report.
Screenshot answers
When the user sends a screenshot, the answer comes from a vision model. This run timed the first visible character of a streamed answer, outside the app, with the request shape the server uses: three screenshots (a planted code bug, a small GitHub code view and a stock financials page), two questions each (a one-sentence lookup and a thorough explanation), three runs of each, so 18 streamed calls per provider. DeepSeek is deepseek-flash with thinking disabled; Gemini is gemini-3.8-flash at thinkingLevel: low, and its time counts text only, not thinking tokens. The file was generated on 17 September 2026 at 21:11 UTC.
| Provider | Calls | First token p50 / p90 | Min / max | Response headers p50 | Whole answer p50 | First token p50, short / long question |
|---|---|---|---|---|---|---|
| DeepSeekdeepseek-flash, thinking disabled | 18 | 832 / 969 ms | 460 / 1126 ms | 405 ms | 2151 ms | 832 / 780 ms |
| Gemini 3.8 Flashgemini-3.8-flash, thinkingLevel low | 18 | 1985 / 3853 ms | 1413 / 4990 ms | 1984 ms | 2666 ms | 1985 / 1960 ms |
Source: the summary block of vision-ttft.json, a copy of benchmark/vision/results/ttft.json; the 36 rows are in the same file. Model names and settings as recorded in benchmark/vision/REPORT.md.
Natively keeps Gemini 3.8 Flash as the screenshot primary and DeepSeek as the fallback, because the same benchmark's accuracy ladder found DeepSeek misreading a small code line on an IDE screenshot in 4 of 5 attempts while Gemini had no failed check in 490; the verdict and the counts are in benchmark/vision/REPORT.md. A separate probe on 18 September 2026 turned DeepSeek's thinking on at its lowest effort for the same streamed calls: the first token moved from 969 ms to 4542 ms at the median, 1106 ms to 6925 ms at p90, over 8 streams per setting (benchmark/vision/results/thinking-low.json).
Meeting notes, after the call
This is a different path and a different quantity, kept here so the page does not overstate what streams. Post-meeting notes are not a live answer: the pipeline makes about 11 model calls per summary (one extraction call per transcript chunk, two polish calls, a follow-up draft and a title), none of them streamed, so time to first word does not apply and the report records it as not applicable. These are end-to-end times for the whole summary with the production configuration (DeepSeek, the deepseek-v4-flash id, which DeepSeek serves with V4.1 Flash, thinking disabled), measured on 16 September 2026 between 20:04 and 20:48 UTC from one machine in India: 11 transcripts (9 written for the benchmark and 2 real public lectures), 3 runs each.
| Transcript length | Runs | Whole summary p50 / p90 |
|---|---|---|
| Short (a 7 to 9 minute meeting) | 6 | 24.5 / 26.2 s |
| Medium | 6 | 26.7 / 28.7 s |
| Long | 15 | 50.2 / 58.5 s |
| Very long (a 90-minute lecture) | 6 | 64.5 / 72.1 s |
| All | 33 | 41.3 / 62.4 s |
Source: the latency tables of benchmark/reports/REPORT.md and benchmark/reports/narrative/latency.md, computed from pipeline_ms in benchmark/results/aggregates.json; conditions from benchmark/reports/narrative/method.md. That report's own conclusion is that no configuration it tried meets its 8 to 15 second target and that the lever is the pipeline's output volume, not the model.
What this does not show
- One machine and one network per run. The in-app runs are a development build on one Mac with the prompt recorder on; the notes run is one machine in India; the screenshot and reasoning runs do not record the machine. Your figures depend on your network and on the provider's load at that hour.
- One time window each. Provider latency moves through the day, and none of these runs was repeated on another day.
- No competitor measured side by side. No file in the repository times another product, so this page makes no comparison with one.
- The provider-only runs skip the app's own work (the IPC hop, retrieval, prompt building); the in-app runs include it. They are different quantities and should not be subtracted from each other.
- Transcription is excluded everywhere: the clock starts once the question is already text in the transcript. Speech recognition, local or cloud, adds its own time before that.
- The in-app runs are Natively 2.9.1 at one commit, and the AgentRouter run reaches DeepSeek through a third-party router rather than DeepSeek's own endpoint. Model ids are the ones recorded at the time; DeepSeek serves the deepseek-v4-flash id with V4.1 Flash.
- The questions are authored benchmark items, not real users' calls, and the in-app hotkey path receives each line as final text.
- The prompt captures behind the 431-prompt reasoning run (system prompts and recorded user messages) are not in the repository, so that run can be repeated with new prompts but not replayed byte for byte.
- Percentiles are the nearest-rank values each harness prints. Groups of 6 runs or 18 calls carry wide error bars; no confidence intervals are given for latency.
Method and data
Each measurement below names the harness that ran it, the file the numbers were copied from, and what you can download. Natively is source-available, free for personal use (LICENSE); the harnesses live in the repository. The files published on this page hold timings, token counts and ids only: no keys, no answer text, no personal data.
How to rerun
The command lines are the ones in each file's header. The in-app run needs an isolated Natively instance and a DeepSeek key in the repository's .env; the reasoning run needs an AgentRouter key; the screenshot run needs DeepSeek and Gemini keys. The reasoning harness reads the prompts of earlier runs from their natively_benchmark_wire.jsonl captures, which the in-app run writes.
# In the app: an isolated instance, then a partition against it cd <checkout> && NATIVELY_E2E=1 NATIVELY_PROMPT_DEBUG=1 npm run dev:agent NATIVELY_ROOT=<checkout> node benchmarks/natively-answer-quality/run.mjs --partition final --run-id <id> # Two runs of the same items, paired per item node tools/latency-paired.mjs results/<runA> results/<runB> # Two instances at once, the same questions in ABBA order node latency.mjs --a <rootA> --b <rootB> --label-a base --label-b fix --per-mode 10 # Reasoning off, low and default, through AgentRouter; then the tables node tools/latency-reasoning.mjs --name <out> --arms off,low,default --key-var AGENTROUTER_API_KEY_1 node tools/latency-reasoning.mjs --name <out> --report # Screenshots: three runs per cell, as in the published file cd benchmark/vision/harness && node ttft.mjs --runs 3 --shots ../shots --out ../results/ttft.json
The published reasoning run sampled 36 prompts per mode and 90 each for the lecture and technical-interview modes (one lecture prompt dropped out with the failed call), split evenly across easy, normal and hard; the per-mode counts are in the report. The harness picks its sample by a hash of the item id, so the same flags on the same captures give the same prompts.
