Why running AI on your laptop is harder than it sounds — and what NVIDIA just did about it

NVIDIA unveiled RTX Spark at Computex this week: a chip designed to run frontier-class AI models directly on a personal computer, no cloud required. The announcement sent AMD, Intel, and Qualcomm shares lower. Here is what the chip actually has to solve, why it has been so hard to solve, and what you should watch before deciding whether this one is different.

The problem starts with weight. When you send a message to a cloud AI assistant, a large language model running on a data-centre server reads your prompt and generates a reply. That model is defined by its parameters — billions of numerical values, called weights, that encode what the model has learned. A modest 7-billion-parameter model stored at standard precision (FP16, meaning each weight takes 16 bits of memory) occupies roughly 14 gigabytes. A 70-billion-parameter model occupies ten times that.

To generate output, the chip running the model has to read every single one of those weights, on every single forward pass through the model — every token it produces. The speed at which it can do that is called memory bandwidth, measured in gigabytes per second. This is the central hardware constraint in AI inference.

What consumer hardware currently offers. High-end desktop GPUs — an RTX 4090, for example — have around 1,000 GB/s of memory bandwidth. That is fast enough to run a 7B model at a few dozen tokens per second: usable, if not instant. Laptop hardware is where it gets painful. A thin-and-light laptop with LPDDR5X RAM (the fast consumer memory standard) gets roughly 77 GB/s. At that bandwidth, a 14 GB model generates tokens slowly enough to feel broken. The maths is straightforward: bandwidth divided by model size gives you a rough tokens-per-second ceiling before compute even enters the picture.

This is why on-device AI has mostly meant small, constrained models on phones and laptops — not the frontier-class models that make cloud AI feel impressive.

What RTX Spark is trying to do. NVIDIA's answer is to bring a GPU-class chip — specifically a Blackwell-architecture GPU, the same generation as its current data-centre products — into a power envelope small enough for a laptop or compact desktop. The chip pairs that GPU with an efficient CPU and unified memory on a single package, which means the GPU and CPU share the same memory pool rather than copying data between separate chips. That matters because it removes a significant bottleneck: the GPU does not have to wait for data to arrive from a separate memory bank.

NVIDIA also leans heavily on FP4 inference. FP4 is a low-precision number format — four bits instead of the sixteen in FP16, or thirty-two in FP32. Using FP4 to store and compute weights shrinks the model in memory by a factor of four compared to FP16. Our 7B model drops from 14 GB to around 3.5 GB. At 77 GB/s, that changes the arithmetic meaningfully. At higher bandwidth — which NVIDIA claims for RTX Spark, though detailed specs were not published before Computex — the improvement compounds further.

Quantization (the process of reducing weight precision from FP16 to FP4 or similar) does carry a quality cost. The model loses some accuracy. How much depends on the model, the quantization method, and the task. For many everyday workloads — coding autocomplete, summarisation, local transcription — the quality loss is acceptable. For tasks requiring precise reasoning over long contexts, it matters more. NVIDIA has not published which model families RTX Spark is tuned for, or what quality benchmarks look like at FP4 on its architecture.

How NVIDIA's approach differs from the existing on-device silicon. Before RTX Spark, the on-device AI silicon landscape had three main players, all using a different architectural approach: dedicated NPUs (Neural Processing Units — fixed-function accelerators built specifically for transformer operations). Qualcomm's Hexagon NPU in the Snapdragon X Elite, Intel's NPU in Meteor Lake, and Apple's Neural Engine all fall into this category.

An NPU is purpose-built for the specific matrix operations that transformer models run constantly. It is efficient: optimised silicon doing the one thing it was designed to do. But it is relatively inflexible. The software stack has to be built for it specifically — usually via intermediary formats like ONNX or Core ML or Qualcomm's QNN — and developers have to actively port their models to target the NPU.

NVIDIA's GPU is programmable. It runs CUDA, NVIDIA's computing platform, which already has a vast developer ecosystem built around it. Developers who have been running models on NVIDIA cloud GPUs can, in principle, run the same software stack on RTX Spark without significant porting effort. TensorRT, NVIDIA's inference optimisation library, works on Blackwell. That is the ecosystem bet NVIDIA is making: not just better silicon, but a toolchain developers already know.

Where the metaphor breaks. The CUDA advantage is real, but it has limits worth naming. CUDA runs on RTX Spark, but the chip is still power-constrained in a way a data-centre A100 is not. A thin-and-light laptop has a thermal design power (the maximum heat the system can dissipate) of roughly 15 to 45 watts. Running a large model at full throughput requires sustained power draw that thermal envelopes may not support. NVIDIA has not yet published RTX Spark's TDP or full memory specs. Until those are public, claims about "frontier-class inference" on a laptop chip should be read with that constraint in mind.

The developer ecosystem argument also has a track record to reckon with. Qualcomm launched the Snapdragon X Elite in 2024 with a 45 TOPS (trillion operations per second) NPU capability. Mainstream LLM applications did not migrate to local inference at scale within the first year. Apple's Neural Engine runs across billions of devices — and still only runs Apple-curated models via Core ML. Third-party developers cannot freely target it for arbitrary models. On-device AI silicon has consistently outpaced on-device AI software adoption.

Intel's simultaneous move tells you something. The same week NVIDIA announced RTX Spark, Intel's CTO Sachin Katti was briefing the Financial Times on "Crescent Island" — a new inference GPU targeting data-centre deployments, framed around a tokens-as-a-service commercial model. Intel is betting that the inference volume stays in the data centre, not on the client. That is not a concession; it is a competing thesis. The two bets are mutually incompatible at scale, and it is not obvious which one is right.

Why it matters if it works. Local inference changes two things that cloud inference cannot fix. First, latency: a round-trip to a data-centre API takes roughly 200 to 800 milliseconds for a typical LLM call, depending on load and geography. Local inference on capable hardware can get well below 100 milliseconds. For real-time use cases — coding autocomplete that feels instant, live transcription, voice interfaces — that gap is the difference between a useful tool and a frustrating one. Second, privacy: data that never leaves the device cannot be logged, processed, or retained by a third party. For enterprise use cases involving sensitive documents, that is not a nice-to-have.

RTX Spark is designed in alongside Dell and Lenovo as a platform play, not a discrete add-in card. OEM design-in matters because it means the thermal and power systems are built around the chip from the start — which is how Apple Silicon got its efficiency profile. That is the right approach. Whether NVIDIA can match Apple's level of vertical integration on a Windows platform, with its much messier software stack, is a different question.

What to watch. NVIDIA's full Computex technical brief should publish within days of this piece. The numbers to look for: RTX Spark's memory bandwidth, TDP, and memory capacity ceiling. Those three figures will tell you whether the 14-GB-model problem is actually solved, or softened. After that, watch developer adoption: does Ollama support RTX Spark natively? Does LM Studio? Does the CUDA path just work, or does it require model-specific tuning? The silicon announcement is the easy part. The software story takes longer to tell.

Footnotes and links

Further reading

• Andrej Karpathy, "LLM inference scaling" notes — practical walkthrough of the memory bandwidth problem in inference, referenced widely in the ML community.
• Hugging Face model cards for Llama 3 8B — shows real-world memory requirements for a current-generation small model.
• Qualcomm Snapdragon X Elite product page — for reference on the NPU architecture RTX Spark is competing against.
• NVIDIA Blackwell architecture whitepaper — technical foundation for the GPU design RTX Spark is built on (link to be added once Computex technical brief publishes).

¶