Skip to content
Theorem Weekly TW markTWTheoremWeekly

The evidence behind the week in machine learning.

Model updates

How much VRAM and RAM does a PC need to run local AI models?

Engadget reports that while 16GB systems can run small models, achieving useful speed requires GPUs with high memory bandwidth.

Key points

  • Running 8B parameter models requires roughly 8GB of VRAM for decent speed.
  • Quantized models in Ollama range from 4.9GB for Llama 3.1 8B to 43GB for Llama 3.3 70B.
  • Nvidia's RTX 5090 generated 205 tokens per second from OpenAI's gpt-oss-20b in LMSYS testing.
How much VRAM and RAM does a PC need to run local AI models?
Depth
8B parameterRunning 8B parameter models requires roughly 8GB of VRAM for decent speed · reported
8BQuantized models in Ollama range from 4.9GB for Llama 3.1 8B to 43GB for Llama 3.3 70B · reported
70BQuantized models in Ollama range from 4.9GB for Llama 3.1 8B to 43GB for Llama 3.3 70B · reported
1 outlet · newest source 165h ago

Engadget reported hardware requirements for running local AI models on personal computers. The outlet explained that performance depends entirely on machine power, specifically memory capacity and bandwidth.You can run AI on your computer, but you need enough memory and a fast graphics card.Local inference performance is memory-bound, requiring high-bandwidth VRAM for acceptable token generation rates. Users can run local chatbots without internet or subscriptions, but response speed varies by hardware.

Evidence: reportedWhat hardware does your PC need to run a local AI model?

aifleet-engadget

What was released or claimed

The report outlined memory needs for various model sizes. It stated that everyday PCs with 16GB of RAM can run small models with seven to eight billion parameters. For faster performance, users need graphics cards with at least 8GB of VRAM. Larger models require 32GB of system memory and 16GB of VRAM.

Evidence: reportedWhat hardware does your PC need to run a local AI model?

aifleet-engadget

Evidence: reportedWhat hardware does your PC need to run a local AI model?

aifleet-engadget

Evidence: reportedWhat hardware does your PC need to run a local AI model?

aifleet-engadget

Who it affects

The guidance targets users of tools like Ollama and LM Studio. These individuals want to run local chatbots without data leaving their devices. The report notes that prices for high-memory graphics cards are currently high, affecting budget-conscious users.

What is still unknown

The material does not specify exact latency for models running solely on system RAM versus GPU memory. It also does not detail how quantization affects model accuracy, only that it reduces file size to roughly a quarter of the original.

So what

Users must balance model size against available memory. A 16GB PC can run OpenAI's gpt-oss-20b, but it consumes nearly all available memory. Speed depends more on memory bandwidth than raw processing power, making GPU VRAM critical for performance.

Evidence: reportedWhat hardware does your PC need to run a local AI model?

aifleet-engadget

What the paper shows

The analysis tests hardware constraints for local inference. Results show that memory bandwidth is the primary bottleneck, described by Nvidia engineers as "memory-bound." Limitations include high costs for 16GB+ VRAM cards and the trade-off between model size and speed on standard PCs.

Evidence: reportedWhat hardware does your PC need to run a local AI model?

aifleet-engadget

Why it matters

  • Researchers testing local models must account for memory bandwidth limits, not just parameter count.
  • Quantization allows larger models to fit in limited RAM, but speed suffers without GPU VRAM.
  • High-end GPU prices create a barrier for running frontier-like models locally.
  • Mixture-of-experts designs like gpt-oss-20b offer speed benefits but still require substantial memory.

Topics: local AI, hardware requirements, Ollama, quantization, VRAM

Updated

Then / now / next

Then

Users relied on cloud-based AI services due to hardware limitations.

Now

Local tools allow running models on PCs, but hardware requirements are specific.

Next

Monitor GPU pricing and new model optimizations for lower hardware thresholds.

Story timeline

  1. Then Users relied on cloud-based AI services due to hardware limitations.
  2. 09 Nov 2026 What hardware does your PC need to run a local AI model?
  3. Now Local tools allow running models on PCs, but hardware requirements are specific.
  4. Next Monitor GPU pricing and new model optimizations for lower hardware thresholds.

The strongest doubt

Counterpoint from the sources

Local models run slower than cloud services and require significant upfront hardware investment.

Numbers in context

8Breported70BreportedCircle area is proportional to the magnitude; labels retain signs. Figures may describe different measures. Read their contexts above.

Try it

Run Llama 3.1 8B on Ollama

Good result: The model loads in roughly 4.9GB and generates text at readable speeds on compatible hardware.

Ask the article

Local keyword search of this article; no generated answers.

Did this article give you the evidence you needed?

No vote saved on this device.
Your choice stays on this device. No public tally.

Questions readers ask

How much RAM do I need for a 7B model?

Ollama's general rule of thumb is 8GB of RAM for a 7B model.

Does quantization affect model speed?

Quantization reduces file size, allowing models to fit in memory, but speed depends on memory bandwidth.

What is a good token generation speed?

Five to 10 tokens per second keeps pace with most people's reading speed.

Connected stories

This storyDo LLM Parameter Counts Actually Predict Model Intelligence?Do LLM Parameter Counts ActuallyHow does Claris Harbor leverage Thunderbolt 5 to cluster Macs for private AI model serving?How does Claris Harbor leverage

Reading receipt

  • Running 8B parameter models requires roughly 8GB of VRAM for decent speed.
  • Quantized models in Ollama range from 4.9GB for Llama 3.1 8B to 43GB for Llama 3.3 70B.
  • Nvidia's RTX 5090 generated 205 tokens per second from OpenAI's gpt-oss-20b in LMSYS testing.

Still open: How does quantization impact the accuracy of smaller models?

Still open: What are the exact costs of used GPUs with 16GB VRAM?

Time spent: 0:00 · Depth: Full

Sources

  1. Engadget

Related stories

Background reading