Before You Buy a GPU: Test Your Team’s Local AI Workload
By Greg Nowak. Last updated 2026-09-10.
Before approving a GPU purchase, ask your team to run one useful task on the hardware you already have. Document classification or standard reply drafting would give you a manageable starting point. Can the team use the results? Do they arrive quickly enough? What happens when requests overlap?
Ollama’s June 5, 2026 release gives you a reason to check again if you have tried this before. It broadened GGUF model compatibility and enabled Vulkan by default, extending GPU acceleration to more hardware, including AMD and Intel devices. The NVIDIA performance improvement Ollama reported came from a specific combination of model, quantization, and GPU. Your workload needs its own test. Ollama’s release article sets out the changes and test conditions.
The purchasing question is straightforward: can the equipment you own handle the work you need? A useful pilot should leave you with evidence about output quality, capacity, and unresolved limitations, so you can make that decision.
Start with a short acceptance brief. For document classification, list the permitted categories, decide where ambiguous documents should go, and identify mistakes you cannot accept. For reply drafting, specify which facts must survive, which statements must stay out, and how much editing is reasonable. Someone who understands the work needs to judge the outputs.
Choose representative inputs the team is permitted to use. Include everyday examples alongside long documents, missing information, and difficult cases that still fall within the task. Hold back a separate set for the final evaluation. Otherwise, you risk judging prompt changes only on examples you have already used to refine them.
Measure the manual process too. For reply drafting, the useful comparison is how long it takes to reach an approved reply, including review and correction. For classification, check both the category assigned and whether uncertain cases reach a reviewer. Agree on the quality and timing thresholds before comparing hardware.
Once the Linux pilot is configured, check where processing actually happens. Ollama’s hardware documentation lists GPU and driver requirements, and Vulkan on Linux may need additional components. It also describes an NVIDIA suspend/resume issue that can prevent GPU discovery and cause processing to fall back to the CPU. A slow run deserves investigation before it becomes a reason to replace the GPU. Check the machine against Ollama’s hardware support documentation.
Keep a configuration record: machine specification, operating system, driver, Ollama version, model file, quantization, and request settings. If restarting or suspending the machine is part of normal use, check the execution path again afterwards. Another person should be able to repeat the test from your notes.
Memory requirements depend on the workload you define. Ollama documents that context length and parallel request processing affect memory use, while available memory determines whether multiple models can be loaded. Requests can queue when resources are unavailable. The ollama ps command shows whether a loaded model is on the GPU, CPU, or both; Ollama’s FAQ explains these behaviours and the settings that control them.
Set the intended context length explicitly. Test inputs should include the full instructions, supporting text, and conversation history you expect to send. Measure peak GPU and system memory with the intended parallel request setting. Report capacity against that exact configuration: a named model, a defined input range, and a measured request pattern. A successful short request tells you little about whether a long document will fit.
A controlled benchmark can help explain performance differences between configurations or machines. The llama-bench documentation covers separate prompt processing and text generation tests, combined tests, adjustable GPU offloading, repeated measurements, and CSV and JSON exports. Its measurements exclude tokenization and sampling time.
Use benchmark input and output lengths that resemble the task. Keep the model and settings consistent when comparing machines, and save the exported results with the configuration record. These measurements help diagnose performance. You still need to run the actual workflow through Ollama to establish application response time and output quality.
In that application test, record the time to first output and the time to the complete result, along with failures and corrections. Compare the first request after model loading with later requests while the model stays resident. Ollama supports preloading and configurable model residency, so include those choices in any timing report.
Test the demand you expect from the team. Begin with one request, add overlapping requests, then reproduce a plausible busy period. Ten people with access do not necessarily mean ten simultaneous requests. Ask the business owner what wait is acceptable for this workflow, and show the slower responses alongside typical timings. Rejected requests belong in the results too.
| What the pilot shows | What to do next | What it means for a purchase |
|---|---|---|
| Quality and response time meet the targets at expected demand. | Document the tested capacity and assign an operating owner. | Continue on existing hardware. |
| Quality is acceptable, but overlapping requests take too long. | Compare scheduling, parallel request settings, and candidate hardware. | Require a measured capacity improvement before buying. |
| Long inputs exceed the tested memory budget. | Test an input limit the team can accept or revise the configuration. | Specify memory for the workload the team has agreed to use. |
| Outputs need too much correction. | Revise the task, prompt, or model, then evaluate again. | Resolve quality before approving hardware spend. |
| GPU use changes unexpectedly between runs. | Check drivers, device discovery, and service configuration. | Resolve the inconsistency and repeat the measurements. |
The operating estimate needs the same care as the performance test. Use measured consumption and explicit assumptions to account for electricity, machine availability, administration, updates, and staff review time. For a purchase option, spread the proposed hardware cost over an agreed planning period. Separate initial setup from recurring work and state the expected monthly task volume.
Cost per accepted result is a useful measure to include: divide the allocated monthly operating cost, including review, by the number of outputs that meet the acceptance criteria. Show what happens at lower and higher usage. Assess the benefit using the team’s measured completion time, and label any future savings that remain assumptions.
Greg could help put this into practice through a local AI feasibility sprint focused on one internal task. The scope could include configuring the Linux pilot, selecting representative inputs, agreeing quality criteria, and measuring memory use, response times, concurrent requests, and operating costs. The deliverable should be a short decision report with settings and results that someone else can reproduce.
That report should recommend a specific next step: keep the existing hardware, narrow the workload, test a specified upgrade, or pause the rollout. If a GPU purchase is recommended, it should identify the measured constraint the upgrade would address and explain how the candidate machine will be validated. Bring one internal task and details of the hardware you have to a feasibility discussion with Greg.
Related on GrN.dk
- An AI Voice Agent Needs More Than a Phone Number and a Realtime Model
- From Supplier PDFs to Product Data: Where AI Needs a Second Check
- Your AI Gateway Can Name the User. Decide What That Log Is For
Need help with this kind of work?
Discuss your local AI feasibility pilot Get in touch with Greg.