By Greg Nowak. Updated 10 October 2026.
Before approving a GPU purchase, run one useful team workflow on the hardware you already own. Drafting standard replies or classifying incoming documents gives you something concrete to test: usable results, acceptable waiting times, and the amount of checking people still need to do.
A local AI pilot should answer a purchasing question: what prevents this workflow from working well today, and would better hardware actually fix it? Here is how to collect enough evidence to make that decision.
Start with a task someone can judge
Choose a repeatable task with a clear owner. An operations team might route documents into agreed categories. An agency might draft replies from approved account information. Keep the first pilot narrow enough that a person who knows the work can recognise a good result.
Write a short acceptance brief before choosing a model:
- Quality: which facts must survive, which mistakes are unacceptable, and when should a person take over?
- Speed: how long can someone reasonably wait, including during a busy period?
- Review: how much correction is acceptable before the task loses its value?
- Inputs: what document lengths, formats, and missing information must the workflow handle?
Use representative material the team is permitted to process. Include routine examples, long inputs, ambiguous cases, and incomplete information. Keep a separate evaluation set untouched while refining prompts. Measure the manual process too: compare the time needed to reach an approved result, including corrections.
Check what your existing machine is actually doing
For a Linux pilot, record the CPU, system RAM, GPU and VRAM, operating system, driver, Ollama version, exact model, quantization, and request settings. Quantization is the model’s numerical precision setting; changing it creates a different test configuration.
Check the machine against Ollama’s current hardware requirements. Vulkan is enabled by default when its backend is installed, but Linux can still need additional driver components. Ollama also documents an NVIDIA suspend/resume issue that can cause CPU fallback. Investigate an unexpectedly slow run before treating it as evidence that the GPU needs replacing.
After loading the model, run ollama ps. Its processor column shows whether the model is loaded on the GPU, CPU, or both. Repeat this check after restarting or suspending the machine if either is part of normal use.
If this is intended to be a strictly local pilot, select a local model and disable Ollama’s cloud features with OLLAMA_NO_CLOUD=1 on the server, then restart it. Check the surrounding application’s data path too. Ollama’s FAQ explains these settings and the processor display.
Test long inputs and overlapping requests
Set the intended context length explicitly. Context includes instructions, supporting material, and conversation history; allow room for the answer as well. In an interactive ollama run session, /set parameter num_ctx 8192 sets an example 8,192-token context. Choose a value that suits your workflow and model.
Longer contexts and parallel requests increase memory requirements. Record OLLAMA_NUM_PARALLEL, measure peak GPU and system memory, and test the longest permitted inputs. Requests can queue, and an overloaded server can return HTTP 503 errors. The concurrency guidance explains the controls.
Begin with one request, then reproduce plausible overlaps and a busy period. Ten people with access do not necessarily mean ten simultaneous requests. Test the arrival pattern the team expects, including other work sharing the machine.
Record typical waits, the slower responses, failures, and rejected requests. Keep failed cases in the report. A capacity claim should identify the model, input range, configuration, and request pattern it applies to.
Measure the workflow, then diagnose performance
Measure time to first useful output and time to the complete result from the application. Separate the first request after model loading from later requests while it stays resident. Record any preloading and keep_alive choices.
For diagnosis, Ollama’s generation API returns loading, prompt-processing, and generation durations in nanoseconds, plus token counts. Keep these alongside application timings: the team also waits for document preparation, queueing, and the interface.
If you need a controlled hardware comparison, llama-bench provides separate prompt-processing and text-generation tests, combined tests, repeated measurements, GPU offloading controls, and CSV or JSON output. Match the tested lengths to your task and keep the model and settings consistent. Its measurements exclude tokenization and sampling time, so validate the complete workflow through Ollama afterwards.
Let the findings decide the purchase
| Pilot finding | Next step | Buying implication |
|---|---|---|
| Quality and timing pass at expected demand. | Document capacity and assign an operating owner. | Use existing hardware. |
| Quality passes; overlapping requests are too slow. | Compare scheduling, parallel settings, and candidate hardware. | Require a measured capacity improvement. |
| Long inputs exceed available memory. | Test acceptable input limits or a revised configuration. | Size memory for the agreed workload. |
| Results need excessive correction. | Revisit the task, prompt, or model. | Resolve quality before buying. |
| GPU use changes unexpectedly. | Check drivers, discovery, and service configuration. | Repeat measurements after fixing the cause. |
Include the cost of getting an accepted result
Estimate electricity, administration, updates, staff review, and any effect on other work using the machine. Separate initial setup from recurring costs. For a purchase option, allocate hardware cost over an agreed planning period and state the expected monthly volume.
Cost per accepted result = allocated monthly operating cost, including review, divided by accepted outputs. Show lower and higher usage scenarios. Compare the benefit with measured completion time, and label projected savings as assumptions.
Turn the pilot into a decision you can act on
I can help scope and run a local AI feasibility pilot around one internal workflow. The deliverable can be a concise decision report covering quality, memory, response times, operating costs, and reproducible settings, with a recommendation to continue, narrow the task, test an upgrade, or pause.
Bring one task and the specifications of your existing hardware. Get in touch with Greg to discuss the pilot and what evidence you need before committing to a purchase.
Related on GrN.dk
- Logistics Optimization in 2026: Fix the Flow Before You Buy More Tech
- Your AI Workflow Needs an Acceptance Test Before It Meets Customers
- Nginx 1.30 Changed Upstream Connections: What to Test Before You Upgrade
Need help with this kind of work?
Discuss your local AI pilot with Greg Get in touch with Greg.
