Skip to content
KRASTOR

Insights

Self-Hosted LLM for Business: What It Takes and What It Costs

How a 10 to 250 person business can run a self-hosted LLM: hardware, real cost figures, security basics and how to size it before you buy anything.

November 17, 2026

A self-hosted LLM is an open-weight language model that runs on hardware your business controls, so prompts and documents never leave your network. For a company with 10 to 250 employees, it is a realistic project when the model is sized to a specific job, the hardware is matched to your volume and someone owns security and upkeep after launch. We build and run these systems, and this guide covers what the work involves, so you can decide whether it fits your operation.

TL;DR

  • A self-hosted LLM is an open model (Llama, Qwen, Mistral, DeepSeek) running on GPUs you control, served by software you operate.
  • Small businesses use small and mid-size models. A research cost analysis found small models on a roughly $2,000 consumer GPU can pay back in months at modest volume.
  • The model is the easy part. Authentication, network isolation, monitoring and updates decide whether it is safe and reliable.
  • Size the project by workload (documents, volume, uptime needs), not by what is trendy.
  • Our Private AI and on-premise deployment work starts with a diagnostic of your workflows, then builds and runs the system.

Table of Contents

What "self-hosted" actually means

Self-hosted means an open-weight model running on GPU infrastructure your team controls, served by inference software your team operates. Open-weight means the model's trained parameters are published, so you can download and run them yourself instead of paying a provider per request.

The terms get mixed up, so here is the vocabulary:

  • On-premise: the hardware is in your building or your own server room. It describes where the machine sits.
  • Private: the model is used exclusively by one organization. That can be on your premises or in dedicated hosting.
  • Air-gapped: no network path to the outside at all. This is a stricter requirement than on-premise.
  • Cloud API: someone else runs the model and you send them your data.

An on-premises LLM is typically four layers: GPU hardware, locally held model weights, serving software such as vLLM, and the application your staff actually uses. Most of the effort goes into the last two, because a model that nobody can reach from the tools they already use does not save anyone time.

Why a business would run its own model

Three reasons come up again and again.

Data stays inside. A self-managed model keeps sensitive data under direct control, which is why it appeals to finance, healthcare and government. Smaller businesses meet the same pressure through customer contracts: a law-adjacent firm, a medical-adjacent supplier, a contractor handling client financials, a manufacturer with proprietary designs. We cover the practical side in our guide to AI data privacy for small business.

Predictable cost at steady volume. Instead of a per-token bill that grows with usage, you carry hardware and operating costs that stay flat. If your staff run the same document tasks all day, flat costs are easier to budget.

It works through outages and at the edge. A model on your own network keeps answering when the internet link is down, which matters for plants, job sites and venues. Our edge deployment case shows what that looks like in practice.

Some businesses care about all three. Others care about only one, and that one reason is enough to shape the design.

Does the math work for a business your size?

The cost headlines you see online are about data-center clusters with eight or a hundred H100 GPUs. That is not your problem. The relevant research is on small and mid-size deployments.

An academic cost-benefit analysis of on-premise LLMs looked at this directly. For organizations under 10 million tokens a month (a token is roughly three-quarters of a word, so that is several million words of text), small open models such as Qwen3-30B on a consumer GPU like the roughly $2,000 RTX 5090 break even in 0.3 to 3 months. For medium organizations at 10 to 50 million tokens a month, it describes a sweet spot with $15,000 to $30,000 dual-A100 setups, breaking even in 3.8 to 34 months.

A separate January 2026 study of consumer Blackwell GPUs reports electricity costs of $0.001 to $0.04 per million tokens for self-hosted inference, and hardware paying for itself in under four months at moderate volume (30 million tokens a day). It also found a 4-bit floating point format (NVFP4) delivered 1.6 times the throughput of the standard format with 41% less energy and only a 2 to 4% quality loss.

A worked example

Take a 60-person distributor whose team summarizes supplier emails, drafts quotes and searches internal documents. Say that adds up to a few million tokens a month. By the research's sizing, that is the small-model, consumer-GPU range, where hardware in the low thousands of dollars is the main outlay. The budget that matters then is not the GPU. It is the work of connecting the model to your quoting, email and document systems, and the time to keep it running. That is a scoping question, which is exactly what a diagnostic answers before anyone buys hardware.

The figures above are published research results for specific hardware and workloads, not a quote for your business. Token volume, model size and uptime needs move the numbers, and GPU prices change, so treat them as a way to size the conversation.

What the costs include

When you total a self-hosted project, count these lines:

  1. Hardware, including a spare plan, since GPUs fail and replacements cost real money.
  2. Power and cooling, which grow with GPU count.
  3. Setup and integration with your CRM, accounting, email and document stores.
  4. Ongoing operation: updates, monitoring, security patches and model upgrades.
  5. Training so staff use it for the right jobs.

Lines 3 to 5 are where most do-it-yourself projects stall. A model running on a server in a closet, with no owner, no monitoring and no connection to daily work, produces little.

Hardware and skills in plain terms

The main hardware number is GPU memory. A model has to fit in it, along with working space for each conversation. Bigger models need more memory and more GPUs, and quantization (storing the model's numbers at lower precision) lets a model fit on smaller cards with a modest quality trade-off, as the Blackwell study above shows.

Two tiers cover most small and mid-size business work:

TierTypical hardware (from the research)Typical jobs
Small modelOne consumer GPU such as an RTX 5090, roughly $2,000Summaries, drafting, classification, internal Q&A over documents
Mid-size modelDual-A100 server, $15,000 to $30,000Many concurrent users, longer documents, heavier workloads

Skills matter as much as hardware. Someone has to install the serving software, put authentication in front of it, manage updates and watch for failures. For most owner-led companies, that person does not exist on staff, and hiring one for a single server is hard to justify.

Security: the part people skip

Running a model on your own network does not make it secure by default. Local tooling is often built for experimentation. One production deployment guide notes that Ollama's API has no authentication out of the box, and recommends:

  • API-key authentication at a gateway in front of the model
  • TLS encryption terminated at a reverse proxy such as Nginx
  • Network segmentation, with the model on an internal network and its port never exposed to the wider host
  • Health checks that restart failed services automatically

There is also a visibility problem. Palo Alto Networks points out that a self-managed model can be installed by any developer on a virtual machine, leaving IT without built-in visibility into which models are running or who can reach them. The answer is an inventory and a single, owned deployment, rather than several informal ones.

Keep sensitive traffic in-house by policy

Most businesses do not need every request to stay local. A practical pattern is one OpenAI-compatible endpoint (vLLM, SGLang and Ollama all speak that API) behind a gateway, with routing rules based on how data is classified. INFRO recommends treating failover as policy: traffic that must stay in-house should return an error on failure rather than silently falling back to an outside service. That one rule prevents the most common privacy leak in hybrid setups.

Choosing a model for your workflows

Start from the task, not the leaderboard. Public benchmark rankings measure general reasoning, coding or math, and your work is probably none of those. A model that tops a chart may be oversized for invoice-matching and slower than a small model tuned to your documents.

A workable selection process:

  1. List the jobs. Lead follow-up drafts, proposal first drafts, invoice coding, scheduling replies, document search.
  2. Collect real samples of each job from your own files.
  3. Run candidate models on those samples and have the person who does the work judge the output.
  4. Measure speed and concurrency at the number of people who will use it at the same time.
  5. Pick the smallest model that passes. Smaller models cost less to run and fit smaller hardware.

This is also why the model choice belongs after the workflow mapping. The right model for an HVAC dispatcher's scheduling assistant differs from the right one for a brewery's distributor invoicing.

What local AI can run in a business like yours

Local models handle text-heavy, repetitive work well. In the businesses we serve, the common candidates are:

  • Lead follow-up: drafting replies to inquiries and reminding staff of open quotes
  • Proposals and estimates: first drafts from your templates and past jobs
  • Invoicing: reading and matching invoices to purchase orders
  • Scheduling: answering and organizing booking requests
  • Internal knowledge: answering staff questions from manuals, procedures and contracts
  • Reporting: summarizing activity from several systems in one view

These run best when the AI is wired into the systems your team already uses, not offered as a separate chat window. For a manufacturer, see our AI implementation guide for manufacturing. For a broader look at costs, The $200 Stack and our piece on AI spend per employee show how to measure what you spend now.

A sensible order of operations

  1. Map where time and money leak. Identify the manual tasks that cost the most hours.
  2. Decide what data must stay local. Contracts, client records and regulated information come first.
  3. Estimate volume. Count documents and requests per month, then convert to tokens.
  4. Size hardware and pick a model against your own samples.
  5. Build in security from day one: gateway, authentication, segmentation, logging.
  6. Connect to real workflows so the output lands in your CRM, email or accounting tool.
  7. Monitor and improve after launch, with cost and savings tracked monthly.

Our full explanation of what self-hosted AI takes and when it pays goes deeper on steps 3 and 4. Regulated businesses should also read our piece on intelligence sovereignty for regulated industries.

Where we fit

We integrate AI systems into businesses with 10 to 250 employees across the Southeast, from western North Carolina and Upstate South Carolina to the Charlotte metro. Each engagement begins with a diagnostic, which you can book as a free 30-minute call. From there, The Blueprint is a paid assessment starting from $1,500 one-off, followed by a build and, if you want it, ongoing operation.

For on-premise work specifically, our Private AI service covers deploying models on your own hardware. Our approach has no vendor lock-in and no tool markups, we keep monitoring the system after the build, and you get monthly financial reports on operating costs and savings.

Our perspective

Most of the self-hosted AI conversation is about models and GPUs. In our experience with owner-led businesses, the more useful question is which costly task you want gone. We start with the workflow, not the hardware. A diagnostic finds the inefficiencies that cost the most, and only then do we decide whether a private model on your own equipment is the right tool for them. When it is, we build it into the way your people already work, so it is not another login nobody opens. The build is the beginning, not the end. A private model needs monitoring, updates and security attention, and we stay engaged after launch, with monthly reports showing what the system costs and what it saves. That is how a business owner can see whether the system is earning its place. If keeping your data on your own premises matters to you, you should not have to become an infrastructure company to get it. Greyson

FAQ

Is it worth self-hosting an LLM?

It is worth it when data must stay on your infrastructure, a contract or regulator requires it, or steady volume keeps the hardware busy. Published research shows small models on consumer GPUs paying back within months for businesses under 10 million tokens a month. A diagnostic of your workloads tells you which side of that line you are on.

How much does it cost to host an LLM for a business?

It depends on model size and volume. The research above puts a small-model setup around a $2,000 consumer GPU and a mid-size dual-A100 server at $15,000 to $30,000, before power, integration and upkeep. Our Blueprint, from $1,500 one-off, scopes the real figure for your business.

Can a small business run its own AI model?

Yes. Small open models run on a single GPU and handle drafting, summarizing, classification and document search. The harder part is connecting the model to your systems and keeping it secure and monitored, which is the work we take on.

What is the difference between on-premise AI and private AI?

On-premise describes where the hardware is. Private describes exclusive use by a single organization, whether that hardware is in your office or in dedicated hosting. Air-gapped is stricter again: no network path outside.

Sources

Recommended

The Reading List

Not ready to book? Start here.

The Architecture Manifesto, our AI project failure data, and one or two things worth reading, sent when there’s something worth sending. No cadence, no filler.

No spam. Unsubscribe anytime.

Engagement starts here

Start with the assessment.

A few questions about how work moves through your business. Your score and the one gap to fix first are emailed to you.

Not limited to what's listed. Every engagement starts by assessing what your business actually needs, and we build whatever it requires.