Self-hosted AI means you run the model, and the software around it, on infrastructure you control: a workstation, a server in your office, or GPU machines in your own cloud account. Your prompts and documents stay inside that boundary, you pick the model version, and you carry the sizing, security and upkeep. For a business with 10 to 250 employees, it fits when sensitive data must stay put or when steady, heavy usage makes per-token bills painful. For most other uses, a managed service is the quicker start.
This guide walks through the parts of a self-hosted stack, what hardware you need, which models you can run, what it costs, and how to decide. We build and run these systems for businesses across the Southeast, so we also cover what the work looks like after the install.
TL;DR
- Self-hosted AI is a stack (model, runtime, application layer), not a single product. You own every layer you host, including security and updates.
- GPU memory (VRAM) decides which models you can run. Plan compute, storage and networking before you buy anything.
- Self-hosting turns variable per-token bills into fixed costs. It pays off with high, steady volume, and the savings disappear when hardware sits idle.
- Self-host the layers that touch sensitive data and use managed services for the rest. A hybrid setup is common and sensible.
- Start with one low-risk workflow, measure it, then expand.
Table of Contents
- What self-hosted AI actually means
- The three layers of a self-hosted stack
- Hardware: what you need to run a model in-house
- Which models can you self-host?
- What self-hosted AI costs
- Security and privacy: what you take on
- Self-hosted, managed or hybrid: how to choose
- A sensible rollout in six steps
- Who runs it after launch?
- Work with Krastor on self-hosted AI
- FAQ
- Sources
What self-hosted AI actually means
The plain definition: you operate the inference server (the program that loads a model and answers prompts), the interface people use, and usually the retrieval and workflow tools that connect the model to your documents and systems. One vendor guide frames the benefit as control over where prompts go, which model version runs and how cost scales, with the trade-off that you own sizing, security and upkeep.
That definition covers a range of setups:
- A workstation or small server in your office. One machine with a capable GPU serves a small team. Common for a first pilot.
- Your own servers or a server room. More capacity and redundancy, plus someone responsible for them.
- GPU instances in your own cloud account. You still control the environment and the data path, but you rent the hardware instead of buying it.
People also say "local AI," "private AI" and "on-premise AI." The terms overlap, and vendors define them differently. What matters for your decision is a simple question: who can see your data, and who is responsible when something breaks? If you want the on-premises version of this question handled end to end, our private AI and on-premise deployment page describes how we approach it.
The three layers of a self-hosted stack
Every working setup has the same three parts, whatever the brand names on top.
- Model weights. The downloaded file that holds what the model learned. Sizes range from a few gigabytes to hundreds of gigabytes, according to the EPRI AI Model Playbook.
- A runtime. Software that loads the weights and processes prompts. Ollama, vLLM and llama.cpp are common open options.
- An application layer. The chat window, the form, the automation or the connection into your CRM, accounting package or scheduling tool. This is where employees actually feel the value.
The same EPRI page describes a fuller architecture: a user interface, orchestration and control logic, context and data tools, the model service itself, and governance and assurance layers on top. For a 40-person HVAC company, that translates into something concrete. A dispatcher types a question or a new lead arrives by email. Orchestration pulls the right customer record, the model drafts a reply or a quote summary, and a log records what happened. The model is the smallest part of that picture. Most of the effort goes into the connections and the controls.
That is why a model installed on a server, with nothing connected to it, rarely changes how a business runs. The value shows up when the model sits inside a workflow your team already follows.
Hardware: what you need to run a model in-house
The EPRI playbook tells teams to plan three things: compute (GPUs or other accelerators for responsive inference), storage (model files plus logs and curated data) and networking (secure, low-latency internal connections for interactive apps).
VRAM is the number to watch
GPU memory is the main limiter. Puget Systems' hardware primer cites typical consumer and prosumer cards at roughly 8 GB to 48 GB of VRAM, and VRAM capacity sets which model sizes fit for GPU inference. A model that does not fit in memory either will not run or will spill to the processor.
That spill matters. The same primer says CPU inference, which uses system RAM, can be roughly 10x to 100x slower than GPU inference for the same model. Fine for an overnight batch job, painful for an employee waiting on a chat reply.
A rough sizing rule
A hardware calculator offers two handy rules of thumb. With 4-bit quantization (a compression technique that shrinks models), expect about 0.5 bytes per weight. And aim for system RAM of about 1.5 times your VRAM, with a 16 GB floor, so the model loads and the operating system stays responsive.
Worked example: a model with 20 billion weights at 4-bit needs roughly 10 GB for the weights alone (20 billion x 0.5 bytes). Add working memory for the conversation and you will want a card with comfortably more than that, plus at least 16 GB of system RAM. Treat this as a starting estimate, not a purchase order. Real needs depend on the model, context length and how many people use it at once.
Containers and scaling
The EPRI page points to containerization with Docker for portability and Kubernetes for scaling and reliability. A small business with one workflow rarely needs Kubernetes on day one. A single well-configured server running containers is often enough, as long as you plan how you would grow.
If you prefer to rent rather than buy, cloud providers offer accelerated compute you control. AWS's published infrastructure guide covers choosing an accelerator and running it at a cost that makes sense, and it mentions both AWS Trainium and NVIDIA GPUs.
Which models can you self-host?
The open-weight model field changes quickly, so we avoid naming a "best" one in an article meant to be read for a year. What stays stable is the approach: choose the model for the workload, then check its license.
One concrete, primary-source example: OpenAI's gpt-oss-120b and gpt-oss-20b are open-weight reasoning models released under the Apache 2.0 license. They run on infrastructure you control, are not served through the OpenAI API, and work with common stacks such as vLLM, Ollama and llama.cpp. The smaller one is the kind of model that a single well-equipped machine can realistically serve.
Tasks that suit self-hosted models in a small business:
- Summarizing call notes, emails and long documents
- Drafting first-pass replies to leads and customer questions
- Answering staff questions from your own manuals, price lists and SOPs (retrieval-augmented generation)
- Classifying incoming requests in batches, such as sorting service tickets or invoices
- Support chatbots grounded in your own content
Read the license before you build on it
Open-weight does not always mean unrestricted. Licenses can limit commercial use, redistribution, attribution, acceptable use and the number of users. Verify the terms on the model publisher's own page before you commit a workflow to it. Apache 2.0, as used for the gpt-oss models above, is a permissive license, but each model has its own terms.
What self-hosted AI costs
The cost structure is the biggest difference from a managed service. The EPRI playbook notes that self-hosting front-loads spending on compute, supporting equipment and IT labor. If usage lands below your projection, hardware sits underused. If it lands above, the setup is undersized and you either expand or live with slower responses. Managed providers carry the opposite risk: per-token bills that grow with usage.
The three buckets
| Cost bucket | What it includes | Behaves like |
|---|---|---|
| Hardware or GPU rental | Purchased GPU server, or monthly cloud GPU instances | Fixed, whether busy or idle |
| Power and cooling | Electricity, cooling, space | Fixed, scales with machine count |
| People | Setup, patching, monitoring, model updates | Fixed, often underestimated |
Electricity is the smallest piece but easy to quantify. PromptCost calculates that an RTX 4090 running continuously at 450 W and $0.12 per kWh costs about $39 per month in power, with cooling and maintenance adding to that. The hardware purchase and the staff hours dominate the real number.
The break-even question
A useful test comes from one vendor explainer: divide your total monthly self-hosting cost (hardware or rental, power, engineering time) by the tokens you actually serve. If that figure falls below the hosted price for a model that meets your quality bar, self-hosting pays. Owned or reserved GPUs cost the same when idle, so low utilization erases the savings.
SpecPicks' cost analysis puts numbers on the pattern: break-even against hosted APIs typically arrives in 4 to 14 months for heavy daily users and never for occasional ones. Token volume is the deciding factor.
So the honest question is not "is self-hosting cheaper?" but "how many requests will we really send, every day, for the next two years?" If you cannot answer that yet, a short measurement period on a managed service will tell you. Our piece on AI spend per employee shows one way to gather that baseline, and the AI opportunity calculator helps size where the savings are.
Security and privacy: what you take on
Keeping data in-house is the top reason businesses self-host. It also moves the security work onto you. The EPRI playbook lists the duties: network segmentation, access control, secure API endpoints and key management, identity and access management, defenses against prompt injection and misuse, controls against unauthorized use, and strict data access controls with logging and audit trails.
Translated for a small business:
- Keep the model server off the public internet. Expose only the application layer people need, behind logins. One Kubernetes deployment example from a published tutorial uses an internal-only service type so the model runtime is not directly reachable from outside the cluster.
- Give people access by role. The bookkeeper and the field tech should not see the same documents through the assistant.
- Decide what you log. Logs help with debugging and quality review, and they can also hold sensitive prompts. Store only what you need, redact secrets, restrict who can read logs and set a retention period before the system touches sensitive work.
- Plan for updates. Model versions, runtimes and operating systems all need patches and re-testing.
If you operate in a regulated space, such as healthcare, the rules go further than good hygiene. Our guides on HIPAA-compliant AI and intelligence sovereignty for regulated industries cover what to put in place. For legal questions specific to your business, talk to your attorney or compliance officer.
Self-hosted, managed or hybrid: how to choose
Use this table as a first filter.
| Your situation | Likely best fit |
|---|---|
| Client or patient data must not leave your environment | Self-hosted for that workflow |
| High, steady daily volume of similar requests | Self-hosted, after a cost check |
| Occasional or bursty use, quick start wanted | Managed service |
| No one to own the system day to day | Managed service, or a partner who runs it |
| Mix of sensitive and general tasks | Hybrid |
Hybrid is the answer we see most. Sensitive documents, internal knowledge bases and high-volume classification run on models you host. Hard, occasional tasks that need the strongest available model go to a managed provider, with rules about what data may be sent there. You get privacy where it matters without paying to own capacity you rarely use.
A simple rule: self-host the layers that touch sensitive data, and buy the rest unless your volume justifies owning them.
## Our perspective Most businesses we meet do not have a hosting problem. They have a workflow problem: leads that go cold for two days, invoices typed in twice, schedules held together by one person's memory. Where the model runs is a decision you make after you know which workflow is costing you money. That is why every engagement of ours begins with a diagnostic. We find the costly inefficiency first, then choose the lightest setup that fixes it. Sometimes that is a managed model wired into your existing tools. Sometimes, when the data is sensitive or the volume is steady, it is a private deployment on your own hardware. We build both, and we do not mark up tools or lock you in. The part we care about most is what happens after launch. A self-hosted system needs monitoring, patching, model updates and someone watching the costs. We stay on to do that, and we send monthly financial reports showing what the system costs to run and what it saves. If the numbers do not justify the system, you will see it in black and white, and we would rather you see it from us first. Greyson
A sensible rollout in six steps
The most consistent advice in the sources we reviewed matches our experience: start small. A narrow pilot on one low-risk workflow exposes the maintenance effort before you have scaled it.
- Pick one workflow. Choose something repetitive and low risk: summarizing inbound emails, drafting follow-ups, or answering internal policy questions.
- Write down the data rules. What data can the system touch? Who can see outputs? How long are logs kept?
- Size the hardware to that workflow. Use the VRAM and RAM rules above and a model that fits comfortably, not one that barely loads.
- Choose and verify the model. Check license terms on the publisher's page. Test it against twenty real examples from your business.
- Connect it to your tools. Wire the output into the CRM, scheduling or invoicing system your team already uses, so people do not have to change habits.
- Measure for 60 to 90 days. Track requests, response time, errors, hours saved and running cost. Expand, adjust or switch to managed for this workflow based on the numbers.
Before going to production, also confirm current pricing, GPU availability, performance benchmarks and any compliance claims against the primary sources for the model and runtime you chose. Vendor pages change.
The AI integration checklist and our AI readiness assessment guide walk through the groundwork for steps 1 and 2.
Who runs it after launch?
This is the question that decides whether a self-hosted project succeeds. Self-hosting shifts cost, security, monitoring and operational ownership of the inference service to your team. In a 25-person company, that usually means one overloaded person or an outside partner.
Before you commit, name the owner. Write down who patches the server, who watches usage and cost, who retrains staff when the workflow changes, and who answers when the assistant produces something wrong. If the answer is "nobody yet," you have two options: assign it, or bring in help. Our guide to hiring an AI consultant lists questions worth asking any provider, including us.
Work with Krastor on self-hosted AI
We integrate AI systems into businesses with 10 to 250 employees across the Southeast, including western North Carolina, Upstate South Carolina and the Charlotte metro. Our engagements start with a diagnostic that finds where manual work costs you the most. The Blueprint, from $1,500 one-off, turns that into a plan, including whether a private, on-premises deployment makes sense for your data and volume.
If you already know you want models running on your own hardware, start with our Private AI service. We also run the system after launch, with monitoring and monthly reports on cost and savings.
FAQ
Is self-hosted AI cheaper than ChatGPT or other managed services?
Not automatically. Self-hosting replaces per-token billing with fixed costs for hardware, power and engineering time. One analysis estimates break-even in 4 to 14 months for heavy daily users and never for occasional ones. Measure your real usage before you buy hardware.
Do I need a GPU to run AI in-house?
For responsive chat and automation, yes, in practice. CPU-only inference works but can be roughly 10x to 100x slower than GPU inference for the same model. CPU-only setups suit overnight batch jobs better than live use.
Is self-hosted AI automatically secure?
No. Keeping data in your environment removes one exposure, but you then own network segmentation, access control, logging and patching. The EPRI playbook lists these as the customer's responsibilities. Regulated businesses should involve their compliance officer or attorney.
Can I mix self-hosted and cloud AI?
Yes, and many businesses do. Keep sensitive data and high-volume routine tasks on models you host, and send occasional hard tasks to a managed provider under clear data rules. The split is easiest to design after a workflow audit.
What can a small business realistically run first?
A modest open-weight model on a single GPU machine, serving one workflow such as document summaries, internal Q&A or lead-reply drafts. Openly licensed models like gpt-oss-20b are built to run on infrastructure you control.
Sources
- EPRI AI Model Playbook: Self-Hosting Considerations
- Puget Systems: What hardware do you need to run a local LLM?
- OpenAI Help Center: open-weight models (gpt-oss)
- SpecPicks: Renting AI compute vs running it at home
- PromptCost: RTX 4090 local vs cloud GPU
- AWS: AI Infra on AWS Guide
- You.com: What Is Self-Hosted AI?
- Layer3 Labs: Local AI Hardware Calculator
- Markaicode: Kubernetes chatbot with Ollama