Private AI, When Not to Send Data to an AI API

A business should avoid sending data to a third-party AI API when a contract, a regulation or its own risk tolerance says that data must not leave its control: client records under confidentiality terms, health or financial data, personal data that would require new transfer agreements, trade secrets, or source code you cannot risk exposing. In those cases the alternative is private AI, where an open-weight language model runs on servers you own or rent and control, so prompts, documents and outputs stay inside your own environment. For everything else, a hosted API with the right business terms is usually cheaper and more capable.

Most businesses need both, applied to different data. This post explains how to sort your data, what private AI involves, and the trade-offs.

What actually happens when you send data to an AI API?

When you call a hosted model, your prompt and any documents you include travel to the provider's servers, are processed there, and the answer comes back. What happens to that data afterwards depends on the provider's terms and the plan you use.

Business and API plans from established providers generally state that customer data is not used to train their models and is kept only for a limited period, often for abuse monitoring. Consumer chat apps can have different terms. Read the actual terms for the specific plan, and check:

• Whether inputs and outputs are used for training.
• How long they are retained, and whether zero-retention options exist.
• Where processing happens, by country or region.
• Which subprocessors are involved.
• Whether the provider offers a data processing agreement.

For a lot of business data, those terms are acceptable. The question is which data they are not acceptable for.

When should data stay off third-party AI APIs?

Work through your data by category. Keep it in-house when:

1. A contract forbids sharing it. Client confidentiality clauses, NDAs and some supplier agreements prohibit passing information to third parties, which can include an AI provider.
2. Regulation restricts where it goes. Health records, financial data, legal files and personal data covered by data protection law may require specific safeguards, transfer agreements or local processing.
3. It is a core trade secret. Formulas, unreleased product plans, pricing models and proprietary source code.
4. A client or regulator might ask you to prove where it went. If you cannot document the flow, you have a problem in an audit.
5. The volume is high and continuous. Processing large document volumes through a paid API every day can cost more than running your own model.

If none of these apply, a hosted API under business terms is usually the sensible choice.

What is private or on-premise AI?

Private AI means running a language model inside infrastructure you control. The usual options:

• On-premise: a server with suitable GPUs in your own office or data room.
• Dedicated hosting: a server rented from a hosting provider, used only by you, in a region you choose.
• Private cloud: GPU instances in your own cloud account, with network access restricted to your systems.

The model itself is an open-weight model, one whose trained weights are published so anyone can download and run them. Several capable families exist, in sizes ranging from small models that run on a single graphics card to large ones that need several.

Around the model, you run the same supporting pieces as any AI system: a serving layer that handles requests, a retrieval index if the model answers from your documents (see our post on retrieval-augmented generation), logging, access control and monitoring.

Can a private model be as good as a hosted one?

For broad, open-ended tasks, the largest hosted models are generally ahead of what most businesses can run privately. For narrow, well-defined tasks, the gap is often small or absent. Those tasks include classifying documents, extracting fields, summarising in a fixed format and answering from a defined set of documents.

Two things close the gap further:

• Retrieval, so the model answers from your own material instead of relying on what it memorised.
• Fine-tuning, training an open-weight model further on examples of your specific task, which can make a smaller model perform well on that one job.

The right test is your own task. Build a test set of real examples, run them through a hosted model and a private one, and compare accuracy. Our post on extracting data from PDFs describes how to build that kind of test set.

What are the trade-offs?

Control

Private wins. Data never leaves your environment, you choose the model version and it does not change under you, and you can document every step for an audit.

Capability

Hosted wins on the hardest general tasks. Private is competitive on narrow ones.

Cost

Hosted is cheaper at low or irregular volume, because you pay per use. Private has a fixed cost for hardware or a dedicated server whether you use it or not, so it pays off at steady, high volume.

Upkeep

Hosted needs almost none. Private needs someone to maintain the server, update the software, monitor performance and evaluate new models as they appear.

How do you decide, step by step?

1. List the AI use cases you want, and the data each one touches.
2. Classify that data: public, internal, confidential, regulated.
3. Check your contracts and obligations for each confidential or regulated category.
4. Send public and internal data to a hosted API under business terms.
5. Test a private model on the confidential and regulated use cases against a measured accuracy target.
6. Size the infrastructure from expected volume, and compare its cost with the API cost for the same work.
7. Document the data flows for each use case, so you can answer a client or auditor quickly.

Many organisations end up with a split: a hosted model for drafting and general research, and a private model for anything involving client files.

Working with Syntora Ai

Syntora Ai sets up private and on-premise AI for organisations that cannot send data to a third-party API, using open-weight models fine-tuned on their data and served on hardware they control. If you are unsure which of your data can go where, write to hello@syntorahq.ai or see our private AI practice.