Short answer: you cannot put protected health information (PHI) into a consumer chatbot account, and you can use a large language model with PHI only where a business associate agreement (BAA) covers the exact service you call. OpenAI, Anthropic, Microsoft and Google each offer a BAA, and each limits it to named services and configurations (checked on 16 November 2026). The alternative is a private LLM for healthcare: an open-weight model on a dedicated server you control, where prompts and outputs stay on one machine. That removes the model vendor from the data flow. It does not remove the HIPAA Security Rule, and it does not remove the need for a BAA with whoever hosts the server.
This article is written for IT, security and compliance leads at covered entities and business associates. It is information, not legal advice, and no hosting choice makes an organisation HIPAA compliant on its own: compliance is the result of your risk analysis, your policies, your contracts and your controls. Confirm every contractual point with counsel and with the vendor's current documents.
Can I use ChatGPT with PHI?
Not through a personal or team account without a BAA. Under HIPAA, a vendor that creates, receives, maintains or transmits electronic PHI on your behalf is a business associate, and 45 CFR 164.308(b)(1) lets a covered entity permit that only after it obtains satisfactory assurances, in a written contract, that the vendor will safeguard the information. A chat window with no BAA behind it has no such contract.
OpenAI does sign a BAA, but its Healthcare Addendum limits the agreement to what it calls Eligible Services: OpenAI's Zero Retention API, ChatGPT Enterprise, and any other service OpenAI explicitly identifies. API traffic is excluded unless it goes through an approved organisation ID with zero retention activated and to an endpoint eligible for zero data retention. So the honest answer to "can I use ChatGPT with PHI" is: only in the specific products and configurations the signed addendum names, and never in the default consumer product.
Which LLM API vendors sign a BAA, and what does it cover?
The table summarises each vendor's own published terms as read on 16 November 2026. Scope lists change often (Google's covered-products list was last updated on 11 November 2026), so read the current document before you rely on any row.
| Vendor | BAA available | What the vendor says is covered | Limits to check |
|---|---|---|---|
| OpenAI | Yes, on request | The Zero Retention API, ChatGPT Enterprise, and services OpenAI explicitly identifies | API use counts only through an approved organisation ID with zero retention activated and a ZDR-eligible endpoint |
| Anthropic | Yes, for its "HIPAA-ready services" | The first-party API (enabled through sales after the primary owner signs), Claude Enterprise once HIPAA is activated and the BAA accepted, and Claude in Microsoft Foundry with a HIPAA configuration | Covered models require 30-day data retention and are not available with zero data retention unless authorised. Excluded: the Console, Batch API, Files API, code execution, web fetch, connectors and beta features |
| Microsoft Azure | Yes, through the Product Terms, with no separate contract to sign | "In-scope Azure services" for customers that are covered entities or business associates | We did not read Microsoft's in-scope service list, so confirm on that list that the model service you plan to call is named |
| Google Cloud | Yes, the customer reviews and accepts the Google Cloud BAA | The covered-products list includes "Generative AI on Agent Platform", "Gemini Enterprise Agent Platform", "Gemini Enterprise" and "Gemini Code Assist" | The Gemini Developer API and AI Studio are not mentioned on the page |
Two consequences follow. Features drift out of scope: a developer who switches on a batch endpoint, a file upload or a beta tool may have moved PHI outside the BAA. And retention settings are part of the contract: one vendor ties coverage to zero retention, another requires 30-day retention for covered models.
Where the API route is the better choice: low volumes, pilots, and work that needs the quality of a frontier model. If a clinic sends a few hundred requests a day, a BAA-covered API costs far less than a dedicated GPU and needs no operations team.
What changes when the model runs on a server you control
A self-hosted open-weight model changes the data flow, not the law. Prompts, retrieved documents, transcripts and outputs are processed in the memory of one machine and written only to disks you manage. No model vendor receives PHI, so there is no vendor retention window, no feature-by-feature scope list and no question about training on your inputs.
What stays exactly the same:
- The hosting provider is a business associate. HHS's cloud computing guidance is widely cited for the position that a provider storing ePHI is a business associate even when the data is encrypted and the provider holds no key. We could not load hhs.gov on 16 November 2026, so read the guidance linked below rather than relying on our summary. In practice: you need a BAA with the company that hosts the server.
- The Security Rule applies in full, including the contingency plan. The next section maps it to an LLM server. A dedicated server is a single machine, so redundancy and backups are your job: mirror the two NVMe drives and keep encrypted copies off the server.
- The minimum necessary standard, your risk analysis and workforce training still govern what goes into a prompt and who may send it.
MassiveGRID signs a BAA, and the HIPAA compliant hosting page lists the products it covers. Before you place PHI on a GPU server, confirm with sales in writing that the BAA covers that specific server, and if the location of the hardware matters to your policies, ask sales which data center a server will be delivered in before ordering. The servers themselves are described on the GPU dedicated servers page: bare metal, one customer per machine, root access, a clean operating system and nothing pre-installed.
Security Rule safeguards mapped to a self-hosted LLM
| Safeguard (45 CFR) | What the rule says today | What it means for an LLM server |
|---|---|---|
| Access control, 164.312(a) | Unique user identification and an emergency access procedure are required; automatic logoff and encryption are addressable | Every clinician and service authenticates as itself at a gateway. No shared API key, no open port on the model |
| Audit controls, 164.312(b) | Record and examine activity in systems that contain or use ePHI | Log who called which model, when, and with how many tokens. Keep prompt text out of general logs |
| Integrity, 164.312(c) | Protect ePHI from improper alteration or destruction | Outputs written to the record are reviewed and signed by a human; vector stores and transcripts are backed up |
| Authentication, 164.312(d) | Verify that a person or entity seeking access is the one claimed | Single sign-on with multi-factor authentication in front of the chat interface and the API |
| Transmission security, 164.312(e) | Guard against unauthorised access to ePHI in transit; encryption is addressable | TLS on every hop from workstation to gateway; the model itself listens on localhost only |
| Contingency plan, 164.308(a)(7) | Backup, disaster recovery and emergency mode plans are required | RAID 1 on the NVMe pair, encrypted off-server backups, a written rebuild procedure that has been rehearsed |
"Addressable" does not mean optional: you implement the specification or document why an equivalent measure is reasonable.
The proposed Security Rule update: still a proposal
HHS published a notice of proposed rulemaking on 6 January 2025, "HIPAA Security Rule To Strengthen the Cybersecurity of Electronic Protected Health Information" (RIN 0945-AA22, 90 FR 898). It is not final. The current Unified Agenda entry lists it under long-term actions with final action dated July 2027, and that date is a target, not a commitment.
As its title says, the proposal aims to strengthen the safeguards for ePHI with more specific requirements than the current text. Our guide to the Security Rule update lists the proposed figures, and the 72-hour restoration architecture post covers the recovery side. We have not verified what the preamble says about AI tools, so we make no statement about it: read 90 FR 898 itself before quoting it in a policy.
A reference architecture for PHI on a GPU server
The pattern below keeps the model unreachable from the network and puts identity, TLS and audit logging in one place. It assumes Ubuntu 24.04 LTS and working NVIDIA drivers (see the first-hour driver and Docker guide).
- Encrypt the data volume with LUKS before any PHI arrives. Put model caches, vector stores, transcripts and application databases on a LUKS volume created with
cryptsetup. On a remote server, decide in advance how the volume is unlocked after a reboot: a passphrase typed over SSH keeps the key off the disk, at the price of a manual step. Verify withlsblk -fthat the volume shows ascrypto_LUKS. - Bind the model to localhost. Ollama binds 127.0.0.1 port 11434 by default, so leave
OLLAMA_HOSTunset. For vLLM, pass the loopback address and an API key explicitly:
Verify from the server withvllm serve <model-repo> --host 127.0.0.1 --port 8000 --api-key "$VLLM_API_KEY"curl http://localhost:8000/v1/models(send the key in the request header) and confirm withss -ltnpthat nothing listens on a public address. - Put an authenticated gateway in front. A reverse proxy terminates TLS, checks each user against your identity provider with multi-factor authentication, and forwards to localhost. Firewall the server so that only ports 443 and SSH are reachable, and restrict SSH to your management addresses or a VPN.
- Log metadata, not prompts. The gateway records user, time, model, endpoint, token counts and status. Prompt and completion bodies stay out of access logs, error trackers and monitoring dashboards. If you must retain conversations for the medical record, store them in the encrypted application database with the same access rules as the record, and set a retention period.
- Back up off the server. Encrypt backups before they leave the machine and store them with a party that is also under a BAA. Test a restore.
- Control the supply chain. Pin model versions and container tags, and block outbound traffic from the inference host except to the update sources you approve.
The vLLM API guide and the Ollama and Open WebUI walkthrough cover the installation steps in detail.
Use cases, model choices and server sizing
Clinical note drafting. A model turns a visit transcript or bullet points into a draft note that the clinician edits and signs. A model of about 27 billion parameters is a common starting point: on Ollama, qwen3.8:27b is an 18 GB download under Apache 2.0, and medgemma, a medical variant of Gemma 3, lists 3.3 GB and 17 GB sizes (read its licence terms before clinical use; the Ollama tag list we read did not show them). Both fit the 24 GB RTX PRO 4000 server at $449.99 per month.
Coding assistance. Suggesting ICD-10 and CPT codes from a note is a retrieval problem as much as a generation problem: pair the model with a code-set index (a self-hosted RAG stack) and let a certified coder accept or reject each suggestion.
Transcription with Whisper. OpenAI's Whisper README lists the large model at about 10 GB of VRAM and the turbo model at about 6 GB and roughly eight times the speed. Turbo is an optimised large-v3 released under the MIT licence. The Whisper on GPU guide has the setup.
| Workload | Example footprint | Server | Monthly |
|---|---|---|---|
| One 27B-class model for note drafting, or Whisper turbo for dictation | 18 GB download, or about 6 GB | RTX PRO 4000, 24 GB | $449.99 |
| Note model, Whisper and an embedding model loaded together | 18 GB + about 6 GB + 1.2 GB (bge-m3) = about 25 GB before context | RTX PRO 6000, 96 GB | $1,656.15 |
| A 70B-class model with long context for many concurrent users | 43 GB download (llama3.3:70b) plus context memory | RTX PRO 6000, 96 GB | $1,656.15 |
The second row shows why the 24 GB card is a one-task machine: the arithmetic already exceeds 24 GB before any context window is allocated. Setup is a one-time $346.48 on the 24 GB server and $1,656.15 on the 96 GB server. We publish no throughput figures, so measure with your own notes and audio before you commit to a size. For a wider view of the options, see private AI hosting.
Limits: what a private LLM does not solve
- It is not compliance. A server and a BAA are two inputs to a compliance programme. The risk analysis, policies, training, access reviews and incident response are yours.
- Model quality is lower than the frontier. Open-weight models that fit in 24 GB or 96 GB are weaker than the largest hosted models on hard reasoning. For clinical text, that means a human reviews every output before it enters the record or reaches a patient.
- You run it. Patching the OS, drivers and inference stack, rotating keys, monitoring the GPU and restoring from backup are operational work that an API hides.
- One machine is one point of failure. Plan a second server or a documented manual fallback.
- De-identification is not automatic. Asking a model to strip identifiers does not meet the de-identification standard by itself. Treat model inputs and outputs as PHI unless your privacy officer has determined otherwise.
Frequently Asked Questions
Is there a HIPAA compliant LLM?
No model is HIPAA compliant by itself, because HIPAA regulates how covered entities and business associates handle PHI, not software products. An LLM can be used in a compliant way either through an API service covered by a signed BAA, within that BAA's scope, or as a self-hosted model on infrastructure covered by a BAA with the hosting provider and protected by the Security Rule safeguards. This is general information, not legal advice.
Does OpenAI sign a BAA for the API?
Yes, on request, but the BAA is limited to what OpenAI's Healthcare Addendum calls Eligible Services: the Zero Retention API, ChatGPT Enterprise and services OpenAI explicitly identifies. API use is covered only through an approved organisation ID with zero retention activated and a ZDR-eligible endpoint (addendum read on 16 November 2026). Consumer ChatGPT accounts are not covered.
Do I still need a BAA if I self-host the model?
Yes, with the company that hosts the server, unless the hardware sits in your own facility. A hosting provider that maintains ePHI on your behalf is a business associate. MassiveGRID signs a BAA, described on the HIPAA hosting page; confirm with sales that it covers the specific GPU server before you place PHI on it.
Which GPU server is enough for a private healthcare LLM?
A single 27B-class model at 4-bit quantisation is an 18 GB download and fits the 24 GB RTX PRO 4000 server at $449.99 per month. Running a note-drafting model, Whisper and an embedding model at the same time, or a 70B-class model, calls for the 96 GB RTX PRO 6000 server at $1,656.15 per month. Both are listed on the GPU dedicated servers page.
Is the HIPAA Security Rule update final?
No. The notice of proposed rulemaking was published on 6 January 2025 and the current Unified Agenda entry lists final action for July 2027 under long-term actions. Until a final rule is published, the existing Security Rule applies.
A private model on hardware that is yours alone
MassiveGRID GPU dedicated servers are bare metal with one customer per machine: an RTX PRO 4000 with 24 GB for $449.99 per month or an RTX PRO 6000 with 96 GB from $1,656.15, with root access and a 1 Gbps port with unlimited traffic, hosted in an ISO 27001 certified data center. MassiveGRID signs a BAA: confirm with sales that it covers the GPU server you order before placing PHI on it.
See Private AI HostingFurther Reading
- OpenAI: Healthcare Addendum and Business Associate Agreement (Eligible Services)
- Anthropic: Business Associate Agreements for commercial customers
- Microsoft Learn: HIPAA and the Azure BAA under the Product Terms
- Google Cloud: HIPAA compliance and covered products
- eCFR: 45 CFR 164.312, Technical safeguards (see also 164.308 and 164.314)
- Unified Agenda: RIN 0945-AA22, HIPAA Security Rule NPRM status
- HHS: Guidance on HIPAA and cloud computing
- OpenAI Whisper: model sizes and VRAM requirements