If you run Ollama, LiteLLM or Open WebUI on a server, your AI keys are now a target. In this tutorial we build a hardened LLM gateway in Python: a small, self-hosted proxy that sits in front of your models with per-user API keys hashed at rest, no default keys, rate limits, token budgets and a tamper-evident audit log.
It also ships a config auditor that scans your docker-compose, LiteLLM and Open WebUI files for default master keys, exposed ports and disabled logins, plus a read-only self-check for your own endpoints. Everything runs offline with a mock model, and it works with a real Ollama model too.
This is a practical AI and cyber security final-year project for BTech, MCA and MTech students, and a useful tool for anyone who self-hosts LLMs. The complete code of the hardened LLM gateway, with 190 tests, is free on GitHub under the MIT License.
Table of Contents
Why a Hardened LLM Gateway Matters in November 2026
Stolen AI keys turned into a business this autumn. Here is what was reported, with dates:
- 2 Nov 2026: Google Threat Intelligence Group and Mandiant published “The Rise of LLMJacking”. They saw fraudulent AI charges above $100,000 in a single day in some incidents, attackers testing publicly exposed Ollama endpoints, key checks that used tiny
max_tokens=1requests to stay quiet, and a LiteLLM/key/regenerateflaw used to turn ordinary virtual keys into admin keys. - 10 Nov 2026: an AWS Builder Center guide, “LLMjacking: how to keep stolen credentials off your Amazon Bedrock bill”, called the AI gateway a high-value target: keep its keys out of config, require authentication and patch it.
- 13 Nov 2026: Amazon Threat Intelligence (CJ Moses) reported AI agents running a self-funding campaign to steal LLM API keys. Decoy sensors captured the agents’ own records. Stolen keys paid for the AI that hunted for more keys. Targets included LiteLLM, Open WebUI, Ollama, Dify, Langflow and One API, and the playbook listed default admin credentials such as the LiteLLM default master key, anonymous management endpoints and secrets in environment files.
- 16 Nov 2026: The Hacker News summarised the report and the checks defenders should run: change default admin credentials, finish new installs, and search configs, CI pipelines and databases for embedded keys.
- 18 Nov 2026: CISA added LiteLLM CVE-2026-40217 (a guardrail sandbox escape) to its Known Exploited Vulnerabilities catalog, as explained by Corgea (19 Nov) and the Cloud Security Alliance (20 Nov).
Amazon’s own conclusion is the key point: none of the techniques were new. The attacks worked where basic hygiene was missing, such as default credentials, keys left in files, unauthenticated inference endpoints and unpatched software. That is exactly what a gateway and a config auditor can fix.
The agents succeeded where basic security hygiene was missing, including default or hardcoded credentials, keys left in code and public artifacts, unauthenticated interfaces and inference endpoints, and unpatched software. (AWS Security Blog, 13 Nov 2026)
What Is a Hardened LLM Gateway?
A hardened LLM gateway is a thin proxy between your users and your model server. Clients talk to the gateway with their own personal key. The gateway checks the key, applies limits, forwards only safe routes to Ollama or LiteLLM using its own private upstream key, and writes one audit line per request. The model server itself has no public port at all.
Think of it as the security guard at the gate. Ollama has no login of its own, and LiteLLM and Open WebUI are powerful admin tools, so they should live behind the guard, not on the open internet.
| Problem in a typical self-hosted stack | What the gateway does |
|---|---|
| Ollama port 11434 open to the internet, no auth | Ollama stays on a private network; only the gateway can reach it |
Everyone shares one master key like sk-1234 | Personal keys per user, HMAC-SHA256 hashed at rest, shown once |
Admin APIs (/key/*, /api/pull, /guardrails/*) reachable | Only chat, embeddings and model list are served; the rest is 404 and logged |
| No idea who used what | Hash-chained JSONL audit log with user, model, tokens and latency |
| A leaked key can burn money all night | Requests-per-minute limit, daily token budget, revoke and rotate in one command |
| Weak secrets copied from tutorials | The gateway refuses to start with default or weak secrets |
Features of the Hardened LLM Gateway Project
- Per-user API keys in the format
lgw_<id>_<secret>, stored only as HMAC-SHA256 with a server-side pepper. Expiry, revoke and rotate included. - No default keys. A fresh install has zero keys, no master key and no admin password.
- Start-up preflight that refuses known defaults (
sk-1234,changeme,t0p-s3cr3t), short or low-entropy secrets, a world-readable key file and public bind addresses without an explicit opt-in. - Rate limiting with a token bucket per key, a daily token budget, and a temporary IP lock-out after repeated failed logins.
- Per-key model allow-lists, a
max_tokenscap and a request size limit, which also helps against OWASP’s “unbounded consumption” risk. - Route allow-list: Ollama model management and LiteLLM admin endpoints are never forwarded.
- JSONL audit log with a SHA-256 hash chain, no keys and no prompt text, plus
logs verifyandlogs reportdetections. - Config auditor for docker-compose, LiteLLM
config.yaml, Open WebUI.envand Ollama systemd units, with a 0-100 hardening score and CI-friendly exit codes. - Read-only exposure self-check for your own hosts, a CLI, a Flask admin UI and a hardened docker-compose stack.

Tech Stack and Requirements
| Part | Used for |
|---|---|
| Python 3.10+ | Gateway, CLI, auditor (standard library for crypto, hashing and HTTP) |
| Flask | Gateway routes and the small admin UI |
| PyYAML | Parsing docker-compose and LiteLLM config files safely |
| Ollama (optional) | Real open model, e.g. qwen2.5-coder:3b; LiteLLM or vLLM also work |
| pytest | 190 automated tests |
| Docker Compose (optional) | Hardened reference stack with Open WebUI |
You do not need a GPU. The built-in mock model runs the demo and every test on any laptop.
Project Structure
Hardened-LLM-Gateway-Python/
├── llm_gateway/
│ ├── gateway.py # Flask app: auth, limits, allow-lists, forwarding, admin UI
│ ├── keys.py # per-user keys, HMAC-SHA256 at rest, expiry, revoke, rotate
│ ├── secrets_check.py # default/weak secret detection, provider-key patterns
│ ├── settings.py # env settings + preflight (refuses to start on ERROR)
│ ├── ratelimit.py # token bucket, daily token budget, IP lock-out
│ ├── audit.py # hash-chained JSONL audit log + detections
│ ├── auditor.py # LiteLLM / Open WebUI / Ollama / compose config auditor
│ ├── exposure.py # read-only self-check of your own endpoints
│ ├── upstream.py # OpenAI-compatible HTTP upstream + offline mock
│ └── cli.py, demo.py, wsgi.py, templates/, static/
├── examples/insecure/ # deliberately bad configs for the auditor
├── examples/secure/ # hardened docker-compose + LiteLLM config
├── tests/ # 190 pytest tests
└── Dockerfile # non-root image
How the Hardened LLM Gateway Works
Every request goes through the same pipeline, and every step can stop it:
- IP lock-out: too many failed keys from one address returns 429 for 15 minutes.
- Route allow-list: only
GET /v1/models,POST /v1/chat/completionsandPOST /v1/embeddingsexist. Everything else is a logged 404. - Key check: the key comes from the
Authorization: Bearerheader (never from the URL) and is compared as an HMAC in constant time. - Limits: requests per minute and the daily token budget for that key.
- Validation: JSON body, model allow-list for that key,
max_tokensclamped to the cap. - Forward: the gateway calls the model with its own upstream key. The client’s header is never forwarded.
- Audit: one JSON line with user, model, status, tokens, latency and a short prompt hash.
client (curl / OpenAI SDK / Open WebUI)
│ Authorization: Bearer lgw_<id>_<secret>
▼
hardened gateway: IP lock-out → route allow-list → key HMAC → rate limit/budget
→ JSON + model allow-list + max_tokens cap → forward → audit
│ gateway's own upstream key (never sent to clients or logs)
▼
Ollama / LiteLLM / vLLM on a private network (no published ports)
Step 1: Per-User API Keys Hashed at Rest
A key is created with Python’s secrets module and shown exactly once. Only an HMAC-SHA256 of it, keyed with a pepper from the environment, is written to keys.json (mode 600). If someone copies the key file, they still cannot use or reverse the keys without the pepper.
def verify(self, presented: str | None, now: float | None = None) -> tuple[ApiKey | None, str]:
"""Return (key, "ok") or (None|key, reason). Constant-time hash compare."""
self.refresh()
if not presented:
return None, "missing_key"
m = KEY_RE.match(presented.strip())
if not m:
return None, "malformed_key"
rec = self._keys.get(m.group(1))
expected = rec.key_hash if rec else hash_key(self._pepper, "decoy")
ok = hmac.compare_digest(expected, hash_key(self._pepper, presented.strip()))
if not rec or not ok:
return None, "unknown_key"
st = rec.status(now)
if st != "active":
return rec, f"{st}_key"
rec.last_used = time.time() if now is None else now
return rec, "ok"
Two small details matter. The comparison uses hmac.compare_digest, so timing does not leak how many characters matched, and an unknown key ID still runs a decoy hash for the same reason. The running gateway also reloads the file when the CLI changes it, so a revoked key stops working at once.
Step 2: Refuse to Start with Default or Weak Secrets
Copy-pasted secrets are the easiest door for attackers. The checker rejects values from docs and tutorials, placeholders, short strings, repeated patterns and low-entropy values:
def secret_problems(value: str | None, min_length: int = MIN_LENGTH) -> list[str]:
"""Return a list of human-readable reasons why *value* is unsafe.
An empty list means the secret looks strong enough.
"""
if value is None or value.strip() == "":
return ["is empty or not set"]
v = value.strip()
low = v.lower()
problems: list[str] = []
if low in KNOWN_DEFAULTS:
problems.append("is a well-known default value")
elif any(h in low for h in PLACEHOLDER_HINTS):
problems.append("looks like a placeholder from an example file")
if re.fullmatch(r"(sk-)?\d+", low):
problems.append("contains only digits")
if len(v) < min_length:
problems.append(f"is too short ({len(v)} chars, need at least {min_length})")
if len(set(v)) < 10:
problems.append("uses too few distinct characters")
elif shannon_entropy(v) < 3.5:
problems.append("has low entropy (looks guessable)")
if re.search(r"(.{1,4})\1{4,}", v):
problems.append("contains a repeating pattern")
return problems
The preflight runs this on the pepper, the admin token and any upstream key, and it also refuses a world-readable key file or a 0.0.0.0 bind without GATEWAY_ALLOW_PUBLIC_BIND=1. If anything is an ERROR, serve exits before opening a port.

Step 3: Rate Limits, Token Budgets and Route Allow-Lists
Each key has its own token bucket (for example 30 requests per minute) and a daily token budget. Unknown routes never reach the model, which matters because attackers specifically look for model-management and key endpoints:
ok, retry = rl.allow(rec.key_id, rec.rpm)
if not ok:
log.write("rate_limited", **base, retry_after=retry)
r = _err(429, f"rate limit of {rec.rpm} requests/minute exceeded", "rate_limited")
r.headers["Retry-After"] = str(max(1, int(retry + 0.999)))
return r
if tb.remaining(rec.key_id, rec.daily_tokens) <= 0:
log.write("budget_exceeded", **base, daily_tokens=rec.daily_tokens)
return _err(429, "daily token budget used up", "budget_exceeded")
payload = request.get_json(silent=True)
if not isinstance(payload, dict):
log.write("bad_request", **base, reason="body is not a JSON object")
return _err(400, "body must be a JSON object", "bad_request")
model = payload.get("model")
if not isinstance(model, str) or model not in allowed_models(rec):
log.write("model_denied", **base, model=str(model)[:80])
return _err(403, f"model {model!r} is not allowed for this key", "model_denied",
allowed=allowed_models(rec))
The gateway also caps max_tokens, limits the body size and sets user to the key owner so upstream logs can show who asked. These controls map to OWASP’s LLM10:2025 Unbounded Consumption risk.
Step 4: A Tamper-Evident JSONL Audit Log
Every event, allowed or denied, becomes one JSON line. Each line stores the SHA-256 of the previous line, so editing, deleting or re-ordering lines breaks the chain. Sensitive field names such as api_key, prompt and messages are refused by the logger itself.
{"event": "request", "user": "pardeep", "key_id": "a2f1ca238b56", "model": "qwen2.5-coder:3b",
"status": 200, "max_tokens": 160, "prompt_tokens": 46, "completion_tokens": 123, "total_tokens": 169,
"latency_ms": 5485, "prompt_sha256": "484ecc91192ec2cc", "ip": "127.0.0.1",
"time": "2026-11-21T03:47:10Z", "prev": "2bcd55f1…"}
logs report turns these lines into findings with a next step. It flags key spraying from one IP, probes for management paths, bursts of max_tokens=1 requests (the quiet key-check pattern Google described), one key used from many IPs, revoked keys still in use and spend close to the budget.
for (kid, ip), n in probes.items():
if n >= probe_threshold:
findings.append({"severity": "HIGH", "rule": "D3-key-validation", "subject": kid,
"detail": f"{n} requests with max_tokens<=1 for key {kid} ({users.get(kid, '?')}) from {ip}",
"advice": "Tiny single-token requests are a known pattern for checking whether a stolen key works. Rotate this key and ask the owner."})

Step 5: Config Auditor for LiteLLM, Open WebUI and Ollama
Most stacks are already deployed, so the auditor reads what you have and tells you what to fix. It is read-only, never connects anywhere and works on a folder, a single file or pasted text in the admin UI.
| Rule | Severity | What it finds |
|---|---|---|
| LGW-L01 | HIGH | Default or weak LiteLLM master key such as sk-1234 |
| LGW-L02 | HIGH | LiteLLM port 4000 (admin UI and key APIs) on all interfaces |
| LGW-L04 | HIGH | LiteLLM image older than 1.84.0 (CVE-2026-40217 and the /key/regenerate fix) |
| LGW-L07 | HIGH | Provider API key written in config.yaml |
| LGW-O03 | HIGH | Ollama 11434 published on all interfaces |
| LGW-W01 / W02 / W04 | HIGH | WEBUI_AUTH=False, weak WEBUI_SECRET_KEY, DEFAULT_USER_ROLE=admin |
| LGW-W03 / O01 / O02 | MEDIUM | Open sign-up, OLLAMA_HOST=0.0.0.0, OLLAMA_ORIGINS=* |
| LGW-D01 / D02 | MEDIUM / HIGH | Weak database password, Postgres or Redis published |
| LGW-S01 | HIGH | OpenAI, Anthropic, AWS, Google, Groq or Hugging Face keys in any scanned file |
Ports are parsed in both short ("11434:11434") and long syntax. A port bound to 127.0.0.1 is fine; one with no host IP is published on every interface:
for ip, pub, target in _ports(svc):
public = _is_public(ip)
where = f"{ip or '0.0.0.0'}:{pub}->{target}"
if not public:
continue
if kind == "ollama" or target == "11434":
f.append(Finding("HIGH", "LGW-O03", "Ollama API published on all interfaces", fname,
_line_of(text, f"{pub}:{target}") or _line_of(text, name), f"service '{name}' publishes {where}. Ollama has no authentication, so anyone who can reach it can run, pull, create or delete models.",
"Remove the ports entry (other containers reach it over the internal network) or bind 127.0.0.1:11434:11434."))
Settings follow the official docs: LiteLLM’s master key and virtual keys, Open WebUI’s environment variables (WEBUI_AUTH, WEBUI_SECRET_KEY, ENABLE_SIGNUP) and the Ollama FAQ on OLLAMA_HOST and OLLAMA_ORIGINS.

Step 6: Read-Only Exposure Self-Check
Config files can lie, so the last check looks at the running service. check-exposure sends a few unauthenticated GET requests (model list, version, public config) to your own host and reports whether anything answers without a key. It never sends credentials, never calls write or management endpoints and never guesses passwords.

On our test box, raw Ollama on port 11434 listed its models to anyone, while the gateway on port 8088 returned 401 or 404 for every probe.
How to Run the Hardened LLM Gateway (Step by Step)
git clone https://github.com/GetProjectsIdea/Hardened-LLM-Gateway-Python.git
cd Hardened-LLM-Gateway-Python
python -m venv .venv && source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -r requirements-dev.txt
python -m llm_gateway demo # offline walkthrough with a mock model
python -m pytest -q # 190 passed
python -m llm_gateway init # .env with random pepper + admin token (chmod 600)
python -m llm_gateway keys create alice --rpm 30 --daily-tokens 200000
python -m llm_gateway serve # http://127.0.0.1:8088 -> Ollama 127.0.0.1:11434/v1
Then point any OpenAI-compatible client at the gateway:
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8088/v1", api_key=ALICE_KEY)
reply = client.chat.completions.create(model="qwen2.5-coder:3b",
messages=[{"role": "user", "content": "Hello"}])
print(reply.choices[0].message.content)
Audit an existing stack and check the log:
python -m llm_gateway audit-config ./my-llm-stack # exit code 1 on HIGH findings
python -m llm_gateway audit-config ./my-llm-stack --json # for CI pipelines
python -m llm_gateway check-exposure http://127.0.0.1:11434 # your own host only
python -m llm_gateway logs verify && python -m llm_gateway logs report
Hardened Docker Compose Stack
The examples/secure folder runs Ollama, the gateway and Open WebUI together. Ollama has no published ports, Open WebUI talks to models only through the gateway using its own key, and the gateway container is read-only, non-root and drops all Linux capabilities. Required secrets use ${VAR:?message}, so Compose refuses to start if one is missing.
gateway:
build: ../..
networks: [llm]
ports:
- "127.0.0.1:8088:8088" # loopback only; publish via a TLS reverse proxy
environment:
GATEWAY_KEY_PEPPER: "${GATEWAY_KEY_PEPPER:?run llm_gateway init}"
GATEWAY_UPSTREAM_URL: http://ollama:11434/v1
read_only: true
cap_drop: [ALL]
security_opt: ["no-new-privileges:true"]
Running the auditor on this folder gives a score of 100/100, while the insecure example scores 0/100 with 13 HIGH findings.
Testing the Hardened LLM Gateway with a Real Ollama Model
The mock model is great for tests, but we also ran the gateway in front of a real open model: qwen2.5-coder:3b on Ollama 0.40.16, CPU only. Requests without a key and with sk-1234 got 401, /api/pull got 404, and the personal key got a code answer in about 5 to 14 seconds. The audit line recorded tokens and latency but not the prompt.

The repository has 190 pytest tests that run in under a second. They cover weak and default secrets, key hashing and file permissions, expiry, revoke and rotate, live reload of revocations, rate limits and budgets, IP lock-out, 10 blocked management paths, body and max_tokens limits, the audit hash chain (edit, delete and re-order), every detection, every auditor rule, the exposure check, the admin UI (login, CSRF, shown-once keys) and the CLI. They also pass from a fresh clone.
Hardening Checklist for Self-Hosted LLMs
- Never publish Ollama (11434), LiteLLM (4000), Postgres or Redis on
0.0.0.0. Use private Docker networks. - Replace every default secret: LiteLLM master key,
UI_PASSWORD,WEBUI_SECRET_KEY, database passwords. - Give each person or app its own key with a budget, and revoke keys when people leave.
- Keep provider keys out of config files and Git. Load them from a secret store and set spend limits at the provider.
- Patch LiteLLM to 1.84.0 or later and rotate the master key and virtual keys afterwards.
- Keep Open WebUI login on, sign-up off and the default role at
pendingoruser. - Put TLS in front of anything users reach, and watch for unexpected spend or unusual usage, as Amazon recommends.
Limitations and Responsible Use
This hardened LLM gateway is a teaching project, not an audited product. Rate limits and budgets are in memory per process, so use one worker or move them to Redis. Streaming requests are answered as one SSE chunk because the gateway counts tokens on the full response. A gateway does not replace patching: keep LiteLLM, Open WebUI and Ollama updated.
The project is strictly defensive. The insecure examples contain only well-known default values and a fake key, and check-exposure must only be used on systems you own or are authorised to test.
Future Scope and Viva Tips
- Move limits and budgets to Redis and add per-team quotas with email or Slack alerts.
- Add OIDC single sign-on for the admin UI and short-lived keys.
- Export metrics to Prometheus and forward the JSONL log to a SIEM.
- Add SARIF output so the auditor shows findings in GitHub code scanning.
- Combine it with a prompt-injection filter in front of the model.
This is day 4 of our daily series. Day 1 was an AI coding agent security scanner in Python, day 2 a third-party script monitor in Python and day 3 a sandboxed coding agent in Python that asks permission before every command. You can pair this gateway with our prompt injection firewall in Python and check your dependencies with the Python supply-chain security scanner. For research angles such as LLM abuse detection, see AI PhD thesis topics 2026, or browse 50 major project ideas for BTech and MCA.
FAQs: Hardened LLM Gateway
Is this hardened LLM gateway a good final-year project?
Yes. It combines API security, hashing, rate limiting, log analysis, config parsing, a Flask web app, Docker and automated testing, and it is based on attacks reported in November 2026.
Do I need a GPU or Ollama to try it?
No. The offline mock model runs the demo and all 190 tests on any laptop. Ollama with a small model such as qwen2.5-coder:3b also works on a CPU.
Does it work with LiteLLM and Open WebUI?
Yes. Point GATEWAY_UPSTREAM_URL at LiteLLM with a strong virtual key, or point Open WebUI at the gateway as an OpenAI-compatible connection with its own key. The auditor checks LiteLLM, Open WebUI and Ollama configs.
Why are API keys hashed with HMAC instead of bcrypt?
The keys are long random values, not human passwords, so a fast keyed hash with a secret pepper is enough and keeps every request fast. Without the pepper, a stolen key file is useless.
Is there a default admin key or password?
No. A fresh gateway has zero keys, and the admin UI is disabled unless you set a strong GATEWAY_ADMIN_TOKEN. The gateway refuses to start with values like sk-1234 or changeme.
Is the source code free?
Yes. The full code is on GitHub under the MIT License.
References
- AWS Security Blog: Amazon Threat Intelligence observes threat actors running cybercrime campaigns with ARTEX AI agents (CJ Moses, 13 Nov 2026)
- The Hacker News: Amazon Uncovers AI Agent Campaign Targeting LiteLLM, Dify, and Other Apps for API Keys (16 Nov 2026)
- Google Cloud / GTIG: The Rise of LLMJacking (2 Nov 2026)
- AWS Builder Center: LLMjacking, how to keep stolen credentials off your Amazon Bedrock bill (10 Nov 2026)
- Corgea: CVE-2026-40217 in CISA KEV, LiteLLM guardrail sandbox escape and the /key/regenerate rebind (19 Nov 2026)
- Cloud Security Alliance: Guardrails as an Execution Channel, LiteLLM CVE-2026-40217 Enters KEV (20 Nov 2026)
- CISA Known Exploited Vulnerabilities catalog and OWASP LLM10:2025 Unbounded Consumption
- LiteLLM docs: virtual keys, Open WebUI environment variables and Ollama FAQ
Download the Source Code
The complete hardened LLM gateway project, with the gateway, CLI, Flask admin UI, config auditor, exposure check, Docker files, README, screenshots and 190 tests, is free on GitHub. Clone it, star it and open an issue if you get stuck.
Conclusion
Attackers now use AI agents to find exposed AI gateways and drain their keys, but the doors they use are old ones: default keys, open ports and secrets in files. In this tutorial you built a hardened LLM gateway in Python that closes those doors with personal hashed keys, start-up checks, limits, an audit log and a config auditor.
Run the auditor on your own stack today, fix the HIGH findings first, and put the gateway in front of your models. Built something on top of it? Tell us in the comments below.