Python MIT License Open Source Project

Hardened LLM Gateway in Python: Secure Self-Hosted Ollama, LiteLLM and Open WebUI with Hashed Keys and Audit Logs (Free Code)

Build a hardened LLM gateway in Python for Ollama, LiteLLM and Open WebUI: hashed per-user keys, no default keys, rate limits and audit logs. Free code.

Project Specifications & Technical Summary

Difficulty Beginner
License MIT License
Advertisement

If you run Ollama, LiteLLM or Open WebUI on a server, your AI keys are now a target. In this tutorial we build a hardened LLM gateway in Python: a small, self-hosted proxy that sits in front of your models with per-user API keys hashed at rest, no default keys, rate limits, token budgets and a tamper-evident audit log.

It also ships a config auditor that scans your docker-compose, LiteLLM and Open WebUI files for default master keys, exposed ports and disabled logins, plus a read-only self-check for your own endpoints. Everything runs offline with a mock model, and it works with a real Ollama model too.

This is a practical AI and cyber security final-year project for BTech, MCA and MTech students, and a useful tool for anyone who self-hosts LLMs. The complete code of the hardened LLM gateway, with 190 tests, is free on GitHub under the MIT License.

Why a Hardened LLM Gateway Matters in November 2026

Stolen AI keys turned into a business this autumn. Here is what was reported, with dates:

  • 2 Nov 2026: Google Threat Intelligence Group and Mandiant published “The Rise of LLMJacking”. They saw fraudulent AI charges above $100,000 in a single day in some incidents, attackers testing publicly exposed Ollama endpoints, key checks that used tiny max_tokens=1 requests to stay quiet, and a LiteLLM /key/regenerate flaw used to turn ordinary virtual keys into admin keys.
  • 10 Nov 2026: an AWS Builder Center guide, “LLMjacking: how to keep stolen credentials off your Amazon Bedrock bill”, called the AI gateway a high-value target: keep its keys out of config, require authentication and patch it.
  • 13 Nov 2026: Amazon Threat Intelligence (CJ Moses) reported AI agents running a self-funding campaign to steal LLM API keys. Decoy sensors captured the agents’ own records. Stolen keys paid for the AI that hunted for more keys. Targets included LiteLLM, Open WebUI, Ollama, Dify, Langflow and One API, and the playbook listed default admin credentials such as the LiteLLM default master key, anonymous management endpoints and secrets in environment files.
  • 16 Nov 2026: The Hacker News summarised the report and the checks defenders should run: change default admin credentials, finish new installs, and search configs, CI pipelines and databases for embedded keys.
  • 18 Nov 2026: CISA added LiteLLM CVE-2026-40217 (a guardrail sandbox escape) to its Known Exploited Vulnerabilities catalog, as explained by Corgea (19 Nov) and the Cloud Security Alliance (20 Nov).

Amazon’s own conclusion is the key point: none of the techniques were new. The attacks worked where basic hygiene was missing, such as default credentials, keys left in files, unauthenticated inference endpoints and unpatched software. That is exactly what a gateway and a config auditor can fix.

The agents succeeded where basic security hygiene was missing, including default or hardcoded credentials, keys left in code and public artifacts, unauthenticated interfaces and inference endpoints, and unpatched software. (AWS Security Blog, 13 Nov 2026)

What Is a Hardened LLM Gateway?

A hardened LLM gateway is a thin proxy between your users and your model server. Clients talk to the gateway with their own personal key. The gateway checks the key, applies limits, forwards only safe routes to Ollama or LiteLLM using its own private upstream key, and writes one audit line per request. The model server itself has no public port at all.

Think of it as the security guard at the gate. Ollama has no login of its own, and LiteLLM and Open WebUI are powerful admin tools, so they should live behind the guard, not on the open internet.

Problem in a typical self-hosted stackWhat the gateway does
Ollama port 11434 open to the internet, no authOllama stays on a private network; only the gateway can reach it
Everyone shares one master key like sk-1234Personal keys per user, HMAC-SHA256 hashed at rest, shown once
Admin APIs (/key/*, /api/pull, /guardrails/*) reachableOnly chat, embeddings and model list are served; the rest is 404 and logged
No idea who used whatHash-chained JSONL audit log with user, model, tokens and latency
A leaked key can burn money all nightRequests-per-minute limit, daily token budget, revoke and rotate in one command
Weak secrets copied from tutorialsThe gateway refuses to start with default or weak secrets

Features of the Hardened LLM Gateway Project

  • Per-user API keys in the format lgw_<id>_<secret>, stored only as HMAC-SHA256 with a server-side pepper. Expiry, revoke and rotate included.
  • No default keys. A fresh install has zero keys, no master key and no admin password.
  • Start-up preflight that refuses known defaults (sk-1234, changeme, t0p-s3cr3t), short or low-entropy secrets, a world-readable key file and public bind addresses without an explicit opt-in.
  • Rate limiting with a token bucket per key, a daily token budget, and a temporary IP lock-out after repeated failed logins.
  • Per-key model allow-lists, a max_tokens cap and a request size limit, which also helps against OWASP’s “unbounded consumption” risk.
  • Route allow-list: Ollama model management and LiteLLM admin endpoints are never forwarded.
  • JSONL audit log with a SHA-256 hash chain, no keys and no prompt text, plus logs verify and logs report detections.
  • Config auditor for docker-compose, LiteLLM config.yaml, Open WebUI .env and Ollama systemd units, with a 0-100 hardening score and CI-friendly exit codes.
  • Read-only exposure self-check for your own hosts, a CLI, a Flask admin UI and a hardened docker-compose stack.
Hardened LLM gateway admin dashboard with start-up checks, detections, per-user API keys and the JSONL audit log
The admin dashboard: start-up checks, live detections, per-user keys with budgets, and the audit log (no keys or prompts are stored).

Tech Stack and Requirements

PartUsed for
Python 3.10+Gateway, CLI, auditor (standard library for crypto, hashing and HTTP)
FlaskGateway routes and the small admin UI
PyYAMLParsing docker-compose and LiteLLM config files safely
Ollama (optional)Real open model, e.g. qwen2.5-coder:3b; LiteLLM or vLLM also work
pytest190 automated tests
Docker Compose (optional)Hardened reference stack with Open WebUI

You do not need a GPU. The built-in mock model runs the demo and every test on any laptop.

Project Structure

Hardened-LLM-Gateway-Python/
├── llm_gateway/
│   ├── gateway.py        # Flask app: auth, limits, allow-lists, forwarding, admin UI
│   ├── keys.py           # per-user keys, HMAC-SHA256 at rest, expiry, revoke, rotate
│   ├── secrets_check.py  # default/weak secret detection, provider-key patterns
│   ├── settings.py       # env settings + preflight (refuses to start on ERROR)
│   ├── ratelimit.py      # token bucket, daily token budget, IP lock-out
│   ├── audit.py          # hash-chained JSONL audit log + detections
│   ├── auditor.py        # LiteLLM / Open WebUI / Ollama / compose config auditor
│   ├── exposure.py       # read-only self-check of your own endpoints
│   ├── upstream.py       # OpenAI-compatible HTTP upstream + offline mock
│   └── cli.py, demo.py, wsgi.py, templates/, static/
├── examples/insecure/    # deliberately bad configs for the auditor
├── examples/secure/      # hardened docker-compose + LiteLLM config
├── tests/                # 190 pytest tests
└── Dockerfile            # non-root image

How the Hardened LLM Gateway Works

Every request goes through the same pipeline, and every step can stop it:

  1. IP lock-out: too many failed keys from one address returns 429 for 15 minutes.
  2. Route allow-list: only GET /v1/models, POST /v1/chat/completions and POST /v1/embeddings exist. Everything else is a logged 404.
  3. Key check: the key comes from the Authorization: Bearer header (never from the URL) and is compared as an HMAC in constant time.
  4. Limits: requests per minute and the daily token budget for that key.
  5. Validation: JSON body, model allow-list for that key, max_tokens clamped to the cap.
  6. Forward: the gateway calls the model with its own upstream key. The client’s header is never forwarded.
  7. Audit: one JSON line with user, model, status, tokens, latency and a short prompt hash.
client (curl / OpenAI SDK / Open WebUI)
   │  Authorization: Bearer lgw_<id>_<secret>
   ▼
 hardened gateway: IP lock-out → route allow-list → key HMAC → rate limit/budget
                   → JSON + model allow-list + max_tokens cap → forward → audit
   │  gateway's own upstream key (never sent to clients or logs)
   ▼
 Ollama / LiteLLM / vLLM on a private network (no published ports)

Step 1: Per-User API Keys Hashed at Rest

A key is created with Python’s secrets module and shown exactly once. Only an HMAC-SHA256 of it, keyed with a pepper from the environment, is written to keys.json (mode 600). If someone copies the key file, they still cannot use or reverse the keys without the pepper.

def verify(self, presented: str | None, now: float | None = None) -> tuple[ApiKey | None, str]:
    """Return (key, "ok") or (None|key, reason). Constant-time hash compare."""
    self.refresh()
    if not presented:
        return None, "missing_key"
    m = KEY_RE.match(presented.strip())
    if not m:
        return None, "malformed_key"
    rec = self._keys.get(m.group(1))
    expected = rec.key_hash if rec else hash_key(self._pepper, "decoy")
    ok = hmac.compare_digest(expected, hash_key(self._pepper, presented.strip()))
    if not rec or not ok:
        return None, "unknown_key"
    st = rec.status(now)
    if st != "active":
        return rec, f"{st}_key"
    rec.last_used = time.time() if now is None else now
    return rec, "ok"

Two small details matter. The comparison uses hmac.compare_digest, so timing does not leak how many characters matched, and an unknown key ID still runs a decoy hash for the same reason. The running gateway also reloads the file when the CLI changes it, so a revoked key stops working at once.

Step 2: Refuse to Start with Default or Weak Secrets

Copy-pasted secrets are the easiest door for attackers. The checker rejects values from docs and tutorials, placeholders, short strings, repeated patterns and low-entropy values:

def secret_problems(value: str | None, min_length: int = MIN_LENGTH) -> list[str]:
    """Return a list of human-readable reasons why *value* is unsafe.

    An empty list means the secret looks strong enough.
    """
    if value is None or value.strip() == "":
        return ["is empty or not set"]
    v = value.strip()
    low = v.lower()
    problems: list[str] = []
    if low in KNOWN_DEFAULTS:
        problems.append("is a well-known default value")
    elif any(h in low for h in PLACEHOLDER_HINTS):
        problems.append("looks like a placeholder from an example file")
    if re.fullmatch(r"(sk-)?\d+", low):
        problems.append("contains only digits")
    if len(v) < min_length:
        problems.append(f"is too short ({len(v)} chars, need at least {min_length})")
    if len(set(v)) < 10:
        problems.append("uses too few distinct characters")
    elif shannon_entropy(v) < 3.5:
        problems.append("has low entropy (looks guessable)")
    if re.search(r"(.{1,4})\1{4,}", v):
        problems.append("contains a repeating pattern")
    return problems

The preflight runs this on the pepper, the admin token and any upstream key, and it also refuses a world-readable key file or a 0.0.0.0 bind without GATEWAY_ALLOW_PUBLIC_BIND=1. If anything is an ERROR, serve exits before opening a port.

Hardened LLM gateway refusing to start with sk-1234 and blocking Ollama /api/pull and LiteLLM /key/generate
The demo: the gateway refuses to start with sk-1234, rejects unknown keys and returns 404 for model-management and key APIs.

Step 3: Rate Limits, Token Budgets and Route Allow-Lists

Each key has its own token bucket (for example 30 requests per minute) and a daily token budget. Unknown routes never reach the model, which matters because attackers specifically look for model-management and key endpoints:

ok, retry = rl.allow(rec.key_id, rec.rpm)
if not ok:
    log.write("rate_limited", **base, retry_after=retry)
    r = _err(429, f"rate limit of {rec.rpm} requests/minute exceeded", "rate_limited")
    r.headers["Retry-After"] = str(max(1, int(retry + 0.999)))
    return r
if tb.remaining(rec.key_id, rec.daily_tokens) <= 0:
    log.write("budget_exceeded", **base, daily_tokens=rec.daily_tokens)
    return _err(429, "daily token budget used up", "budget_exceeded")
payload = request.get_json(silent=True)
if not isinstance(payload, dict):
    log.write("bad_request", **base, reason="body is not a JSON object")
    return _err(400, "body must be a JSON object", "bad_request")
model = payload.get("model")
if not isinstance(model, str) or model not in allowed_models(rec):
    log.write("model_denied", **base, model=str(model)[:80])
    return _err(403, f"model {model!r} is not allowed for this key", "model_denied",
                allowed=allowed_models(rec))

The gateway also caps max_tokens, limits the body size and sets user to the key owner so upstream logs can show who asked. These controls map to OWASP’s LLM10:2025 Unbounded Consumption risk.

Step 4: A Tamper-Evident JSONL Audit Log

Every event, allowed or denied, becomes one JSON line. Each line stores the SHA-256 of the previous line, so editing, deleting or re-ordering lines breaks the chain. Sensitive field names such as api_key, prompt and messages are refused by the logger itself.

{"event": "request", "user": "pardeep", "key_id": "a2f1ca238b56", "model": "qwen2.5-coder:3b",
 "status": 200, "max_tokens": 160, "prompt_tokens": 46, "completion_tokens": 123, "total_tokens": 169,
 "latency_ms": 5485, "prompt_sha256": "484ecc91192ec2cc", "ip": "127.0.0.1",
 "time": "2026-11-21T03:47:10Z", "prev": "2bcd55f1…"}

logs report turns these lines into findings with a next step. It flags key spraying from one IP, probes for management paths, bursts of max_tokens=1 requests (the quiet key-check pattern Google described), one key used from many IPs, revoked keys still in use and spend close to the budget.

for (kid, ip), n in probes.items():
    if n >= probe_threshold:
        findings.append({"severity": "HIGH", "rule": "D3-key-validation", "subject": kid,
                         "detail": f"{n} requests with max_tokens<=1 for key {kid} ({users.get(kid, '?')}) from {ip}",
                         "advice": "Tiny single-token requests are a known pattern for checking whether a stolen key works. Rotate this key and ask the owner."})
Hardened LLM gateway audit log verify and report: key spraying, management probes and max_tokens=1 key checks
logs verify checks the hash chain; logs report turns raw events into findings with a next step.

Step 5: Config Auditor for LiteLLM, Open WebUI and Ollama

Most stacks are already deployed, so the auditor reads what you have and tells you what to fix. It is read-only, never connects anywhere and works on a folder, a single file or pasted text in the admin UI.

RuleSeverityWhat it finds
LGW-L01HIGHDefault or weak LiteLLM master key such as sk-1234
LGW-L02HIGHLiteLLM port 4000 (admin UI and key APIs) on all interfaces
LGW-L04HIGHLiteLLM image older than 1.84.0 (CVE-2026-40217 and the /key/regenerate fix)
LGW-L07HIGHProvider API key written in config.yaml
LGW-O03HIGHOllama 11434 published on all interfaces
LGW-W01 / W02 / W04HIGHWEBUI_AUTH=False, weak WEBUI_SECRET_KEY, DEFAULT_USER_ROLE=admin
LGW-W03 / O01 / O02MEDIUMOpen sign-up, OLLAMA_HOST=0.0.0.0, OLLAMA_ORIGINS=*
LGW-D01 / D02MEDIUM / HIGHWeak database password, Postgres or Redis published
LGW-S01HIGHOpenAI, Anthropic, AWS, Google, Groq or Hugging Face keys in any scanned file

Ports are parsed in both short ("11434:11434") and long syntax. A port bound to 127.0.0.1 is fine; one with no host IP is published on every interface:

for ip, pub, target in _ports(svc):
    public = _is_public(ip)
    where = f"{ip or '0.0.0.0'}:{pub}->{target}"
    if not public:
        continue
    if kind == "ollama" or target == "11434":
        f.append(Finding("HIGH", "LGW-O03", "Ollama API published on all interfaces", fname,
                         _line_of(text, f"{pub}:{target}") or _line_of(text, name), f"service '{name}' publishes {where}. Ollama has no authentication, so anyone who can reach it can run, pull, create or delete models.",
                         "Remove the ports entry (other containers reach it over the internal network) or bind 127.0.0.1:11434:11434."))

Settings follow the official docs: LiteLLM’s master key and virtual keys, Open WebUI’s environment variables (WEBUI_AUTH, WEBUI_SECRET_KEY, ENABLE_SIGNUP) and the Ollama FAQ on OLLAMA_HOST and OLLAMA_ORIGINS.

Advertisement
Hardened LLM gateway config auditor flagging default LiteLLM master key, exposed Ollama port and WEBUI_AUTH=False
The config auditor finds 13 problems in a deliberately insecure LiteLLM + Open WebUI + Ollama docker-compose file.

Step 6: Read-Only Exposure Self-Check

Config files can lie, so the last check looks at the running service. check-exposure sends a few unauthenticated GET requests (model list, version, public config) to your own host and reports whether anything answers without a key. It never sends credentials, never calls write or management endpoints and never guesses passwords.

Read-only exposure self-check: raw Ollama on port 11434 is open, the hardened LLM gateway is protected
A read-only self-check on the same machine: raw Ollama answers anyone, the gateway answers only with a key.

On our test box, raw Ollama on port 11434 listed its models to anyone, while the gateway on port 8088 returned 401 or 404 for every probe.

How to Run the Hardened LLM Gateway (Step by Step)

git clone https://github.com/GetProjectsIdea/Hardened-LLM-Gateway-Python.git
cd Hardened-LLM-Gateway-Python
python -m venv .venv && source .venv/bin/activate     # Windows: .venv\Scripts\activate
pip install -r requirements-dev.txt

python -m llm_gateway demo                # offline walkthrough with a mock model
python -m pytest -q                       # 190 passed

python -m llm_gateway init                # .env with random pepper + admin token (chmod 600)
python -m llm_gateway keys create alice --rpm 30 --daily-tokens 200000
python -m llm_gateway serve               # http://127.0.0.1:8088 -> Ollama 127.0.0.1:11434/v1

Then point any OpenAI-compatible client at the gateway:

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8088/v1", api_key=ALICE_KEY)
reply = client.chat.completions.create(model="qwen2.5-coder:3b",
                                       messages=[{"role": "user", "content": "Hello"}])
print(reply.choices[0].message.content)

Audit an existing stack and check the log:

python -m llm_gateway audit-config ./my-llm-stack           # exit code 1 on HIGH findings
python -m llm_gateway audit-config ./my-llm-stack --json    # for CI pipelines
python -m llm_gateway check-exposure http://127.0.0.1:11434 # your own host only
python -m llm_gateway logs verify && python -m llm_gateway logs report

Hardened Docker Compose Stack

The examples/secure folder runs Ollama, the gateway and Open WebUI together. Ollama has no published ports, Open WebUI talks to models only through the gateway using its own key, and the gateway container is read-only, non-root and drops all Linux capabilities. Required secrets use ${VAR:?message}, so Compose refuses to start if one is missing.

  gateway:
    build: ../..
    networks: [llm]
    ports:
      - "127.0.0.1:8088:8088"          # loopback only; publish via a TLS reverse proxy
    environment:
      GATEWAY_KEY_PEPPER: "${GATEWAY_KEY_PEPPER:?run llm_gateway init}"
      GATEWAY_UPSTREAM_URL: http://ollama:11434/v1
    read_only: true
    cap_drop: [ALL]
    security_opt: ["no-new-privileges:true"]

Running the auditor on this folder gives a score of 100/100, while the insecure example scores 0/100 with 13 HIGH findings.

Testing the Hardened LLM Gateway with a Real Ollama Model

The mock model is great for tests, but we also ran the gateway in front of a real open model: qwen2.5-coder:3b on Ollama 0.40.16, CPU only. Requests without a key and with sk-1234 got 401, /api/pull got 404, and the personal key got a code answer in about 5 to 14 seconds. The audit line recorded tokens and latency but not the prompt.

Real qwen2.5-coder 3B model answering through the hardened LLM gateway via Ollama on a CPU
A real open model (qwen2.5-coder:3b via Ollama, CPU only) answering through the gateway, plus the matching audit line.

The repository has 190 pytest tests that run in under a second. They cover weak and default secrets, key hashing and file permissions, expiry, revoke and rotate, live reload of revocations, rate limits and budgets, IP lock-out, 10 blocked management paths, body and max_tokens limits, the audit hash chain (edit, delete and re-order), every detection, every auditor rule, the exposure check, the admin UI (login, CSRF, shown-once keys) and the CLI. They also pass from a fresh clone.

Hardening Checklist for Self-Hosted LLMs

  • Never publish Ollama (11434), LiteLLM (4000), Postgres or Redis on 0.0.0.0. Use private Docker networks.
  • Replace every default secret: LiteLLM master key, UI_PASSWORD, WEBUI_SECRET_KEY, database passwords.
  • Give each person or app its own key with a budget, and revoke keys when people leave.
  • Keep provider keys out of config files and Git. Load them from a secret store and set spend limits at the provider.
  • Patch LiteLLM to 1.84.0 or later and rotate the master key and virtual keys afterwards.
  • Keep Open WebUI login on, sign-up off and the default role at pending or user.
  • Put TLS in front of anything users reach, and watch for unexpected spend or unusual usage, as Amazon recommends.

Limitations and Responsible Use

This hardened LLM gateway is a teaching project, not an audited product. Rate limits and budgets are in memory per process, so use one worker or move them to Redis. Streaming requests are answered as one SSE chunk because the gateway counts tokens on the full response. A gateway does not replace patching: keep LiteLLM, Open WebUI and Ollama updated.

The project is strictly defensive. The insecure examples contain only well-known default values and a fake key, and check-exposure must only be used on systems you own or are authorised to test.

Future Scope and Viva Tips

  • Move limits and budgets to Redis and add per-team quotas with email or Slack alerts.
  • Add OIDC single sign-on for the admin UI and short-lived keys.
  • Export metrics to Prometheus and forward the JSONL log to a SIEM.
  • Add SARIF output so the auditor shows findings in GitHub code scanning.
  • Combine it with a prompt-injection filter in front of the model.

This is day 4 of our daily series. Day 1 was an AI coding agent security scanner in Python, day 2 a third-party script monitor in Python and day 3 a sandboxed coding agent in Python that asks permission before every command. You can pair this gateway with our prompt injection firewall in Python and check your dependencies with the Python supply-chain security scanner. For research angles such as LLM abuse detection, see AI PhD thesis topics 2026, or browse 50 major project ideas for BTech and MCA.

FAQs: Hardened LLM Gateway

Is this hardened LLM gateway a good final-year project?

Yes. It combines API security, hashing, rate limiting, log analysis, config parsing, a Flask web app, Docker and automated testing, and it is based on attacks reported in November 2026.

Do I need a GPU or Ollama to try it?

No. The offline mock model runs the demo and all 190 tests on any laptop. Ollama with a small model such as qwen2.5-coder:3b also works on a CPU.

Does it work with LiteLLM and Open WebUI?

Yes. Point GATEWAY_UPSTREAM_URL at LiteLLM with a strong virtual key, or point Open WebUI at the gateway as an OpenAI-compatible connection with its own key. The auditor checks LiteLLM, Open WebUI and Ollama configs.

Why are API keys hashed with HMAC instead of bcrypt?

The keys are long random values, not human passwords, so a fast keyed hash with a secret pepper is enough and keeps every request fast. Without the pepper, a stolen key file is useless.

Is there a default admin key or password?

No. A fresh gateway has zero keys, and the admin UI is disabled unless you set a strong GATEWAY_ADMIN_TOKEN. The gateway refuses to start with values like sk-1234 or changeme.

Is the source code free?

Yes. The full code is on GitHub under the MIT License.

References


Download the Source Code

The complete hardened LLM gateway project, with the gateway, CLI, Flask admin UI, config auditor, exposure check, Docker files, README, screenshots and 190 tests, is free on GitHub. Clone it, star it and open an issue if you get stuck.

Conclusion

Attackers now use AI agents to find exposed AI gateways and drain their keys, but the doors they use are old ones: default keys, open ports and secrets in files. In this tutorial you built a hardened LLM gateway in Python that closes those doors with personal hashed keys, start-up checks, limits, an audit log and a config auditor.

Run the auditor on your own stack today, fix the HIGH findings first, and put the gateway in front of your models. Built something on top of it? Tell us in the comments below.

Advertisement

Team Get Projects

Contributor & Author
View All Projects →

Contributor at GetProjects.org, sharing open-source codebases, tutorials, and practical software architectures.

0 Discussion Comments

Markdown Supported

Join the Discussion & Code Review

Share architectural feedback, benchmark results, or troubleshooting notes.