Every day, employees search for answers that were written down somewhere in the company long ago: in policies, manuals and meeting notes. A corporate GPT answers such questions from the company's own documents, cites its sources and runs on language models hosted in the EU. This post in the series "Projects in detail" describes the structure and architecture of such a system and a method for calculating its value.
Starting point
In many companies, knowledge is scattered across intranet pages, file shares and PDF manuals. Anyone with a question searches for a long time or calls someone who might know. Simple questions end up tying down exactly the specialists who are needed for other work.
Without an approved tool, there is a risk that employees use private AI accounts for work texts. This is called shadow AI. Company content then ends up with providers that have no contract with the company, and nobody knows which data goes where. A ban alone does not fix this: a better alternative is needed.
Goal
A corporate GPT solves both problems at once. It replaces private chat accounts with a secure tool and opens direct access to the company's own knowledge. That leads to five requirements:
- One chat for all employees, signed in with their existing company account via single sign-on (SSO).
- Answers from internal documents with a reference to the source, so every statement can be checked.
- Interchangeable language models, without rebuilding the interface.
- Costs that are visible and can be capped.
- Data processed in the EU.
Training a custom model is almost never part of it. For questions about documents, a good base model with retrieval-augmented generation (RAG) is enough: for each question, the model looks up the relevant passages instead of memorising them.
Architecture
The system consists of four layers: interface, knowledge, gateway and models. Each one can be replaced on its own.
- LibreChat (librechat.ai): the open-source chat interface. It handles sign-in, conversation history, file uploads and model selection.
- Single sign-on: sign-in through the company's existing identity provider. Anyone who leaves the company loses access to the chat together with their company account.
- RAG service with a vector database: splits approved documents into sections, saves them as vectors and returns the relevant passages for each question.
- LiteLLM (docs.litellm.ai) as the model gateway: one interface to all models, with routing, budgets and logging.
- Language models in the EU: for example Azure OpenAI in EU regions and models from Mistral.
- Reporting: usage and cost, taken straight from the gateway data.
Implementation
Implementation starts with a small pilot group from several departments and with real questions from their daily work. Most of the effort goes into the documents, not the chat. Outdated versions, duplicate manuals and scanned PDFs without a text layer produce poor answers. Before import, there is a clean-up: one valid version per document, one person responsible for it and a date for the next review.
The second ground rule concerns permissions. The chat may only know what the person asking is allowed to read. It makes sense to start with documents that are already open to all employees. Areas with tighter access are added later as separate knowledge collections that only certain groups can see.
The gateway can issue virtual keys, for example per department or per application, each with its own budget. If a model fails or is overloaded, LiteLLM switches to a fallback model. To LibreChat, all of this looks like a single OpenAI-compatible endpoint. Adding a new model therefore means one entry in the gateway, not a rebuild of the interface.
Every answer names the documents it comes from. That builds trust and makes errors visible.
A tool without onboarding stays empty. Short training sessions, a page with example questions and contact people in each department are therefore part of the project. Since February 2025, Article 4 of the AI Act has required companies to take measures that promote AI literacy among their staff anyway.
Infrastructure and EU compliance
With Azure OpenAI, the deployment type decides where requests are processed. "Global" deployments may process requests in any region where the model is available, so they are ruled out. For "Data Zone" deployments in an EU region, processing stays within EU member states, according to Microsoft's documentation. Mistral is a European provider based in Paris and works well as the second model family in the gateway.
LibreChat, the vector database and LiteLLM are best run in a dedicated environment in an EU region, with the company's cloud provider or a European hosting provider. The gateway logs who uses which model and how much. These metadata are enough for cost reporting, and LiteLLM can be configured not to record conversation content.
Two points need to be settled before launch. First, the classification under the AI Act: an internal chat for knowledge questions is not a high-risk system. It only stays that way as long as it makes no decisions about staff; Annex III explicitly lists such uses as high-risk. The usage policy therefore rules them out. Second, co-determination: a chat system processes employee data. In Austria, a company with a works council usually needs its consent under sections 96 and 96a of the Labour Constitution Act, typically as a works agreement. Involving the works council early avoids a halt shortly before launch.
Outcome and value calculation
The result is a chat that employees open with their company account and that answers questions from approved documents, with sources. Models can be swapped in the gateway without anything changing for users. Private AI accounts for work texts lose their reason to exist.
The value can be calculated with a fixed method and without blanket assumptions:
- Time saved = questions answered per week × search minutes saved per question × internal hourly rate.
- Relief for specialist departments = queries that no longer arrive by phone or email × handling time per query × internal hourly rate.
- Costs = model usage according to the gateway log + platform operation + document maintenance.
- Net value = time saved + relief − costs.
Search minutes should be measured, not estimated. Before launch, a test group answers a sample of real questions the old way and times it, then answers the same questions with the chat. One benefit cannot be expressed in money and still counts: company knowledge no longer leaks out through private accounts.
You can find more projects in the overview of reference projects.