Example Use Case:

Our RAG Knowledge base has a Local LM to power an internal AI assistant.

We can query internal documents (e.g., ISO 20022 mapping rules, system architecture diagrams, policies) without exposing any data externally – and get explainable results grounded in trusted sources.

More on security – our research with ChatGPT™ and Perplexity™

In the context of a RAG pipeline with a Local LM, you’re not “training” the model in the machine learning sense (i.e. updating its weights). Instead, you’re using Retrieval-Augmented Generation to ground the model’s responses in specific, bank-owned content.

To securely operate a local large language model (LM) on a GPU server inside a bank’s firewall and VPN, you must apply a combination of AI-specific and classic IT security measures tailored to the risks and regulatory demands of financial environments. Here are the key measures, directly supported by current best practices and industry guidance:

1. Data Governance and Regulatory Compliance

• Data Classification: Rigorously classify all data processed by the LM (public, confidential, mission-critical). Only allow the LM to access data it is legally permitted to process, and ensure compliance with privacy laws (GDPR, banking secrecy, etc.)[3]. In our context this is managing the data that goes into the Knowledgebase. As we are in a non-production environment, the problem of customer data should have been addressed. Secondly we are curating the data and additional filters are applied. The data is typically coming from your project documents and the tools that are used in software delivery. 
• Data Minimization & Anonymization: Where possible, anonymize or tokenize sensitive data before it is used by the LM to reduce the risk of unintentional disclosure[3]. 
• Documentation & Auditing: Maintain detailed records of data lineage, model updates, and access logs. This supports both internal governance and regulatory audits[3].

2. System and Network Security

• Isolated Environment: Host the LM server in a segregated network segment (e.g., a VLAN or subnet), accessible only from trusted internal systems. Even within the firewall, minimize exposure[2].
• Access Controls: Enforce strong authentication for any access to the LM server (e.g., SSO, role-based access, unique credentials for each user/service)[2].
• Transport Encryption: Use TLS (HTTPS) for all internal API calls to the LM, even if traffic never leaves the internal network. This prevents interception or tampering of sensitive prompts and responses[2].
• Firewall & VPN: Ensure the LM server is not exposed to the public internet. Restrict access to internal VPN users and only necessary systems[2].

3. Application and Model Security

• Secrets Management: Never hard-code credentials, API keys, or secrets in code or prompts. Store them securely (e.g., environment variables, secrets manager) and inject at runtime as needed[2].
• Least Privilege Principle: Run the LM process with the minimum permissions required. Disable unnecessary services and ports on the server. If outbound internet access is not needed, block it to prevent data exfiltration in case of compromise[2].
• System Hardening: Keep the OS, GPU drivers, ML frameworks, and all dependencies up to date with security patches. Regularly scan for vulnerabilities and maintain an inventory of all components[2].

4. AI-Specific Risk Mitigations

• Guardrails & Output Filtering: Implement guardrails to prevent the LM from generating or exposing sensitive, regulated, or inappropriate content. This can include prompt engineering, output filtering, or rule-based overrides[3].
• Monitoring & Logging: Monitor all LM interactions for anomalous or unauthorized activity. Log all access and actions for forensic analysis and compliance[2][3].
• Model Update Governance: Document and control all model updates, including versioning and change management, to ensure traceability and regulatory alignment[3].
• Data Leakage Prevention: Be aware that LMs can memorize and inadvertently disclose sensitive data. Limit training on sensitive datasets and consider techniques like differential privacy or secure enclaves for highly sensitive use cases[3][4].

5. Operational and Organizational Controls

• Explicit Roles & Responsibilities: Assign clear roles for data usage authorization, audit management, and staff training to prevent accidental leaks or misuse[3].
• Staff Training: Regularly train staff on secure AI usage, data privacy, and incident response procedures[3].

Summary Table: Local LLM Security Checklist for Banking