Solving the AI problem by using our AI in your Bank

Banks have valid concerns when it comes to adopting AI.
We solve this by putting an AI Server inside your VPN, running your Windows or Linux with your security, in your non-production environment (dev and test).
This means that AI does not bring a security risk, rather it brings the opportunity to improve security.
In other words if your current test data practices result in customer data being used for testing, our AI can flag that issue.
A quick explanation of “ChatGPT AI”
It is an AI tool by Open AI that understands questions and generates human-like answers. It is powered by a Large Language Model (LLM) which has learned patterns from massive amounts of data, the primary source being the internet. I think of an LLM as a mathematical model of how my brain works. This LLM is hosted in the public domain for anyone to access. Thus the legitimate concern is that your data is leaked into the big wide world of the internet and millions of users. Small Language Models are a subset trained on a specific domain. In this case the domain is the coding, testing and the generic business of banking, such as Cross Border Payments and Liquidity Management.
Retrieval Augmented Generation is a technique that improves the AI answers with documents and data of the context of the project. The basic process is to convert documents and data (called embedding), then store that converted data (a vector) in a special database– The Knowledge Base. Last step is to create a knowledge graph. (If you must know – We use a vector and knowledge graph database😊).
The AI isn’t trained in the traditional data science sense - instead, it’s connected to your bank’s trusted documents, policies, and standards through a secure Retrieval system. When someone asks a question, the AI finds the most relevant internal information and builds its answer from that. This means every response is traceable, explainable, and aligned with how your bank actually operates.
You ask a question, ‘what tests do I need for the charges codes and unstructured addresses ?’ Our Chat then finds the relevant information, RETRIEVES relevant information in The Knowledge Base, AUGMENTS your question with the information in the Knowledgebase and passes it to the LOCAL LM to GENERATE the answer. This is all packaged up on an ‘AI Server’.
How a Local AI Server Solves the problem of Banks using AI
to work better, smarter and with joy
The AI Server is secured with a combination of classic IT security measures (your security measures) to which we add AI specific measures tailored to the risk and regulatory demands of your context.
- Regulatory & Data Privacy
Keeps all data on-prem or within a private cloud, ensuring compliance - Security
No data leaves the bank’s network, reducing surface area for attacks - Model Trust & Explainability
Every response is traceable, explainable, and aligned with how your bank actually operates. - Data Sovereignty
Full control over where data and models are stored and processed - Vendor Flexibility
It is plug-and-play with Language Models from a trusted source and change when needed
(we use the trusted model of IBM Granite with full control) - Cost & Scalability
Avoid recurring API fees; scale horizontally on existing infrastructure

Example Use Case:
Our RAG Knowledge base has a Local LM to power an internal AI assistant.
We can query internal documents (e.g., ISO 20022 mapping rules, system architecture diagrams, policies) without exposing any data externally – and get explainable results grounded in trusted sources.
More on security – our research with ChatGPT™ and Perplexity™
In the context of a RAG pipeline with a Local LM, you’re not “training” the model in the machine learning sense (i.e. updating its weights). Instead, you’re using Retrieval-Augmented Generation to ground the model’s responses in specific, bank-owned content.
To securely operate a local large language model (LM) on a GPU server inside a bank’s firewall and VPN, you must apply a combination of AI-specific and classic IT security measures tailored to the risks and regulatory demands of financial environments. Here are the key measures, directly supported by current best practices and industry guidance:
1. Data Governance and Regulatory Compliance
- Data Classification: Rigorously classify all data processed by the LM (public, confidential, mission-critical). Only allow the LM to access data it is legally permitted to process, and ensure compliance with privacy laws (GDPR, banking secrecy, etc.)[3]. In our context this is managing the data that goes into the Knowledgebase. As we are in a non-production environment, the problem of customer data should have been addressed. Secondly we are curating the data and additional filters are applied. The data is typically coming from your project documents and the tools that are used in software delivery.
- Data Minimization & Anonymization: Where possible, anonymize or tokenize sensitive data before it is used by the LM to reduce the risk of unintentional disclosure[3].
- Documentation & Auditing: Maintain detailed records of data lineage, model updates, and access logs. This supports both internal governance and regulatory audits[3].
2. System and Network Security
- Isolated Environment: Host the LM server in a segregated network segment (e.g., a VLAN or subnet), accessible only from trusted internal systems. Even within the firewall, minimize exposure[2].
- Access Controls: Enforce strong authentication for any access to the LM server (e.g., SSO, role-based access, unique credentials for each user/service)[2].
- Transport Encryption: Use TLS (HTTPS) for all internal API calls to the LM, even if traffic never leaves the internal network. This prevents interception or tampering of sensitive prompts and responses[2].
- Firewall & VPN: Ensure the LM server is not exposed to the public internet. Restrict access to internal VPN users and only necessary systems[2].
3. Application and Model Security
- Secrets Management: Never hard-code credentials, API keys, or secrets in code or prompts. Store them securely (e.g., environment variables, secrets manager) and inject at runtime as needed[2].
- Least Privilege Principle: Run the LM process with the minimum permissions required. Disable unnecessary services and ports on the server. If outbound internet access is not needed, block it to prevent data exfiltration in case of compromise[2].
- System Hardening: Keep the OS, GPU drivers, ML frameworks, and all dependencies up to date with security patches. Regularly scan for vulnerabilities and maintain an inventory of all components[2].
4. AI-Specific Risk Mitigations
- Guardrails & Output Filtering: Implement guardrails to prevent the LM from generating or exposing sensitive, regulated, or inappropriate content. This can include prompt engineering, output filtering, or rule-based overrides[3].
- Monitoring & Logging: Monitor all LM interactions for anomalous or unauthorized activity. Log all access and actions for forensic analysis and compliance[2][3].
- Model Update Governance: Document and control all model updates, including versioning and change management, to ensure traceability and regulatory alignment[3].
- Data Leakage Prevention: Be aware that LMs can memorize and inadvertently disclose sensitive data. Limit training on sensitive datasets and consider techniques like differential privacy or secure enclaves for highly sensitive use cases[3][4].
5. Operational and Organizational Controls
- Explicit Roles & Responsibilities: Assign clear roles for data usage authorization, audit management, and staff training to prevent accidental leaks or misuse[3].
- Staff Training: Regularly train staff on secure AI usage, data privacy, and incident response procedures[3].
Summary Table: Local LLM Security Checklist for Banking
Security Domain | Key Measures |
Data Governance | Classification, anonymization, documentation, regulatory compliance |
Network Security | Internal-only access, TLS encryption, firewall/VPN, isolated environment |
Application Security | Secrets management, least privilege, system hardening, patching |
AI-Specific | Guardrails, output filtering, monitoring, model update governance, data leakage prevention |
Organizational | Roles/responsibilities, staff training, audit trails |
In conclusion:
Running a local LM inside a bank’s firewall and VPN is a strong foundation for AI security. However, you must layer on strict data governance, robust network and application security, AI-specific controls (like guardrails and monitoring), and strong organizational processes to meet the high bar of banking security and regulatory expectations[2][3][5].
Citations: [1] https://www.ey.com/en_za/industries/financial-services/sa-banks-and-generative-ai [2] https://substack.com/home/post/p-161745444 [3] https://arxiv.org/html/2504.02165v1 [4] https://www.edpb.europa.eu/system/files/2025-04/ai-privacy-risks-and-mitigations-in-llms.pdf [5] https://www.getdynamiq.ai/post/generative-ai-and-llms-in-banking-examples-use-cases-limitations-and-solutions [6] https://blog.purestorage.com/purely-educational/how-to-secure-ai-and-model-data-with-storage-infrastructure/ [7] https://www.linkedin.com/pulse/top-10-policy-considerations-banking-when-implementing-ahson-pai [8] https://www.cloudsine.tech/llm-vulnerabilities-8-critical-threats-and-how-to-mitigate-them/ [9] https://papers.academic-conferences.org/index.php/eccws/article/download/2505/2127/8511 [10] https://arya.ai/blog/slm-vs-llm [11] https://www.americanbanker.com/news/how-banks-can-protect-the-data-they-feed-their-ai-models [12] https://www.bai.org/banking-strategies/limiting-the-risky-side-of-ai-in-financial-services/ [13] https://www.checkpoint.com/cyber-hub/what-is-llm-security/llm-security-best-practices/ [14] https://www.perimeter81.com/blog/cloud/cloud-vpn-and-ai [15] https://www.chitika.com/local-llm-rag-security/ [16] https://www.euromoney.com/reports/ai-in-banking-best-practices-playbook/ [17] https://www.paloaltonetworks.com/cyberpedia/ai-security [18] https://www.ey.com/en_gr/insights/financial-services/how-artificial-intelligence-is-reshaping-the-financial-services-industry [19] https://www.bis.org/publ/othp90.pdf [20] https://autogpt.net/the-intersection-of-ai-and-vpn-technology-protecting-your-digital-privacy/
Answer from Perplexity: pplx.ai/share