SOFTIQ x SpaceNet - Building a sovereign LLM infrastructure

Описание: In this video, Maja von Bomhard, Business Development Manager of the Munich-based IT service provider SpaceNet, and Adam Marszowski, an AI expert from SOFTIQ, discuss their collaborative project to build a high-performance LLM (Large Language Model) infrastructure.

The project was driven by a specific need in the German market: small and medium-sized companies often possess valuable, specialised data and seek data sovereignty. These businesses are often reluctant to rely on public hyperscalers and instead prefer local data centre solutions where their data remains secure in a "safe haven".

Key highlights of the technical discussion include:

🔹 The technical stack: Adam explains the choice of vLLM as the inference backend for its efficiency, alongside LiteLLM as an API gateway to handle authorisation and rate limiting.

🔹 Hardware Power: the infrastructure utilizes a multi-GPU setup featuring NVIDIA H100 GPUs to ensure high performance.

🔹 Sovereign Generative AI: unlike standard cloud-based AI, this project focuses on on-premise deployment using open-source models like GPT-OSS 120B, which are specifically tested for high performance in the German language.

🔹 Real-world applications: the infrastructure already supports practical use cases, including document analysis within Nextcloud, integration with automation platforms like Windmill, and RAG (Retrieval-Augmented Generation) architectures.

Maja shares why she chose SOFTIQ as a long-term partner, citing shared values of transparency and commitment, while Adam provides insights into the rapidly evolving landscape of AI models and the importance of staying at the forefront of technological research.