Public AI has introduced a new inference platform designed to facilitate the use of sovereign large language models (LLMs) in compliance with data residency requirements. This initiative addresses the gap between the release of open-weight models by research institutions and the need for production-ready services, particularly in regions with stringent data regulations.
Bridging the Gap: While institutions like EPFL and ETH Zurich publish models, they do not provide the necessary infrastructure for organizations to utilize these models effectively. Public AI's Inference Utility (IU) aims to fill this void by deploying these models on Amazon EC2 instances powered by Intel processors, enabling organizations to access public inference endpoints without the burden of managing their own infrastructure.
Architecture Overview: The platform operates on Amazon Elastic Kubernetes Service (EKS) and is designed to scale efficiently. It allows users to send requests to the model and receive responses seamlessly, ensuring that the entire process stays within Swiss jurisdiction, which is crucial for sovereign AI deployments.
Key Features:
- Utilizes Intel Xeon processors for cost-effective performance.
- Supports a fully containerized architecture for rapid deployment.
- Offers both a user-friendly chat interface and an API for developers.
Since its launch, the Public AI Inference Utility has successfully managed thousands of concurrent users, demonstrating its capability to handle real-world demands while maintaining performance and security.
Future Prospects: With the success of the Apertus model, Public AI is poised to expand its services to include more sovereign LLMs across various jurisdictions. The architecture's flexibility allows for easy replication in new regions, catering to diverse regulatory and latency needs.
Conclusion: As the demand for sovereign AI solutions grows, Public AI's approach offers a viable blueprint for organizations looking to implement LLMs in compliance with local regulations. The collaboration with AWS and Intel positions Public AI to onboard new models efficiently and provide immediate access to usable API endpoints.
