Technical Deep Dive
AI Foundry's service is built around NVIDIA's Blackwell GPU architecture, a purpose-built inference accelerator that represents a generational leap over its predecessor, Hopper. The Blackwell B200 GPU features a dual-die design with 208 billion transistors, connected via a high-speed NVLink-C2C interconnect, delivering up to 20 petaFLOPS of FP4 inference performance. This architecture is specifically optimized for transformer-based models, incorporating second-generation Transformer Engine that dynamically manages precision between FP8 and FP4 to maximize throughput without sacrificing accuracy.
For inference serving, AI Foundry likely employs a multi-instance GPU (MIG) partitioning strategy combined with dynamic batching to maximize utilization across subscribers. The subscription model requires sophisticated rate-limiting and fair-share scheduling to prevent any single user from monopolizing resources. This is a significant engineering challenge: unlike per-token billing where each request is independently metered, a fixed-fee model must ensure quality of service (QoS) for all concurrent users while preventing abuse.
From a latency perspective, Blackwell's NVLink 5.0 provides 1.8 TB/s of bidirectional bandwidth per GPU, enabling efficient model parallelism for large LLMs. For models like Llama 3 70B or Mixtral 8x22B, tensor parallelism across multiple Blackwell GPUs can achieve sub-100ms time-to-first-token (TTFT) for prompts under 2,000 tokens. However, under sustained load from multiple subscribers, tail latency becomes a concern. AI Foundry must implement aggressive request queuing and preemption mechanisms to maintain consistent performance.
| Metric | Blackwell B200 (FP4) | H100 SXM (FP8) | Improvement |
|--------|---------------------|----------------|-------------|
| Peak TFLOPS | 20,000 | 1,979 | 10.1x |
| Memory Bandwidth | 8 TB/s | 3.35 TB/s | 2.4x |
| TDP | 700W | 700W | Same |
| NVLink Bandwidth | 1.8 TB/s | 900 GB/s | 2x |
| Recommended Model Size | Up to 1T params | Up to 175B params | — |
Data Takeaway: Blackwell's FP4 performance advantage is dramatic, but real-world inference throughput depends on model quantization support and batching efficiency. The 10x peak TFLOPS figure is theoretical; practical gains for production LLM serving are likely 3-5x over H100, depending on workload.
A key open-source reference point is the vLLM project (GitHub: vllm-project/vllm, 45k+ stars), which provides a high-throughput serving engine with PagedAttention for efficient KV cache management. AI Foundry could be using a customized fork of vLLM or similar infrastructure (e.g., TensorRT-LLM) to handle the subscription model's dynamic load. The PagedAttention algorithm reduces memory fragmentation by up to 95%, which is critical for maximizing concurrent user capacity on fixed GPU memory.
Key Players & Case Studies
AI Foundry itself is a relatively small player in the AI infrastructure space, headquartered in New Zealand with a focus on sovereign AI capabilities. The company has previously offered GPU rental services but this subscription model is its most disruptive move. The choice of New Zealand is strategic: the country has a growing AI startup ecosystem (e.g., Soul Machines, Orion Health) and relatively low energy costs, making it a viable location for data center operations.
The primary competitive landscape includes:
- Together AI: Offers serverless inference with per-token pricing but recently introduced a "dedicated endpoint" subscription for high-volume users. Their pricing for Llama 3 70B is ~$1.20 per million tokens.
- Fireworks AI: Provides fast inference with a pay-as-you-go model, targeting latency-sensitive applications. They have not adopted flat-rate pricing.
- Groq: Uses custom LPU hardware for ultra-low latency, but charges per token. Their hardware is not available for subscription-style unlimited use.
- Replicate: Offers a mix of per-token and per-second pricing for community models, but no unlimited tier.
| Provider | Pricing Model | Base Hardware | Latency (Llama 3 70B, TTFT) | Cost for 10M tokens/day |
|----------|---------------|---------------|----------------------------|-------------------------|
| AI Foundry | Fixed monthly (~$5,000 est.) | Blackwell B200 | <100ms (claimed) | $5,000 flat |
| Together AI | $1.20/1M tokens | H100 | 150-200ms | $12,000 |
| Fireworks AI | $0.90/1M tokens | H100 | 120-180ms | $9,000 |
| Groq | $0.60/1M tokens | LPU | <10ms | $6,000 |
Data Takeaway: For users generating over ~8 million tokens per day, AI Foundry's subscription model becomes cheaper than per-token alternatives. However, Groq's LPU offers superior latency, which may be critical for real-time applications like voice assistants or autonomous agents.
A notable case study is the developer community on platforms like Hugging Face, where thousands of models are tested daily. A single developer running multiple experiments could easily burn through $500-$1,000 per month on per-token APIs. AI Foundry's subscription would enable unlimited experimentation, potentially accelerating model evaluation and fine-tuning workflows.
Industry Impact & Market Dynamics
The shift from variable to fixed pricing for AI inference mirrors the evolution of cloud computing. AWS, Azure, and GCP all started with pay-as-you-go compute but later introduced reserved instances and savings plans. AI Foundry is essentially offering the equivalent of a reserved instance for inference, but with unlimited usage—a more aggressive version of the model.
This pricing innovation could have several second-order effects:
1. Democratization of experimentation: Fixed costs remove the fear of runaway bills, encouraging developers to build more AI-native applications. This could accelerate the adoption of LLMs in non-traditional sectors like education, healthcare, and government.
2. Commoditization of inference: If successful, this model pressures hyperscalers to offer similar plans. AWS already offers SageMaker Inference with per-hour pricing, but not unlimited. Google Cloud's Vertex AI has per-node pricing but with throughput limits.
3. GPU utilization arbitrage: The subscription model works best when average utilization is high. AI Foundry is betting that the aggregate usage of its subscribers will be predictable enough to plan capacity. This is similar to the airline industry's overbooking strategy—some users will underutilize, offsetting heavy users.
| Metric | Current Market (2025) | Projected (2027) | Source |
|--------|----------------------|------------------|--------|
| Global LLM inference market size | $12.5B | $45.8B | Industry estimates |
| Subscription-based inference share | <1% | 15-20% | AINews projection |
| Average API cost per 1M tokens (Llama 3 70B) | $1.00 | $0.40 | Expected decline |
| Number of active LLM developers | 8M | 25M | Developer surveys |
Data Takeaway: The inference market is growing rapidly, and subscription models could capture a significant share if they prove sustainable. The projected decline in per-token costs suggests that fixed-fee models will need to continuously adjust pricing to remain competitive.
New Zealand's role as a testbed is notable. The country has a population of just 5 million but a high density of AI researchers per capita, partly due to government initiatives like the AI for Good program. AI Foundry can iterate on its pricing and infrastructure without the intense competitive pressure of Silicon Valley. If the model works, expansion to Australia, Singapore, and eventually the US is likely.
Risks, Limitations & Open Questions
Performance Degradation Under Load: The most immediate risk is that unlimited subscribers will overload the system. Unlike per-token pricing where demand is naturally throttled by cost, a flat fee incentivizes maximum usage. AI Foundry must implement fair-use policies or rate limits, which would undermine the "unlimited" promise. Early adopters report that inference speed can drop by 30-50% during peak hours, suggesting that the current infrastructure may not scale linearly with subscriber growth.
Adverse Selection: The subscription model attracts heavy users who generate high costs, while light users may avoid it. This creates a classic adverse selection problem where the average cost per user exceeds the subscription price. AI Foundry needs a diverse user base to balance the economics, but marketing to light users is challenging when per-token alternatives exist.
Hardware Lock-In: By committing to Blackwell GPUs, AI Foundry is betting on NVIDIA's roadmap. If AMD's MI350 or Intel's Falcon Shores offer better price-performance for inference, the company could be at a competitive disadvantage. The subscription model makes it harder to switch hardware without disrupting existing customers.
Regulatory and Data Sovereignty: Operating data centers in New Zealand means data must comply with local privacy laws (e.g., Privacy Act 2020). For global customers, this may raise concerns about data residency and latency. AI Foundry has not disclosed plans for multi-region deployment.
Open Question: What defines "unlimited"? The terms of service likely include a fair-use clause, but the specifics are opaque. If users are limited to a certain number of requests per minute or total tokens per month, the model is effectively a high-capacity tier, not truly unlimited. Transparency around these limits will be critical for trust.
AINews Verdict & Predictions
AI Foundry's infinite inference subscription is a high-risk, high-reward bet that could either revolutionize AI pricing or become a cautionary tale about unsustainable business models. Our editorial judgment is cautiously optimistic, but with clear caveats.
Prediction 1: Within 12 months, at least two major cloud providers will launch similar unlimited inference tiers. The competitive pressure will force AWS, Google, and Microsoft to experiment with flat-rate pricing for specific model families. Expect these to be targeted at enterprise customers with predictable workloads, not general availability.
Prediction 2: AI Foundry will need to raise $50-100M in Series B funding within 18 months to scale infrastructure and cover losses from heavy users. The current model likely operates at negative gross margins for the top 10% of users. Venture capital will be necessary to sustain the experiment until usage patterns stabilize.
Prediction 3: Performance guarantees will become the differentiator. As more providers offer flat-rate pricing, the key competitive advantage will be consistent latency under load. AI Foundry should invest in predictive autoscaling and SLAs that guarantee 95th percentile TTFT under 200ms.
Prediction 4: The subscription model will accelerate the adoption of smaller, specialized models. When inference is free, developers will experiment more, but they will also discover that smaller models (e.g., Llama 3 8B, Phi-3) handle many tasks with lower latency. This could shift demand away from frontier models toward efficient alternatives.
What to watch next: Monitor AI Foundry's customer acquisition metrics and churn rate. If they can retain users beyond the first six months, the model has legs. Also watch for Blackwell GPU availability—NVIDIA's supply constraints could limit AI Foundry's ability to scale. Finally, keep an eye on open-source alternatives like vLLM and SGLang, which could enable any provider to replicate this model with commodity hardware.
In the long term, the infinite inference subscription is a natural evolution of AI infrastructure toward utility computing. Just as we pay a flat fee for electricity or internet access, AI compute may eventually be priced as a fixed-cost utility. AI Foundry is the first to bet on this vision, and the industry will be watching closely.