Technical Deep Dive
The core innovation behind 3D stacking for AI chips is the transition from a 2D planar system-on-chip (SoC) to a vertically integrated system-in-package (SiP). Traditional high-performance AI accelerators, such as NVIDIA's H100 or AMD's MI300X, rely on a 2.5D approach using an interposer to connect the compute die to HBM stacks. The 3D stacking approach goes further by directly bonding logic and memory dies on top of each other using through-silicon vias (TSVs) and hybrid bonding techniques.
Architecture and Algorithms:
The key architectural advantage is the dramatic reduction in data movement energy and latency. In a conventional design, data must travel from the HBM stack through the interposer and into the compute die—a path that can be several millimeters long and consumes significant power. In a 3D-stacked design, the memory can be placed directly above the compute units, reducing the interconnect length to micrometers. This is particularly beneficial for transformer-based models where attention mechanisms require frequent, high-bandwidth memory accesses.
For inference, the 3D stacking approach enables 'near-memory computing' or even 'in-memory computing' (IMC) where simple matrix-vector multiplications are performed directly within the memory array. This eliminates the von Neumann bottleneck entirely for certain operations. Chinese companies like Alibaba's T-Head and startup Enflame are exploring hybrid bonding with copper-to-copper connections at pitches below 10 microns, enabling thousands of connections per square millimeter.
Engineering Approaches:
The two dominant 3D stacking methods being adopted are:
- Face-to-Face (F2F) Hybrid Bonding: Both dies are flipped and bonded directly, offering the highest interconnect density. This is used for logic-on-logic stacking.
- Face-to-Back (F2B) Hybrid Bonding: One die is flipped and bonded to the backside of another, commonly used for logic-on-memory stacking.
A specific open-source repository worth examining is the Chipyard framework (GitHub: ucb-bar/chipyard, ~1,500 stars), which provides a modular, open-source SoC design environment that supports 3D integration exploration. While not a Chinese project, it is widely used by Chinese researchers to prototype 3D-stacked accelerator designs. Another relevant repo is OpenROAD (GitHub: The-OpenROAD-Project/OpenROAD, ~3,500 stars), which includes tools for 3D floorplanning and thermal analysis.
Benchmark Data:
The following table compares the performance characteristics of a hypothetical 3D-stacked Chinese AI chip against a traditional 2.5D design for a 175-billion-parameter LLM inference task (batch size 1, FP16 precision):
| Metric | 2.5D Design (e.g., 7nm + HBM2e) | 3D-Stacked Design (e.g., 12nm + 3D-DRAM) | Improvement Factor |
|---|---|---|---|
| Memory Bandwidth (TB/s) | 2.0 | 8.0 | 4x |
| Memory Latency (ns) | 120 | 20 | 6x |
| Energy per Token (J) | 0.8 | 0.3 | 2.7x |
| Die Area (mm²) | 800 | 600 | 25% reduction |
| Peak Compute (TFLOPS) | 200 | 180 | Slight reduction |
Data Takeaway: The 3D-stacked design achieves a 4x bandwidth improvement and 6x latency reduction, but at the cost of slightly lower peak compute due to thermal constraints. The energy efficiency gain (2.7x) is the most critical for large-scale deployment.
Key Players & Case Studies
The Chinese 3D stacking ecosystem is coalescing around three main groups: established foundries, AI chip startups, and memory manufacturers.
1. SMIC and Hua Hong Semiconductor:
SMIC has publicly stated it is developing 3D NAND and 3D DRAM processes, but for logic-on-memory stacking, it is focusing on hybrid bonding at 28nm and 14nm nodes. Hua Hong is partnering with multiple AI chip startups to provide wafer-level 3D integration services. Both are investing heavily in TSV and micro-bump technologies.
2. AI Chip Startups:
- Enflame Technology: Known for its 'Yunsui' series, Enflame is developing a 3D-stacked inference chip codenamed 'Canghai' that stacks a 12nm logic die directly onto a custom 3D-DRAM layer. Early benchmarks claim 80% utilization on BERT-large inference, compared to 45% for a 2.5D design.
- Biren Technology: Biren's BR100 series already uses a chiplet architecture, and the company has announced a next-generation design that will use 3D stacking to integrate its compute chiplets with HBM3 stacks, targeting a 2.5x improvement in bandwidth per watt.
- Cambricon Technologies: Cambricon's MLU370 series is exploring 3D stacking for its edge inference chips, where power constraints are most severe. They have filed patents for a 'monolithic 3D' approach that integrates SRAM and compute on a single die using sequential stacking.
3. Memory Manufacturers:
- Yangtze Memory Technologies Corp (YMTC): Already a leader in 3D NAND, YMTC is now applying its stacking expertise to develop 3D-DRAM, which is essential for the memory layers in these stacks. Their Xtacking 3.0 architecture allows for independent optimization of memory and logic layers.
- ChangXin Memory Technologies (CXMT): CXMT is focusing on HBM-like 3D DRAM stacks but with a proprietary 'through-mold via' (TMV) process that reduces cost compared to TSV.
Comparison Table:
| Company | Product/Project | Stacking Approach | Target Application | Status |
|---|---|---|---|---|
| Enflame | Canghai | Logic-on-3D-DRAM (F2F) | LLM Inference | Tape-out Q4 2024 |
| Biren | BR200 | Compute chiplet + HBM3 (2.5D/3D hybrid) | Training & Inference | Sampling |
| Cambricon | MLU380 | Monolithic 3D (SRAM + logic) | Edge Inference | R&D |
| SMIC | N/A | Foundry service (hybrid bonding) | Custom AI chips | Process ready at 28nm |
Data Takeaway: Enflame and Biren are the most aggressive, with Enflame targeting a pure 3D approach for inference and Biren using a hybrid 2.5D/3D approach for flexibility. SMIC's foundry services are critical for enabling smaller players.
Industry Impact & Market Dynamics
The shift to 3D stacking is reshaping the competitive landscape in several profound ways.
1. Bypassing Lithography Constraints:
The most immediate impact is that Chinese AI chip companies can achieve competitive performance without access to EUV lithography. A 3D-stacked chip using 12nm or 14nm logic can match or exceed the inference performance of a 5nm planar chip for memory-bound workloads. This effectively neutralizes one of the key advantages of TSMC's advanced nodes for AI inference.
2. New Business Models:
The 3D stacking approach encourages a 'chiplet' ecosystem where specialized dies (compute, memory, I/O) are designed by different companies and integrated by a third-party packager. This is giving rise to a new breed of 'fabless integrators' who do not design chips but focus on 3D integration and thermal management. Companies like JCET and Tongfu Microelectronics are positioning themselves as the primary 3D packaging partners.
3. Market Size and Growth:
The global 3D IC market is projected to grow from $12 billion in 2023 to $45 billion by 2028 (CAGR of 30%). China's share of this market is expected to rise from 15% to 35% over the same period, driven by domestic AI chip demand. The following table shows the projected market breakdown by application:
| Application | 2023 Market ($B) | 2028 Market ($B) | CAGR |
|---|---|---|---|
| AI Inference | 2.5 | 12.0 | 37% |
| AI Training | 1.5 | 6.0 | 32% |
| High-Performance Computing | 3.0 | 10.0 | 27% |
| Mobile & Edge | 2.0 | 8.0 | 32% |
| Other | 3.0 | 9.0 | 25% |
Data Takeaway: AI inference is the fastest-growing segment for 3D stacking, validating the thesis that memory-bound workloads benefit most from vertical integration.
4. Geopolitical Implications:
The US export controls on advanced lithography tools have inadvertently accelerated China's investment in 3D stacking. This creates a dual-use scenario: the same technology that enables competitive AI chips could also be used for military applications, potentially triggering new export controls on 3D packaging equipment.
Risks, Limitations & Open Questions
Despite the promise, 3D stacking faces several critical hurdles.
1. Thermal Management:
Stacking logic dies directly on top of each other creates 'hotspots' where heat cannot be easily dissipated. Traditional heat sinks are ineffective for internal layers. Solutions like embedded microfluidic cooling or thermal TSVs add cost and complexity. For high-power training chips (>300W), thermal runaway is a real risk.
2. Yield and Cost:
A single defect in any layer of a 3D stack can render the entire chip unusable. For a 4-layer stack, the compound yield can drop to (yield_per_layer)^4. If each layer has a 90% yield, the overall yield is only 65.6%. This significantly increases cost. Chinese manufacturers are still perfecting their hybrid bonding processes, and early yields are reportedly below 70%.
3. Design Complexity:
3D stacking requires new EDA tools for thermal-aware floorplanning, stress analysis, and signal integrity across layers. Chinese EDA companies like Empyrean and Xpeedic are developing these tools, but they lag behind Synopsys and Cadence by 3-5 years.
4. Supply Chain Bottlenecks:
The equipment for hybrid bonding (e.g., wafer bonders from Applied Materials or EV Group) is subject to export controls. Chinese companies have developed domestic alternatives, but they are not yet at the same precision level.
5. Standardization:
There is no universal standard for 3D chiplet interfaces in China. While the Universal Chiplet Interconnect Express (UCIe) standard exists, Chinese companies are developing proprietary interfaces, which could fragment the ecosystem.
AINews Verdict & Predictions
3D stacking is not a silver bullet, but it is the most strategically sound path for Chinese AI chip makers in the current geopolitical environment. The technology's ability to decouple performance from lithography node is a game-changer for the inference market, which is where the vast majority of LLM workloads will eventually run.
Predictions:
1. By 2026, at least three Chinese AI chip companies will ship production 3D-stacked inference chips that match or exceed the inference performance of NVIDIA's H100 for specific LLM workloads (e.g., Baichuan, Qwen, or Llama-class models), while consuming 40% less power.
2. Thermal management will become the single biggest differentiator; companies that master embedded cooling will dominate the high-performance segment.
3. The Chinese government will provide substantial subsidies for 3D packaging fabs, leading to a 50% increase in domestic 3D packaging capacity by 2027.
4. Consolidation is inevitable: we expect at least two major mergers between AI chip startups and packaging houses within the next 18 months to create vertically integrated '3D chip champions'.
What to watch: The next major milestone will be the first public benchmark of a 3D-stacked Chinese chip against NVIDIA's Blackwell architecture. If the performance-per-dollar metric is favorable, expect a rapid shift in procurement strategies among Chinese cloud providers like Alibaba Cloud, Baidu AI Cloud, and Tencent Cloud.
The era of flat chips is ending. The future of AI compute in China is vertical.