If you're running infrastructure in China, you've probably noticed that container image pushes behave strangely. Builds that complete in minutes elsewhere seem to stall for half an hour. Your CI pipeline shows progress, but it's painfully, inexplicably slow. You check your connection, you check your registry, everything looks fine, and then you wait.
It's the Great Firewall, not a misconfiguration.
We've been operating production Kubernetes clusters in China for enterprise customers for several years, and image delivery has been a persistent problem. It took longer than we'd like to admit to find a solution that actually works. We're documenting it here because when we went looking for answers, we found almost nothing useful.
Why container image pushes to China are slow
Most cloud providers offer geo-replication for container registries: the idea being that you push once and images replicate automatically to regional endpoints. This works well within a provider's normal global footprint.
China is different.
Azure's China region (chinanorth, chinaeast) is not operated by Microsoft. It's run by a local entity, 21Vianet, under a separate agreement. This means Azure Container Registry's standard geo-replication does not extend to China. The China ACR is effectively a separate service, not a regional endpoint of the global one.
Pushing images to Aliyun (Alibaba Cloud's container registry, which is what most enterprise China deployments end up using) means making a cross-border transfer over a link that is subject to deep packet inspection and significant throttling. Source location matters at the margins (our pushes happen to originate from US-based CircleCI runners, and pushes from Europe tend to perform slightly better than pushes from the US), but the fundamental problem is the destination, not the origin.
A 200MB image that takes 30 seconds to push globally can take 20–30 minutes to reach Aliyun. A multi-arch image is worse. An LLM inference image, several gigabytes in size, is a different problem category entirely.
We discovered just how bad this was when we started helping customers build AI infrastructure for their China clusters. Suddenly the slow push that was an annoyance became a blocker.
Where we started, and what we tried next
Before we tried anything China-specific, our default was the same CI pipeline we use for every other region: a single push-to-registries job that builds the image once and pushes to all configured registries in parallel. When Aliyun got added to the registry list, the job didn't complain. It just took 20–30 minutes to finish, and no amount of CI configuration was going to change that. The cross-border transfer was the bottleneck.
So we started looking for ways around it.
We looked at whether Azure's replication could somehow be extended or proxied into China. It can't, not in any supported way. The 21Vianet boundary is real.
We also designed a custom solution using two operators that would handle cross-registry replication asynchronously. It was technically interesting (and probably a future project), but it was too much infrastructure to build and maintain before we had a more immediate problem to solve.
We also tried running the entire CI/CD pipeline from inside China: self-hosted runners spun up there, building images locally, pushing to Aliyun locally. This solved the push problem completely. It also revealed a new one: the Great Firewall affects egress as well as ingress, and builds need to pull dependencies. Package registries, base images, language toolchains: all of it had to come in from outside China, and all of it was subject to the same throttling and inspection that made pushing slow in the first place.
We tried to work around the pull side with a tunnel between China and Europe, routing dependency fetches through proxy servers to bypass the worst of the inspection. It worked for known sources. It didn't work as a general solution. A modern build pulls from dozens of registries and CDNs that shift over time, and pre-configuring every possible source through a tunnel turned out to be impossible to maintain across dozens of projects.
We didn't ship that approach. But we kept what we'd learned from it. The part of the pipeline closest to Aliyun, the push itself, ran fine inside China. The build didn't need to move; only the last mile did.
The split push approach
The fix is architectural: separate the China push from everything else, and stage it through a geographically closer endpoint.
Our global CI pipeline still builds the image once and pushes to all global targets (gsoci.azurecr.io, Docker Hub, Quay) from our US-based CircleCI runners. That part doesn't change. What we added is a dedicated sync-china-registry workflow running on a self-hosted CircleCI runner physically located inside China.
The China runner pulls from Azure Container Registry's Singapore endpoint instead of the US. Azure geo-replicates images from our primary registry to Singapore in seconds (sometimes a bit longer for larger images), and once the image lands there, the China runner pulls it across a much shorter, much less throttled path. Singapore is physically closer to China than the US, which obviously helps. We also suspect the Great Firewall is less aggressive on this route than on transpacific traffic, though we can't prove that part. What we can say is that pulls from Singapore are reliably fast in a way that pulls from the US never were.
Once the image is on the China runner, the push to Aliyun is just a push to a registry on the same side of the firewall. No cross-border transfer. The image arrives at Aliyun at local-push speed, because it is one.
The change in behavior was immediate. Pushes that took 20–30 minutes dropped to under 2 minutes. Multi-arch builds, which had been particularly painful, now complete at the same speed as any other region.
We made this available as an opt-in flag in architect-orb v7.1.0: adding split-china-push: true to your push-to-registries job in your CircleCI config enables the split workflow. The detailed setup is in our internal documentation, and we're working through enabling it as the default for all our repositories.
What triggered the urgency
We'd known about the slow China push for a couple of years. It was a complaint, not a crisis: engineers would notice, grumble, and wait.
What changed was AI workloads. When we started getting serious about building AI infrastructure for customers with China operations, the image sizes jumped by an order of magnitude. An LLM inference image runs several gigabytes, not 200MB, and it needs to reach China before anything can run. At that point, "wait 30 minutes" became "can't ship."
This problem isn't unique to AI workloads: it affects any organization running Kubernetes in China and pulling from registries outside the country. But AI made the stakes high enough that we finally had to solve it properly rather than work around it.
What to know if you're planning China infrastructure
A few things we've learned that aren't obvious until you've hit them:
- Azure's China ACR is a different service than global ACR. Don't assume geo-replication will reach it. Plan for Aliyun or another China-native registry as a separate push target from the start.
- Use a nearby regional endpoint as a staging point, if you have one. Singapore worked for us, on Azure. Cross-border transfers from a closer source are dramatically faster than transfers from a US or European origin, even though they're still cross-border on paper.
- Self-hosted CI runners inside China are the most reliable approach for China-specific push operations. Cloud-hosted CI runners are outside the firewall by definition: they're subject to the same cross-border throttling as everything else.
- The Great Firewall affects egress as well as ingress. If you're considering running full CI inside China to avoid the push problem, you'll inherit the pull problem: dependencies coming in are throttled the same way images going out are.
- If you're building for AI workloads in China, start with image delivery architecture before anything else. Slow or unreliable image delivery will block everything downstream.
The long-term direction is toward more automation and eventually making the split push the default path rather than opt-in. If you're running into similar problems, or if you've found a different approach that works, we'd be glad to hear about it.
