AWS says CloudFront VPC Origins outage is resolved after 5xx errors

Amazon Web Services said it has resolved a CloudFront outage that returned 5xx errors for customers using VPC Origins, disrupting websites and apps across multiple regions on July 16. The incident matters for cloud operators because it hit a newer CloudFront configuration designed to keep back-end infrastructure private, and AWS said other origin types were not affected.

AWS said the problem began at 01:45 PDT, or 09:45 UTC. CloudFront is AWS's content delivery network, and VPC Origins lets customers connect CloudFront to applications behind private load balancers without exposing those back ends directly to the public internet.

The company did not disclose how many customers were affected, the regions with the most impact, or any service-level credit implications. It said on its service health page that engineers were working to reduce the impact and advised customers that could switch away from VPC Origins to use another origin type as a temporary workaround.

AWS points to internal connection constraint

In a later update, AWS said it had found the cause: an internal constraint on the fleet that manages connections to private VPC origins. According to AWS, when that constraint was reached, the system that distributes routing configuration to network processors failed to load updated configuration data correctly, which affected routing for VPC Origin connections.

AWS said it took mitigation steps that produced a full recovery. The company also said customers that had temporarily changed origin type could revert those changes, and that customers using other origin types were not affected. AWS said the service was operating normally after the fix.

The initial customer-facing symptom was straightforward: affected CloudFront distributions returned server errors instead of serving the intended site or app. Users trying to reach affected services saw an error message saying the app or website could not be reached at that time, with possible causes listed as traffic volume or configuration trouble.

Hugging Face and National Lottery among affected services

Hugging Face, the AI developer platform, said its service was unavailable from most regions worldwide because of the AWS outage while it worked on mitigation. The company did not provide a separate technical cause beyond the AWS incident.

The UK's National Lottery said in a post on X that users could not access its website or mobile app because of a wider AWS outage, and told players to try refreshing later. Reports also appeared on Reddit from Fallout 76 players and other AWS customers describing CloudFront 5xx errors while other AWS services appeared to keep running.

The incident is narrower than a full AWS regional failure, based on AWS's description, but the visible impact was broad because CloudFront sits in front of customer-facing services. For companies using VPC Origins, the outage exposed a dependency in the path between edge locations and private application infrastructure, even when the underlying application stack may have remained healthy.

This story draws on original reporting from The Register.