Fri Oct 24 00:31:07 2025
(*097dfbf6*):: clown cloud https://x.com/copyconstruct/status/1981192300819862000 +public!
*** Cindy Sridharan (@copyconstruct) on X
*** The detailed AWS incident report is out, and it’s worth a read
– DNS records managed by 2 systems; a race condition led to regional record getting unset
– EC2 lease establishment was borked as it depends on DynamoDB
– fluctuating NLB health checks leading to EC2 DNS entry purges
*** X (formerly Twitter)
G36b5PzW4AAHXuR.jpglarge
(*4cfb807c*):: ^ the worst outage in their history, only finally getting an explanation on Thurs, my company surprisingly didn’t get effected that bad by it, but there were a lot of drunk ops this week
(*53e37792*):: You have an abnormally busy week?
(*4cfb807c*):: Nah the company I work at got off easy, they have a ton of us-east-1 resources but we dont lean heavily on ec2 or dynamodb. Dynamodb isn’t really used by anyone except aws at this point.
(*4cfb807c*):: Theres probably a handful of other services they have that could cause cascading service failures like this though, but losing ec2 scale is a pretty big blast radius.
(*097dfbf6*):: clown cloud https://x.com/copyconstruct/status/1981192300819862000 +public!
*** Cindy Sridharan (@copyconstruct) on X
*** The detailed AWS incident report is out, and it’s worth a read
– DNS records managed by 2 systems; a race condition led to regional record getting unset
– EC2 lease establishment was borked as it depends on DynamoDB
– fluctuating NLB health checks leading to EC2 DNS entry purges
*** X (formerly Twitter)
(*4cfb807c*):: https://aws.amazon.com/message/101925/
G36b5PzW4AAHXuR.jpglarge
(*097dfbf6*):: shell uses AWS but only the clown groups were down. Our groups stuff is on Azure clown lol