AWS ECS CloudFormation Timeout

This is more an informational post that may help others to feel less miserable in the same situation as I was! The scenario is this:

You are updating an ECS cluster via AWS CloudFormation but for whatever reason the cluster doesn’t stabilize. So, you see the stack is in UPDATE_IN_PROGRESS state and you don’t receive any message in CloudFormation Events page. If you can’t troubleshoot the issue with ECS and take no action, It will take 3 hours before CloudFormation timeouts and display a message! At this point, as you can guess, CloudFormation will rollback. Situation can be even worse if Rollback can not be proceeded successfully (in our case, there was a lack of resources preventing update and rollback). Again, CloudFormation will stuck in UPDATE_ROLLBACK_IN_PROGRESS state and will timeout after 3 hours! In a conversation I had with AWS support, they said this time is hard-coded and can’t be changed at the moment!

So, in such a situation: Keep Calm And Troubleshoot!

3 thoughts on “AWS ECS CloudFormation Timeout

  1. Killed the frozen stack using this command
    “`while ! aws cloudformation list-stack-resources –stack-name {STACK_NAME} | jq -r ‘.StackResourceSummaries | .[] | select (.LogicalResourceId==”{FAILING_RESOURCE_LOGICAL_ID}”).ResourceStatus’ | grep –color FAILED; do sleep 1; done; aws cloudformation continue-update-rollback –stack-name {STACK_NAME} –resources-to-skip {FAILING_RESOURCE_LOGICAL_ID}“`

Leave a comment