To create a resilient auto scaling policy, it's essential to focus on metrics that accurately reflect application utilization, which often diverges from traditional system resource metrics. While CPU utilization is a common metric, it may not always correlate with actual worker capacity due to various factors.
Instead of relying solely on CPU metrics, tracking worker utilization offers a more reliable approach. By assessing total worker slots, work in flight, and backlog, organizations can derive a utilization value that remains effective across diverse fleets and evolving application behaviors.
Understanding Worker Utilization
For instance, consider an application that processes messages from Amazon SQS and stores results in Amazon DynamoDB. If the application employs a fixed thread pool of 10 workers, it reaches capacity when all threads are busy, irrespective of CPU usage.
In this scenario, workers may spend significant time waiting for responses from DynamoDB, leading to low CPU utilization while the SQS queue fills up with unprocessed messages. This disconnect can mislead auto scaling policies, which may perceive sufficient capacity when, in reality, the application is overwhelmed.
Challenges with Traditional Scaling Policies
AWS offers guidance for scaling based on acceptable backlog per worker, which works well under consistent processing times. However, this approach falters when latency varies, as seen in applications that evolve over time.
For example, an image processing application initially designed for thumbnails may later incorporate 4K images, significantly increasing processing time and backlog. As the application behavior changes, scaling policies must be updated to reflect these new realities.
Implementing Worker Utilization Metrics
Worker utilization metrics focus on the ratio of active work to available processing capacity. This can be calculated by dividing total work by total workers. For SQS-based applications, metrics from Amazon CloudWatch can provide necessary data on messages waiting and currently being processed.
To set up worker utilization-based auto scaling:
- Identify a metric for the amount of work being processed.
- Implement a custom metric representing total workers.
- Optionally track the backlog of work.
Using CloudWatch metric math, organizations can calculate the utilization metric and apply it in a target tracking scaling policy, ensuring a more responsive scaling mechanism.
Balancing Cost and Performance
When determining target utilization values, consider the balance between cost efficiency and application availability. Lower targets provide more headroom for spikes in traffic, while higher targets maximize resource usage but may limit responsiveness. Starting with a moderate value like 0.7 can help align scaling policies with observed application behavior.
While worker utilization informs scaling decisions, it’s crucial to regularly assess CPU and latency metrics for cost optimization. This dual approach allows for independent adjustments of application performance and resource allocation.
Conclusion
Adopting worker utilization metrics for auto scaling enhances resilience and adaptability in dynamic environments. By focusing on the relationship between work and worker capacity, organizations can ensure their applications remain responsive and efficient, regardless of changes in infrastructure or workload demands.
