On September 15, 2026, SRE Daniil Romashov (@mwgtd) opened waf.tf and left a short comment at the top: "One client got stuck in a loop on the token route, sending ~20 req/s and eating the whole budget." That kicked off a 16-day production rate-limiting journey, which the author broke down in detail on HackerNoon.
The request path is simple: access token requests flow through Amazon CloudFront to API Gateway, and from there to AWS Lambda. The Lambda function has a hard reserved concurrency ceiling of 30 concurrent executions per region. One looping client fills all 30 slots, everyone else gets 429 from Lambda, and nothing in the app works without a token. You need to cut this traffic at the edge in CloudFront before extra requests ever reach API Gateway or the backend.
How Weekly Logs Revealed the Line Between Normal Users and Bots
They didn't guess the number. Daniil pulled seven days of CloudFront logs (September 8–15, 2026)—covering 8,921 client IPs—and calculated each IP's peak requests to the token route over five-minute windows.
The biggest fear was the shared NAT IP used by server-side rendering for the European dashboard. All European users sit behind it, so the team expected a huge spike. In reality, that peak was surprisingly modest, and the logs showed a clean gap between the busiest normal client and the quietest abuser.
5-Minute Token Route Request Peaks from CloudFront Logs
The chosen threshold—an average of one request per second over five minutes—landed right in the middle of that gap. Normal clients stay well below it, while every abuser blows past it. The worst offender wasn't even plotted so it wouldn't flatten the chart scale: it hit 7,383 requests in five minutes (up to 3,888 in a single minute). Both the limit and window were placed in locals right next to the log notes.
The engineer deliberately skipped two textbook steps. First, he didn't run the rule in COUNT mode for a week (where WAF only counts matches without blocking). Relying on the logs, he shipped block with 429 right away. However, if your traffic lacks such a clean gap—or thousands of users share a corporate NAT—you shouldn't skip a couple of days in COUNT mode. Second, he skipped full WAF logs (aws_wafv2_web_acl_logging_configuration), sticking with CloudWatch metrics and sampled requests (sampled_requests_enabled = true).
Anatomy of the Terraform Rule and a CloudWatch Alarm Trap
The rule was defined in Terraform via aws_wafv2_web_acl with scope = "CLOUDFRONT". Such a Web ACL must live in us-east-1 and attach to the distribution via web_acl_id.
Inside, it uses an AWS WAF rate-based rule. Valid evaluation_window_sec values are 60, 120, 300 (default), or 600 seconds. The IP key uses the real client address seen by CloudFront, ignoring X-Forwarded-For. To avoid counting the entire site, scope_down_statement narrows the rule to GET requests matching the lowercase token path and excludes the QA runner's IP set. Everything else passes through default_action = allow. Instead of WAF's default 403, custom_response returns an empty-body 429—matching an overloaded Lambda.
# Main rule: limit everyone except the QA runner IP set
rule {
name = "rate-limit-token"
priority = 10
action {
block {
# Replace default 403 with a familiar empty-body 429
custom_response { response_code = 429 }
}
}
statement {
rate_based_statement {
limit = local.waf_token_rate_limit # 300
evaluation_window_sec = local.waf_token_rate_window # 300
aggregate_key_type = "IP"
scope_down_statement {
and_statement {
# uri_path EXACTLY <token path>, LOWERCASE
# method EXACTLY "GET"
# not_statement { ip_set_reference_statement { QA } }
}
}
}
}
visibility_config {
cloudwatch_metrics_enabled = true
metric_name = "acl-rate-limit-token"
sampled_requests_enabled = true
}
}
Twenty-two minutes later, they added a CloudWatch alarm: BlockedRequests in AWS/WAFV2, Sum over 300 seconds, 1 period, threshold > 0. Even a single blocked request fires an SNS alert, ok_actions clears it when counts return to zero, and missing data is treated as notBreaching. The biggest trap here: the Rule dimension requires metric_name from visibility_config, not the rule's name. Otherwise, the alarm silently never triggers.
# Alarm on any blocked request within a 5-minute window
resource "aws_cloudwatch_metric_alarm" "waf_token_blocked" {
alarm_name = "waf-token-rate-limit-blocked"
namespace = "AWS/WAFV2"
metric_name = "BlockedRequests"
statistic = "Sum"
period = 300
evaluation_periods = 1
threshold = 0
comparison_operator = "GreaterThanThreshold"
treat_missing_data = "notBreaching"
dimensions = {
WebACL = aws_wafv2_web_acl.prod.name
# Use metric_name here, NOT rule.name!
Rule = "acl-rate-limit-token"
}
}
The rollout that day (starting at 14:05) took two hours. Seven seconds after committing the alarm, they had to patch another repository to add sns:Publish for cloudwatch.amazonaws.com with an aws:SourceAccount condition in the SNS policy. Then the CI plan failed: they replaced the data lookup for the SNS topic with a hardcoded ARN string and granted the CI user 11 wafv2 permissions and 6 cloudwatch permissions.
16 Days in Production: Failing E2E Tests, a Twin Rule, and a Second Route
On September 24, browser and Node.js E2E tests failed on both stage and prod. All test runs come from a single QA runner IP, and one test page hits the token route from multiple web workers at once.
Instead of blindly whitelisting that IP, Daniil added a twin rule at priority 20. The main rule excludes the QA IP set via not_statement, while the twin rule counts only that IP set with a 3,000-request limit per 5 minutes (10x higher). Their counters stay separate, and even a broken test runner can't hammer prod endlessly.
# Twin rule: separate 3000 ceiling for the QA runner
rule {
name = "rate-limit-token-qa"
priority = 20
action {
block {
custom_response { response_code = 429 }
}
}
statement {
rate_based_statement {
limit = local.waf_token_qa_limit # 3000
evaluation_window_sec = local.waf_token_rate_window # 300
aggregate_key_type = "IP"
scope_down_statement {
and_statement {
# uri_path EXACTLY <token path>, LOWERCASE
# method EXACTLY "GET"
# ip_set_reference_statement { QA }
}
}
}
}
visibility_config {
cloudwatch_metrics_enabled = true
metric_name = "acl-rate-limit-token-qa"
sampled_requests_enabled = true
}
}
A colleague copied the rule to stage without alarms, because a known frontend bug there was already being caught and alerts would only create noise. Then on September 30, they protected a second route (marketplace profile GET) based on September 23–30 logs (prod peak 60, stage peak 126, QA runner 113). They used the same 300 and 3,000 thresholds plus a second prod alarm. Across both ACLs, that came to four rules each (priorities 10, 20, 30, 40)—8 rate-based rules and 2 IP sets in total.
You can test how the main rule, twin rule, and direct CloudFront bypass behave in the simulator below.
AWS WAF Simulator: 300 Limit, 3000 Twin Rule, and CloudFront Bypass
The third mode shows the classic gap in any edge protection: if a client calls API Gateway directly and skips CloudFront, edge limits won't apply.
Closing the Backdoor and an Honest List of Blind Spots
On stage, a colleague closed the CloudFront bypass. CloudFront now attaches a secret header to every origin request, and a regional WAF on API Gateway only allows GET token requests that include it.
Terraform generates the secret (random_password, 44 characters without special symbols) and stores it in AWS SSM as an encrypted SecureString parameter. They added lifecycle { ignore_changes = [value] } to the resource—otherwise the next terraform apply would silently overwrite a manually rotated key. Rotation takes three steps: write the new value to SSM, update the regional WAF, and deploy both services. CI also needed 7 SSM permissions plus ssm:DescribeParameters (and on September 22, permissions for regional ACLs and API Gateway stage associations).
What makes this write-up refreshing is the author's honest list of blind spots. The raw logs and calculation scripts aren't in the repo. He didn't check CloudWatch history for exact alarm trigger counts, didn't re-verify whether Lambda 429 errors vanished for every user, doesn't know who the September abusers were, and didn't inspect the E2E pipeline history in the neighboring repo after bumping the limit to 3,000.
Even so, millions of unwanted requests never left CloudFront, and the 300 threshold hasn't needed a single change. The very first clients blocked were their own—a stage frontend bug and the QA runner. Once fixed, WAF only fires on actual DDoS traffic. As Daniil Romashov notes, a rate limit with block alerts catches your own bugs just as well as attacks—so before calling it a DDoS, check whose IP it is.

