A hosted service caps usage while it stays up. Two common caps are requests per minute and tokens per minute.
Both caps are properties of the service tier, and both vary between services. Design for the lower of the two on the expected traffic.
Budget | Counts | Binds when |
|---|---|---|
Requests per minute | Calls, whatever their size | Many small calls arrive at once. |
Tokens per minute | Input plus output tokens | Few large calls carry long prompts. |
A rejected request waits and tries again with a longer wait each time. Doubling the wait is a common pattern, and a small random extra spreads out retries from many clients at once.
The pattern below uses a base wait of 1.0 s and doubles it: 1.0 s, 2.0 s, 4.0 s. These are example parameters, not figures from any real service.
example parameters: base wait 1.0 s with doubling waits
success on attempt 3: waits 1.0 + 2.0 = 3.0 s of waiting, then the reply is used
limit of 3 attempts, all busy: waits 1.0 + 2.0 + 4.0 = 7.0 s, then the caller stops and reportsEvery retry policy needs a limit: a maximum attempt count, a maximum total wait, or both. Past the limit the caller stops and reports, instead of retrying without end.
The program below stands in for a service with a FAKE scripted outcome list. It checks both endings: success on the third attempt, and a clean stop at the attempt limit.
import sys
def run(plan, limit, base_wait_s):
waits = []
attempts = 0
result = "busy"
for outcome in plan[:limit]:
attempts += 1
result = outcome
if outcome == "ok":
break
waits.append(base_wait_s * (2.0 ** (attempts - 1)))
return attempts, waits, result
a1, w1, r1 = run(["busy", "busy", "ok"], 4, 1.0)
a2, w2, r2 = run(["busy", "busy", "busy", "busy"], 3, 1.0)
ok = (a1 == 3 and w1 == [1.0, 2.0] and r1 == "ok"
and a2 == 3 and w2 == [1.0, 2.0, 4.0] and r2 == "busy"
and a1 <= 4 and a2 <= 3)
if not ok:
print("FAIL: retry counts or waits differ from the worked values")
sys.exit(1)
print("PASS: bounded retries give 3 attempts then stop in both cases")1.0 + 2.0 = 3.0 s. The third attempt succeeds, so no wait follows it.
3 attempts run with waits 1.0, 2.0 and 4.0 s. The caller stops and reports instead of trying a fourth time.