Skip to content
Senior Dev Notes

Networking & APIs· 1 of 13

DNS

Every answer is a lease you cannot call back. The numbers on the lease decide how you migrate, how you fail and how your load spreads.

intermediate44 min readReviewed 23 October 2026
Read first

The question

Section titled “The question”

Your DNS provider stops answering for an hour. Users of a name with a one-day TTL never notice. Users of a name with a five-minute TTL are all failing by minute five, unless their resolver serves stale answers. Now a different bug: the provider stays up but publishes the record with no addresses. The failure spreads at exactly the same speed, serve-stale does nothing, and after you fix the record, users keep failing for another quarter of an hour. Why do these outages behave so differently, and which numbers decide it?

The answer is that a DNS answer is a lease: a copy that a cache may keep for a fixed time, that you cannot revoke, and that every layer between you and the user may shorten, stretch or ignore. The next section sets out what a lease is made of, in one pass; the rest of the page is about what happens to those leases during migrations, outages and load balancing.

Why this problem exists

Section titled “Why this problem exists”

DNS data lives in zones. A zone is the part of the name tree that one operator runs from one set of servers: example.in and every name under it. Each piece of data in a zone is a record, written one per line with five fields:

one record, in zone-file form
shop.example.in.   300   IN      A      192.0.2.10
; name             TTL   class   type   value

The type says what the value means: A is an IPv4 address, AAAA an IPv6 one, CNAME says “this name is an alias, ask for that other name instead”, NS names the servers that run a zone, SOA holds the zone’s own settings and TXT is free text. The class is always IN. All records with the same name and type, say three A records for shop, form one set that is fetched, cached and expired as a unit. The TTL (time to live) is how many seconds anyone who receives that set may keep using it without asking again.

Two kinds of server handle records. An authoritative server holds a zone’s records and answers only for its own zones; it never asks anyone else. A recursive resolver holds no zones: it is the server your machine is configured to ask (your ISP’s, 1.1.1.1, 8.8.8.8, CoreDNS in a cluster), and it finds answers by asking authoritative servers and caching what it learns. Your program talks to neither directly. It calls the operating system’s getaddrinfo(), a small library (the stub resolver) that sends the question to the resolver listed in /etc/resolv.conf and returns the addresses.

A resolver with an empty cache, asked for shop.example.in, starts at the root, whose server addresses it ships with, and follows referrals: replies that say “not mine, ask these servers”.

Step Resolver asks Reply
1 a root server a referral: “ask the .in servers”
2 a .in server a referral: “ask ns1.example.in, at 192.0.2.53”
3 ns1.example.in the answer: 192.0.2.10, TTL 300

Referrals are cached too, for their own TTLs (the root hands out its referrals with 172,800 s, two days), so the next lookup for any .in name starts at step 2. A second user who asks the same resolver for shop.example.in 120 seconds later gets the answer from memory, with a TTL of 180: a resolver hands on only the time its copy has left. That user waits only for the round trip to the resolver (12–20 ms from our test network to 1.1.1.1, against 105–141 ms for a miss), and ns1 never hears about them, or about the next ten thousand, until the 300 seconds run out. That cache is the whole reason DNS scales, and it means the authoritative server only ever sees cache misses.

So one number sets the load on your name servers. Each cache asks at most once per TTL, so a popular name costs about number of caches ÷ TTL queries. With 5,000 resolver caches asking for your API’s name:

TTL Queries reaching your name servers
86,400 s (1 day) about 3.5 per minute
300 s about 1,000 per minute, 17 per second
60 s about 83 per second

The same number also sets two other things. It is the upper bound on how long a change takes to reach well-behaved caches, and it is how long users survive when your name servers are unreachable, because cached copies are then all that keeps the name alive. A short TTL buys fast changes and pays for them in query volume (often billed per million) and in a shorter survival time. There is no TTL that is good at all three; you pick per record.

“5,000 caches” also hides something. A public resolver address is anycast: one address served from many cities, and behind each city many independent cache machines. They fetch at different moments, so the same name shows a different remaining TTL depending on which one you hit. When we asked three public resolvers for example.com at the same moment in October 2026, they reported 264, 81 and 300 seconds left. Caches also evict early and are split by user location: a measurement study by Moura and colleagues found that for about 30% of clients the cache gave no protection at all during an outage (IMC 2018).

The obvious model, and why it breaks

Section titled “The obvious model, and why it breaks”

The model most engineers carry is: the TTL is how long a change takes to reach users. Lower the TTL when you change a record and the change arrives quickly. Code written with that model either resolves once at startup or resolves on every request. Both clients below talk to a small authoritative server we wrote for this page, serving the made-up zone shop.example.test:

naive_client.py
# runnable version: examples/dns/python/naive_client.py
class ResolveOnceClient:
    """Naive 1: look the name up when the process starts, then never again."""


    def __init__(self) -> None:
        self.address, ttl = resolve(API)  # the TTL is right here, and we throw it away
        print(f"t=0.0s  [ once] startup: {API} -> {self.address} (TTL {ttl} s, ignored)")


    def address_for_request(self) -> str:
        return self.address




class ResolveEveryTimeClient:
    """Naive 2: ask DNS before every request. Always fresh, and a lookup on every request."""


    def __init__(self) -> None:
        self.lookups = 0


    def address_for_request(self) -> str:
        self.lookups += 1
        return resolve(API)[0]

The record has a 2-second TTL. At 0.9 s it moves to a new server; at 3.6 s, well past one TTL, the old server is switched off:

python naive_client.py (output, trimmed)
t=0.0s  [ once] startup: api.shop.example.test. -> 203.0.113.10 (TTL 2 s, ignored)
t=0.9s  operator: new server is up, api now points to 198.51.100.20
t=3.6s  operator: waited longer than the TTL, switching off the old server 203.0.113.10
t=3.6s  [ once] request  9 -> 203.0.113.10   FAILED: connection refused (nothing runs there now)
t=3.6s  [every] request  9 -> 198.51.100.20  200 OK


resolve-every-time made 10 DNS lookups for 10 requests; the server answered 11 queries in total.

Resolve-once fails for ever, and its log says “connection refused”, which sends people to debug the network. Resolve-every-time is correct and puts a DNS round trip in front of every request. But the model behind both is wrong in four more ways, and the rest of the page is about them:

  1. The TTL that matters is the one already cached. Lowering it in the same edit as the change does nothing for copies fetched before the edit.
  2. Layers rewrite the TTL. Resolvers cap it, runtimes replace it with their own number.
  3. Connections outlive DNS. A pooled or HTTP/2 connection never asks DNS again until it closes.
  4. “No” is cached too, and so are failures. A name looked up before it existed, or during a bad publish, stays broken after the fix.

The core idea

Section titled “The core idea”

Every DNS answer is a lease with a countdown. Whoever receives it may hand it on with less time left, cut the time short, stretch it, or ignore the countdown entirely, and nobody can call it back early. So the useful question is never “what is my TTL?” but “what is the longest any holder of this answer might keep it?” Plan migrations, failovers and outages around that number.

Follow one answer from the source to your code. The number on the right is how long that layer will keep using it:

What each layer does to one TTLFive boxes stacked from top to bottom, with arrows showing one answer travelling down from the source to the application. The authoritative server publishes the record with a TTL of 300 seconds. The recursive resolver hands out only the time left on its copy, 212 seconds in this example, and may cap it. The operating system's stub resolver keeps nothing in plain glibc, though a local caching daemon may keep it for up to that time. The language runtime applies its own rule: Java keeps lookups for 30 seconds, Chrome 60 seconds, Node's lookup keeps nothing. Finally an open connection keeps using the address for hours, until it closes, without asking DNS again.one answer, from the source up to your codeAuthoritative serverpublishes the record with TTL 300300 sRecursive resolverhands out the time left; may cap it212 sOS stub resolverglibc keeps nothing; a local daemon may≤ 212 sLanguage runtimeJava: 30 s · Chrome: 60 s · Node: none30 sOpen connectionnever asks DNS again until it closeshours
One answer with a 300-second TTL. The resolver passes on only the time it has left, the runtime applies its own rule, and an open connection keeps the address until it closes. The last two rows usually decide how long the old address stays in use.

The authoritative server’s TTL is only the first row. The rows below it are set by software you may not control, and the bottom row is not DNS at all. That is why “we lowered the TTL to 60 seconds” so often ends with traffic on the old server an hour later.

How it works, piece by piece

Section titled “How it works, piece by piece”

What each layer does to a TTL

Section titled “What each layer does to a TTL”

Resolvers cap TTLs, in both directions. Resolver software clamps positive and negative TTLs to configured limits. A negative TTL is how long a resolver keeps a “no such name” answer, and serve-stale means answering from an expired copy when the authoritative servers cannot be reached; each has its own section below. Defaults, as of October 2026:

Resolver Longest positive TTL kept Longest negative TTL kept Serve-stale
BIND 9 604,800 s (7 days), max-cache-ttl 10,800 s (3 h), max-ncache-ttl off (stale-answer-enable no)
Unbound 86,400 s (1 day), cache-max-ttl 3,600 s (1 h), cache-max-negative-ttl off (serve-expired no)

Sources: the BIND 9 reference and the unbound.conf manual. Both also have a minimum TTL option (min-cache-ttl, cache-min-ttl), off by default, which some operators turn on to cut load. A resolver with a 300-second minimum turns your careful 60-second TTL into 300 seconds, and you will never see it in your own tests.

Runtimes replace the TTL with their own number. The standard getaddrinfo() call returns addresses and no TTL at all, so anything built on it has to invent one:

  • Java caches successful lookups for 30 seconds and failed ones for 10, whatever the record says (networkaddress.cache.ttl and networkaddress.cache.negative.ttl); older versions running a security manager cached for ever (InetAddressCachePolicy).
  • Chrome keeps an answer from the OS resolver for 60 seconds, because the OS never told it the TTL; with its built-in resolver it uses the TTL, with a 60-second floor (Eric Lawrence, 2022).
  • Node’s dns.lookup(), which fetch and http use, keeps nothing and runs getaddrinfo() on libuv’s four-thread pool, so slow DNS can stall unrelated file and crypto work. dns.resolve4() with { ttl: true } asks the network and returns real TTLs (Node docs).
  • glibc keeps nothing either; a machine caches only if it runs a local daemon such as systemd-resolved or nscd.

Connections ignore DNS completely. A database pool, an HTTP keep-alive connection, an HTTP/2 or gRPC channel resolves once when it connects and then never again. A service with a pool of 20 connections that live for hours keeps talking to the old address for hours, even if every cache above it switched within a minute. In a migration, the old server must close its connections (an HTTP/2 GOAWAY, a pool’s maximum lifetime, a restart) before the clients will look the name up again. Watch the old server’s open connections, not the clock.

Changing a record: the old TTL decides

Section titled “Changing a record: the old TTL decides”

When you change a record, nothing is pushed anywhere. Each cache keeps serving its copy until that copy expires, then fetches the new value at the next lookup. The copies were fetched at random moments within one TTL, so their expiries are spread evenly over the next full TTL. “Propagation” is that spread of independent timers.

That gives the arithmetic. With a TTL of 3,600 s, if you switch the old server off 10 minutes after the change, about 50 of every 60 minutes of cache lifetime are still ahead: roughly five caches in six still point at the old server. Lowering the TTL in the same edit changes nothing, because the copies already out there carry the old TTL. The only thing that helps is lowering it at least one old TTL before the change, so that by the moment of the change every copy in the world is a short one.

Try both in the simulation. The defaults lower the TTL in the same edit as the address change; then slide the lead time up.

SimulationMoving a record: the TTL that matters is the one already cached

What to watch for. With the defaults, the TTL is lowered in the same edit as the address change, so caches keep their old copy for up to the old hour: 176 of 200 still point at the old server when it goes off at +10 min, and 28% of lookups after that fail. Slide the lead time to 60 min (one old TTL): by the change every cache holds a 60-second copy, the last one switches within about 2 minutes, and only the dotted clients fail. Watch the query counter too: the short TTL costs about fifty times more queries.

The lead time is the whole trick. At zero, the sim switches the old server off with 176 of 200 caches still pointing at it, and 28% of all lookups in the next 80 minutes fail; the last cache switches an hour after the change. With a lead of 60 minutes, which is one old TTL, every cache holds a 60-second copy by the time of the change, the last one switches within about two minutes, and the only failures come from the 2% of clients that never resolve again. A lead of 30 minutes is half a fix: 68 caches still on the old address, 6% of lookups failing.

The price is in the query counter: with 60-second copies, the same caches send about fifty times as many queries. That is why the TTL goes back up once the move is done. The full plan:

  1. One old TTL ahead (plus margin for resolvers that impose a minimum), lower the TTL to 60 s.
  2. Change the address. Keep the old server running.
  3. Ask your authoritative server directly (dig @ns1.your-provider.example api.example.in) to confirm the change, and at least two public resolvers to see what users get.
  4. Make the old server close idle connections, then wait until its traffic is near zero. The tail is pinned clients: find them in its access log.
  5. Switch it off and raise the TTL again.

Changing name servers is slower still. Look inside the referral from step 2 above. A parent zone does not store your records. Where example.in begins, the .in zone stores only a delegation: NS records naming the servers that run the child zone. When those servers have names inside the child zone, there is a loop: to find the address of ns1.example.in you must ask the example.in servers, which means ns1. So the parent also hands out their addresses, called glue. (A zone hosted at a DNS provider names the provider’s servers, which live in the provider’s own zone, so it needs no glue.) The delegation therefore lives in two places: the parent zone holds NS records that point to your servers, and your own zone holds its own copy:

A delegation from .in to example.inTwo boxes. The top box is the parent zone, in., run by the registry. It holds two lines about example.in: an NS record saying example.in is served by ns1.example.in, and a glue A record giving ns1.example.in's address, 192.0.2.53. An arrow labelled "referral" points down to the bottom box, the child zone example.in, run by you. It holds the SOA record, its own copy of the NS record, an A record for shop.example.in and a CNAME for api.example.in. The child's records are the authoritative ones.Zone in. (the registry)example.in. NS ns1.example.in.ns1.example.in. A 192.0.2.53glue: breaks the loopreferralZone example.in. (you)@SOAns1 hostmaster …@NSns1.example.in.shopA192.0.2.10apiCNAMEapi.cdn.net.the truth
The parent zone holds only the delegation: NS records and the glue address of a name server inside the child zone. The child zone holds the real records, including its own copy of the NS records. The two copies have separate TTLs and can disagree.

The .com servers hand out delegations with a TTL of 172,800 seconds, two days, so a resolver that learned the old delegation an hour before your switch can keep asking the old provider for 47 more hours. Most resolvers then prefer the child zone’s own NS records once they have asked it (they are child-centric, as Moura and colleagues measured in Cache Me If You Can, IMC 2019), so the old provider’s copy of your NS records can extend that further. Keep the old provider serving an identical zone for at least two days, and update the NS records in both zones.

Negative caching: “no” is cached, and so is “I could not find out”

Section titled “Negative caching: “no” is cached, and so is “I could not find out””

Every DNS reply starts with a header carrying a result code, the way an HTTP response carries a status, followed by sections, among them the answer (records for the question) and the authority (records about who is responsible). Three codes matter here: NOERROR (0), SERVFAIL (2, “I could not get an answer”) and NXDOMAIN (3, “no such name”). A resolver that asks for a name that does not exist gets one of two negative answers:

  • NXDOMAIN (result code 3): the name does not exist, for any type.
  • NODATA: the name exists but has no records of the type asked for. There is no code for it: the answer is NOERROR with an empty answer section.

Caching “no” exists because most of these questions repeat: typos, scanners, retries, and the search-suffix lookups covered under Kubernetes below would otherwise all be cache misses. A “no” has no record to carry a TTL, so it borrows one from the zone’s SOA (start of authority) record. Every zone has exactly one SOA, at its top, holding the zone’s settings, and the authoritative server puts it in the authority section of every negative answer:

a negative answer for beta.example.in (illustrative, in dig's format)
;; ->>HEADER<<- status: NXDOMAIN
;; ANSWER SECTION: (empty)
;; AUTHORITY SECTION:
example.in.  3600  IN  SOA  ns1.example.in. hostmaster.example.in. 2026102301 7200 900 1209600 300
; SOA TTL 3600, then: primary server, contact, serial, refresh, retry, expire, minimum (300)

Serial, refresh, retry and expire tell secondary servers when to copy the zone. The last field is the minimum, and the resolver caches the “no” for the smaller of the SOA record’s TTL and that minimum (RFC 2308): here min(3,600, 300) = 300 seconds, capped by its own limit from the table above. Many real zones use far longer values: in October 2026, example.com allowed a negative answer to be cached for 1,800 s and wikipedia.org for 3,600 s. Our toy zone uses 5 seconds; the demo asks for a name a moment before the operator creates it:

python ttl_cache.py (output, part 2)
t=4.1s  lookup beta.shop.example.test.: NXDOMAIN, cached for 5.0 s more
t=4.1s  operator: created beta.shop.example.test. -> 198.51.100.30
t=5.6s  lookup beta.shop.example.test.: still NXDOMAIN, cached for 3.5 s more
t=7.1s  lookup beta.shop.example.test.: still NXDOMAIN, cached for 2.0 s more
t=8.6s  lookup beta.shop.example.test.: still NXDOMAIN, cached for 0.5 s more
t=10.1s  lookup beta.shop.example.test.: 198.51.100.30

Two less known rules make negative answers reach further than the name you asked for:

  • An NXDOMAIN covers everything below it. If beta.example.in does not exist, nothing under it can, so a resolver may answer NXDOMAIN for api.beta.example.in from that one cached answer (RFC 8020).
  • In a signed zone, one “no” covers a range of names. DNSSEC, covered at the end of this section, signs records so a resolver can check they came from the zone’s owner. A signature can only cover data that exists, so to sign a “no” the zone sorts its names and signs the gaps between them as NSEC records. A zone holding alpha, gamma and zulu publishes alpha NSEC gamma (“there is nothing between alpha and gamma”) and gamma NSEC zulu, and a question about beta gets the first one as its proof. Resolvers that use those proofs aggressively (RFC 8198; on by default since BIND 9.12, per APNIC) answer NXDOMAIN for any name in that range without asking: beta, delta and fox all come from that one cached record. Create beta and a resolver that cached the alpha–gamma proof can deny it to users who never asked for it before.

Failures are cached as well. When a resolver cannot get an answer at all (timeouts, SERVFAIL, a DNSSEC validation failure), RFC 9520 (December 2023) now requires it to remember the failure for at least 1 second and at most 5 minutes, and not to ask again meanwhile. The reason is load: without it, every user lookup turns into fresh queries to servers that are already down. The IMC 2018 study above saw retries raise traffic to the authoritative servers up to eight times during an attack, and Cloudflare reported about 30 times the usual queries for Facebook’s names on its 1.1.1.1 resolver while they were gone in October 2021 (Cloudflare, 2021).

When the authoritative side fails

Section titled “When the authoritative side fails”

When your name servers stop answering, nothing breaks at first. Each cache keeps answering from its copy, and fails only when that copy expires and it cannot fetch a new one. Without serve-stale, the share of users failing grows roughly as time since the outage ÷ TTL: all of them after one TTL.

Serve-stale (RFC 8767) changes that: when the authoritative servers cannot be reached, the resolver hands out the expired copy instead of an error, and keeps trying to refresh in the background. In numbers: a resolver fetched api.example.in (TTL 300) at 10:00:00, and your name servers went down at 10:03. At 10:06 a user asks; the copy expired at 10:05 and the refresh times out. Without serve-stale the user gets SERVFAIL. With it, the user gets the old address with a TTL of 30 seconds, so the client asks again at 10:06:30, and the same thing repeats until a refresh succeeds or the stale window runs out; the RFC suggests keeping expired data for one to three days. It is off by default in BIND and Unbound (the table above); large public resolvers use some form of it, which is part of why outages look different from different networks.

SimulationYour name servers stop answering

What to watch for. Nothing breaks at the moment of the outage. Failures grow only as copies expire: with a 300-second TTL every cache is failing by minute 5; set the TTL to 3,600 and 58% of lookups fail after half an hour. Then raise the serve-stale share: each step removes the same share of failures, and at 100% users notice nothing. Recovery is immediate when the servers return.

The TTL sets how fast the failure spreads; serve-stale sets how much of it users see. With a 300-second TTL and no serve-stale, every cache in the sim is failing by minute five. At 3,600 s, 58% of lookups fail after half an hour and almost all by the end of the hour. Serve-stale on half the resolvers halves the failure, and on all of them hides it completely. And the moment the servers come back, every failing cache recovers on its next lookup.

Now publish bad data instead. On 19–20 October 2025, a race between two of DynamoDB’s DNS automation workers left the record for dynamodb.us-east-1.amazonaws.com with no addresses: a slow worker applied a stale plan at the moment a faster one’s clean-up deleted that plan. From 11:48 PM PDT nothing could resolve the endpoint, including AWS services that depend on DynamoDB. The record was restored by 2:25 AM, and clients recovered as their cached answers expired, by 2:40 AM (AWS post-event summary).

An empty answer is a valid answer. Serve-stale only applies when the authoritative servers cannot be reached, so it never kicks in, and the “no addresses” answer is cached as a negative answer for the negative TTL, which outlives the fix:

SimulationYour name servers answer, with an empty record

What to watch for. The failure grows exactly as fast as before, and there is no serve-stale slider, because an empty answer is a valid answer. When the record is fixed at +50 min, most caches keep failing: 83% of lookups still fail ten minutes later, and the last failure comes about 15 minutes after the fix, one negative TTL. Slide the negative TTL to 60 s and the tail almost disappears.

The spread during the outage is the same as before. The difference is after the fix at +50 min: 83% of lookups are still failing ten minutes later, and the last failure comes about 15 minutes after the fix, one negative TTL. A dead name server is better than a wrong one. The lesson for the code that writes DNS: check that a plan is still the newest one at the moment you apply it, refuse to publish an empty set for a name that has always had addresses, and have a tested way to put a known-good record back by hand.

How resolvers choose a name server

Section titled “How resolvers choose a name server”

Every zone lists several name servers, and the resolver picks one per query. How it picks decides what a slow or dead name server costs you.

Resolvers keep a running round-trip time for each server address and mostly prefer fast ones, but not exclusively. Unbound picks at random among all servers within 400 ms of the fastest, assumes 376 ms for a server it knows nothing about, and remembers what it learned for 15 minutes (Unbound docs). With three name servers measured at 20, 90 and 600 ms, the band runs up to 420 ms: queries are spread at random over the first two, and the 600 ms server gets none until Unbound forgets its measurement and tries it again. Measured across the internet, most resolvers send some queries to every listed name server over time, and only about half show a preference for the faster ones (Müller et al., IMC 2017).

Three consequences:

  • Every name server must be equally strong. A far-away or small server you added “as a backup” gets a real share of production queries. The study’s advice is anycast behind every listed name server address.
  • A dead name server costs latency, not answers. A resolver that tries it waits for a timeout (a few hundred milliseconds) and then tries another; until its RTT data marks the server bad, a share of cold lookups pay that wait. Our resilient resolver below shows the same cost: a dead first upstream added about 400 ms.
  • Two providers survive what one cannot. On 21 October 2016 the Mirai botnet flooded Dyn, a large DNS provider, and names that used only Dyn became unreachable for many users for hours (APNIC). Because resolvers already spread queries across all listed name servers and fail over between them, listing name servers from two independent providers, both serving identical records, turns a provider outage into extra latency. The cost is keeping two providers in sync, which gets harder with DNSSEC (both must sign with keys the parent vouches for; see RFC 8901).

DNS as a load balancer, and why it is lumpy

Section titled “DNS as a load balancer, and why it is lumpy”

Returning several addresses for one name, or different addresses to different askers, is the cheapest global load balancing there is. Its unit of balancing is the problem: the authoritative server decides once per resolver cache per TTL, and every user behind that cache gets the same answer until it expires.

Two facts make that worse than it sounds. Resolver populations are lopsided: a few large ISP and public resolvers carry most users. And clients often do not choose at random from the list: glibc’s getaddrinfo() sorts addresses by RFC 6724 rules, which can put the same address first for every client in a network regardless of the order DNS sent (CERN, glibc bug 24282), and many clients connect only to the first. Here are four servers behind one name, with resolvers whose sizes fall off the way real ones do:

SimulationRound robin through resolver caches

What to watch for. With 10 resolvers and every client taking the first address, some server is above 1.5 times its share the whole time and the worst moment reaches about 3.5 times, yet the busiest server's two-hour average is only 1.19 times: an average hides it. Two hundred resolvers help less than you would expect, because the biggest one still carries 17% of users. Drop the first-address share to 0% and the load is even. The TTL does not change how uneven it is, only how long one server stays hot.

Three things to take away. The imbalance at any moment is set by how big the largest resolvers are and how many clients take the first address, not by the TTL: with ten resolvers the busiest server peaks at about 3.5 times its share, and some server is always above 1.5 times. The TTL sets only how long one server stays hot: up to 10 minutes at 300 s, under 2 minutes at 30 s, most of an hour at 1,800 s. And averages lie: over two hours the busiest server looks only 1.19 times busier than fair, so dashboards built on averages miss the overload that matters. Size servers behind DNS balancing for the peak, and put a real load balancer behind each address.

Failover through DNS is slow by construction. A health-checked record is only as fast as its chain: Route 53 checks every 30 seconds and needs 3 failures by default, about 90 seconds (10-second checks are an option), and AWS recommends a 60-second TTL on such records (AWS, as of October 2026). Add Java’s 30 seconds or Chrome’s 60, and open connections that never move, and two to three minutes is the realistic best case.

Geographic answers see the resolver, not the user. An authoritative server that answers by location sees where the question came from:

One name, two answersTwo users, one in Mumbai and one in Frankfurt, look up the same name through their own resolvers. Both resolvers ask the same authoritative server, which can see where each query comes from: the resolver's address, plus a slice of the user's address if the resolver sends EDNS Client Subnet. It answers the Mumbai resolver with the Mumbai site's address, 192.0.2.10, and the Frankfurt resolver with the Frankfurt site's address, 198.51.100.10, each with a short TTL so it can change its mind when a site fails health checks.MumbaiFrankfurtResolver192.0.2.10Resolver198.51.100.10Authoritative"where is this asker?"nearestsitetwo users, same name, nearest site
Two resolvers ask for the same name and get the site nearest to each of them. The authoritative server sees the resolver's address, which is close to the user only when the user's resolver is.

A user in Chennai whose public resolver reaches you from Singapore is sent to Singapore. EDNS Client Subnet (ECS, RFC 7871), one of the optional extras a query can carry (EDNS, explained in the next section), lets the resolver pass on the first 24 bits of the user’s IPv4 address: for a user at 198.51.100.86 it sends 198.51.100.0/24. That fixes the answer and splits the cache: each answer is now valid only for one subnet, so a resolver serving users from 1,000 subnets may hold 1,000 copies of one name. One CDN’s measurements found the cache needed for ECS answers grew by up to an order of magnitude, with hit rates under half of the normal ones (Al-Dalky, Rabinovich and Schomp, 2019). Use DNS to pick a region; let a load balancer pick the server.

Size limits: UDP, truncation and TCP

Section titled “Size limits: UDP, truncation and TCP”

DNS answers travel over UDP on port 53 by default: one packet for the question, one for the answer, no connection. The original standard capped UDP answers at 512 bytes, which is plenty for addresses: an answer with one A record measured 72 bytes in our tests. EDNS(0) (RFC 6891) is the extension mechanism added later. The client appends a small pseudo-record, OPT, to its query, and one of its fields says how large a UDP answer the client can accept; dig prints it as ; EDNS: version: 0, flags: do; udp: 1232, where do asks for DNSSEC signatures and udp: 1232 is the buffer size. ECS above travels in the same OPT record.

A bigger buffer has a catch. A packet larger than a link can carry (1,500 bytes on Ethernet, less over tunnels and VPNs) is split into IP fragments that travel separately and are put back together at the end. Lose one fragment and the whole answer is lost. Only the first fragment carries the UDP ports, so many firewalls drop the rest, and an attacker who times it right can substitute a forged later fragment. Since DNS Flag Day 2020 major resolvers advertise 1,232 bytes, a size that never needs fragmenting: IPv6 guarantees that every link carries 1,280-byte packets, minus 40 bytes of IPv6 header and 8 of UDP header.

An answer that does not fit is not sent in part. The server replies with only the header and the TC (truncated) flag, and the client asks again over TCP, which costs at least one more round trip for the TCP handshake. What pushes answers past the line is mostly DNSSEC, the next section: a zone’s public keys (its DNSKEY set) are large with RSA keys, and while a key is being replaced the old and new keys are published together.

DNS answer sizes against the UDP limitsA horizontal bar chart on a scale from 0 to 3,000 bytes, with two dashed vertical lines: 512 bytes, the original UDP limit, and 1,232 bytes, the EDNS buffer size most resolvers now use. An A record with one address is 72 bytes. A DNSKEY answer with one ECDSA P-256 key is 347 bytes. With one RSA-4096 key it is 1,245 bytes, slightly over the 1,232 line. During a key rollover with two RSA-4096 keys it is 1,755 bytes. Our toy TXT record with 12 long strings is 2,835 bytes. Everything past 1,232 bytes comes back truncated over UDP and must be fetched again over TCP.5121,232: TCP beyond thisA record, one address72 BDNSKEY, one ECDSA P-256 key347 BDNSKEY, one RSA-4096 key1,245 BDNSKEY, two RSA-4096 (a rollover)1,755 Bour toy TXT record, 12 strings2,835 B
Answer sizes against the two UDP limits. Plain address records are tiny; signed key sets with RSA keys cross 1,232 bytes, especially during a key rollover when two keys are published; long TXT records cross it by far. Everything right of the red line needs TCP.

The A record and the toy TXT record are our own measurements; the DNSKEY sizes are APNIC’s (Huston, 2021). Our resilient resolver hits the TXT case against the toy server:

python resilient_resolver.py (output, step 2)
t=0.4s  2. an answer too big for UDP
    UDP answer from 127.0.0.1:8053 was truncated (39 bytes, TC=1); retrying over TCP
    TCP answer: 2835 bytes, 12 records
         got 12 TXT strings, 2388 characters

TCP is not optional: RFC 9210 (2022) requires every resolver and server to support it and every network to allow it. A firewall that blocks TCP port 53 breaks only the big answers, typically DNSSEC-signed ones and domains with many verification TXT records, which makes it one of the more confusing partial outages. Keep hot names small: a handful of addresses, no long TXT records on the name your API clients look up.

DNSSEC: what it buys and what it costs

Section titled “DNSSEC: what it buys and what it costs”

Without signatures, a resolver accepts any answer whose 16-bit ID, source port and question match its query. An attacker who knows a resolver is about to ask can flood it with forged replies, each guessing a different ID and port out of 65,536 IDs times tens of thousands of ports. One hit poisons that cache for every user of it, for the whole TTL.

DNSSEC (RFC 4033) adds public-key signatures. A zone has a key pair: the private key stays with whoever signs the zone, and the public key is published in the zone as a DNSKEY record. Each record set gets a signature, an RRSIG record, computed over the set with the private key; anyone holding the public key can check that the set is exactly what the owner signed. A signed answer arrives as the set plus its signature:

a signed answer (illustrative, in dig's format, signature shortened)
shop.example.in.  300  IN  A      192.0.2.10
shop.example.in.  300  IN  RRSIG  A 13 3 300 20261106000000 20261023000000 41520 example.in. Mg5r...
;                                 ^ covers the A set, algorithm 13 (ECDSA P-256),
;                                   valid 23 Oct to 6 Nov 2026, made with key 41520 of example.in

That leaves one question: why trust the DNSKEY? Whoever can forge the A record can forge a key too. So the parent vouches for it. The .in zone publishes a DS record holding a hash of example.in’s key and signs it with its own key; .in’s key is vouched for by a DS in the root; and the root’s key is built into every validating resolver. Checking a name means walking that chain:

The DNSSEC chain of trustAt the top, a trust anchor: the root zone's key, built into every validating resolver; since 11 October 2026 that is KSK-2024. Below it, three zone boxes linked by arrows. The root zone publishes its DNSKEY and a DS record for .in, a hash of .in's key. The .in zone publishes its DNSKEY, which matches that hash, and a DS record for example.in. The example.in zone publishes its DNSKEY, which matches, and signs its records: shop A 192.0.2.10 comes with an RRSIG made with that key. If any link fails to match, a validating resolver answers SERVFAIL.Trust anchorroot key, built into resolversRoot zoneDNSKEY (signs the zone)in. DS = hash of .in's keyZone in.DNSKEY (matches that DS)example.in. DS = hashZone example.in.DNSKEY (matches that DS)shop A 192.0.2.10RRSIG = signature over itany link broken → SERVFAILvouches forvouches for
Each parent publishes a hash of its child's key, and each zone signs its own records. A validating resolver checks every link from the root down; if any link fails, it answers SERVFAIL rather than an answer it cannot verify.

Keys are replaced from time to time, a key rollover: publish the new DNSKEY next to the old one, wait at least one DNSKEY TTL so every cache holds both, sign with the new key, and remove the old one later. Most zones run two keys. The key-signing key (KSK) signs only the zone’s DNSKEY set and is the one the parent’s DS points to, so replacing it means changing the DS at the parent; the zone-signing key signs everything else and can be replaced without the parent.

It protects only users whose resolver validates: in 2025 about 36% of users sat behind validating resolvers, while only about 7% of domains had a signed delegation (CircleID, citing APNIC, February 2026). What it costs:

  • Bigger answers. Every answer carries signatures: 256 bytes per RRSIG with RSA-2048, 64 with ECDSA P-256. With RSA keys, key-set answers cross the 1,232-byte line above and need TCP.
  • More lookups on a cold cache. The resolver must also fetch DNSKEY and DS records for every zone on the way down. They cache well, so the cost lands on cold caches and rarely used zones.
  • It fails closed. A wrong signature anywhere in the chain makes the name disappear for every validating resolver’s users, while every server is up. On 5 May 2026 DENIC, the .de registry, rolled its signing key with a new system that generated a different key on each of three hardware security modules but published only one. Validating resolvers returned SERVFAIL for .de names, including unsigned domains (for those, .de serves a signed proof that no DS record exists, an NSEC-style “no” like the ones above, and those signatures failed too), and the failure grew for about three hours as cached answers expired, until operators such as Cloudflare switched validation off for .de (DENIC, Cloudflare).
  • Validation itself can be attacked. KeyTrap (CVE-2023-50387, patched in February 2024) was a flaw in the specification: a zone with many keys and signatures made the validator try every combination, and one crafted packet could stall BIND for up to 16 hours (ISC advisory, research paper).

The root’s own KSK was replaced this month: as of October 2026, KSK-2024 signs the root’s key set from 11 October 2026 (ICANN).

Choosing TTLs

Section titled “Choosing TTLs”

Almost every DNS decision an application team makes is a TTL. Where I would start:

Record TTL Why
API or app endpoint you control 300 s; 60 s from one old TTL before planned work Moves are planned; load and outage survival matter the rest of the time
Health-checked failover or weighted records 30–60 s Their whole job is to change fast; pay the query cost
Apex (example.in itself) and www behind a CDN whatever the CDN sets, often 300 s The CDN’s own names change behind it
MX, TXT, CAA, verification records 1 hour to 1 day Rarely change; long TTLs ride out a provider outage
NS records in your own zone 1–2 days, to match the parent Short child NS TTLs only add load
SOA minimum (negative TTL) 60–300 s A name tested too early heals within minutes

How to choose. Long TTLs for what rarely changes, because they protect you in a DNS outage and cost nothing. Short TTLs only where change speed is the point. And the TTL that matters on the day of a change is the one you set one old TTL earlier.

Build it yourself

Section titled “Build it yourself”

Two pieces, each closer to production than the naive clients. Everything is standard library only, with a shared dnswire module for the binary messages, and runs against the toy server on your own machine.

A cache that honours TTLs, including negative ones. The interesting part is where the expiry comes from: the shortest TTL in a CNAME chain for a positive answer, the SOA rule for a negative one.

ttl_cache.py
# runnable version: examples/dns/python/ttl_cache.py
    def _ask(self, name: str, rtype: str) -> Entry:
        _, query = w.build_query(name, rtype)
        reply = w.parse_message(w.ask_udp(*self.server, query))
        now = time.monotonic()
        # Follow any CNAMEs in the answer; the chain is only as fresh as its shortest-lived link.
        records = [r for r in reply.answer if r.type in (rtype, "CNAME")]
        addresses = [r.value for r in records if r.type == rtype]
        if reply.rcode == "NOERROR" and addresses:
            return Entry(addresses, "NOERROR", now + min(r.ttl for r in records))
        # NXDOMAIN (no such name) or NODATA (name exists, no records of this type).
        # Negative TTL = the smaller of the SOA record's TTL and its MINIMUM field (RFC 2308).
        soa = next((r for r in reply.authority if r.type == "SOA"), None)
        negative_ttl = min(soa.ttl, soa.data["minimum"]) if soa else 0  # no SOA: do not cache
        return Entry([], reply.rcode if reply.rcode != "NOERROR" else "NODATA", now + negative_ttl)

Run against the same move as the naive clients:

python ttl_cache.py (output, part 1, trimmed)
t=0.9s  operator: new server is up, api now points to 198.51.100.20
t=1.8s  request  5 -> 203.0.113.10   (old) 200 OK
t=2.3s  request  6 -> 198.51.100.20  (new) 200 OK
t=3.6s  operator: waited longer than the TTL, switching off the old server 203.0.113.10


10 requests, 2 DNS lookups, 8 answered from the cache.

Two lookups instead of ten, and the switch happened on its own when the copy expired, 1.4 seconds after the change and before the old server went away. Part 2 of the same run is the negative-caching output shown earlier.

A resolver that survives bad days. The production-shaped version tries several upstreams with a short timeout each, checks that the reply matches the question, retries over TCP on truncation, clamps silly TTLs between a floor and a ceiling, and serves stale when everything is down:

resilient_resolver.py
# runnable version: examples/dns/python/resilient_resolver.py
    def resolve(self, name: str, rtype: str = "A") -> list[str]:
        key, now = (name.lower(), rtype), time.monotonic()
        entry = self.cache.get(key)
        if entry and now < entry.expires:
            return entry.values
        try:
            values, ttl = self._ask_upstreams(name, rtype)
        except LookupFailed as err:
            if entry and now < entry.stale_until:  # serve stale: an old answer beats no answer
                entry.expires = now + STALE_TTL
                print(f"    ! every upstream failed ({err}); serving the expired answer for {STALE_TTL} s")
                return entry.values
            raise
        ttl = max(MIN_TTL, min(MAX_TTL, ttl))
        self.cache[key] = Entry(values, now + ttl, now + ttl + MAX_STALE)
        return values

The demo’s first upstream is a port where nothing listens, and later the toy server stops answering altogether:

node resilient-resolver.mjs (output, trimmed)
t=0.0s  1. the first upstream is down; the second answers
         api.shop.example.test. -> ["203.0.113.10"]  in 406 ms
t=0.8s  3. the authoritative server goes down; the cached answer expires
    ! every upstream failed (127.0.0.1:5399 ETIMEDOUT; 127.0.0.1:8053 ETIMEDOUT); serving the expired answer for 30 s
t=3.8s  4. a name we never cached, while everything is down
         failed, nothing stale to fall back on: 127.0.0.1:5399 ETIMEDOUT; 127.0.0.1:8053 ETIMEDOUT

The dead upstream cost a full 400 ms timeout before the healthy one answered, which is why the timeout per attempt must be short. Step 3 is serve-stale doing its job; step 4 is its limit: it can only replay answers it has already seen. Full code, the toy server and run instructions: examples/dns/.

How real systems do it

Section titled “How real systems do it”

Kubernetes multiplies every external lookup. Each pod’s /etc/resolv.conf lists search suffixes such as my-namespace.svc.cluster.local and sets options ndots:5 (Kubernetes docs): a name with fewer than five dots is first tried with every suffix. A lookup of api.stripe.com is tried as api.stripe.com.my-namespace.svc.cluster.local and two more suffixes, each an NXDOMAIN, before the real name; with A and AAAA asked separately, one lookup becomes eight queries to CoreDNS. Write external names with a trailing dot (api.stripe.com.), or lower ndots for the pod with dnsConfig.

nginx resolves upstream names once, unless told otherwise. A hostname written directly in proxy_pass is resolved at startup and kept: resolve-once, in a config file. As of October 2026, open-source nginx can re-resolve upstream servers since 1.27.3 (nginx docs):

nginx: follow DNS changes for an upstream
resolver 10.0.0.2 valid=30s;          # re-resolve at most every 30 s, whatever the TTL says
upstream payments {
    zone payments 64k;                # resolved addresses live in shared memory
    server api.partner-bank.example resolve;
}

Managed DNS sells the lease as a product. Route 53’s routing policies (weighted, latency, geolocation, failover, multivalue with up to eight healthy addresses) are all ways of choosing which lease to hand to which resolver (Route 53 docs). Every one of them inherits the limits above: decisions per resolver, delivered one TTL late.

Gotchas and war stories

Section titled “Gotchas and war stories”

“It works on my laptop.” Your dig asks one resolver, which may have fetched a fresh copy a second ago. After a change, ask the authoritative server directly for the truth, then two or three public resolvers for what users see. dig +norecurse @8.8.8.8 name shows only what that resolver already has cached.

A TTL change that never happened. Some DNS dashboards apply the TTL to new records only, or round it up to a plan minimum. Before a migration, check the TTL your authoritative server actually serves, not the one in the form.

Every name sits under zones you do not run. The .de failure took down domains whose owners did nothing wrong, and the October 2025 DynamoDB failure took down services that only depended on an endpoint. Know which zones and which DNS-backed endpoints your critical path depends on, and what their TTLs give you when they fail.

Trade-offs and when not to use it

Section titled “Trade-offs and when not to use it”
  • Freshness against load and resilience. Short TTLs mean fast changes, more queries, a bigger bill and less time to survive a provider outage. You choose per record.
  • DNS steering is coarse. It decides per resolver and reacts in TTLs; a load balancer decides per connection or request and reacts in seconds. Use DNS to choose a region.
  • DNSSEC trades availability for authenticity. It stops forged answers for validating users and turns any signing mistake into a total outage for them.
  • Encrypted DNS moves trust. DNS over HTTPS or TLS hides lookups from the network and shows them all to one resolver operator. It does not prove the answer is genuine; DNSSEC does that.

When should you not lean on DNS? Where endpoints change every few seconds, such as autoscaling pods, every layer’s caching works against you: use a service registry or a load balancer with health checks. And DNS cannot make per-request decisions; it never sees requests.

Interview corner

Section titled “Interview corner”
  1. “You changed a record and some users still reach the old server an hour later. Why?” Caches hold the old copy for up to the old TTL; runtimes and pools keep it longer. Strong answers add the plan: lower the TTL one old TTL ahead, keep the old server until its traffic and connections drain, and check the authoritative server directly.
  2. “Your DNS provider is down. What do users see, and when?” Nothing until copies expire, then failures grow with time ÷ TTL, unless resolvers serve stale. Follow-up: “How do you design for it?” Name servers from two providers, long TTLs on records that rarely change.
  3. “How would you do global failover with DNS?” Health-checked short-TTL records, and the honest numbers: about 90 s detection plus a TTL plus runtime caches, with pinned clients never moving. Pair it with load balancers in each region.
  4. “Is DNS UDP or TCP?” Both: UDP first, TCP for truncated answers above 1,232 bytes and for zone transfers, and TCP support is mandatory.
  5. “Should we turn on DNSSEC?” What it protects (forged answers, for validating users), what it costs (size, cold-cache lookups, fail-closed outages), and how to do it safely (automated rollovers, ECDSA).

Check yourself

Section titled “Check yourself”
1. A record has a TTL of 3,600 s. At 14:00 you change the address and set the TTL to 60 s in the same edit. When can you safely switch off the old server?

Not before about 15:00, and only once its traffic and open connections have dropped to near zero. Caches that fetched the record shortly before 14:00 got it with the old 3,600-second TTL. To cut over quickly you had to lower the TTL at 13:00 or earlier.

2. A deploy pipeline polled report.example.in for 10 minutes before the record was created at 09:30. The zone’s SOA has TTL 3600 and minimum 900. At 09:40 some users still get NXDOMAIN. Why, and when does it end?

Every resolver the pipeline’s checks went through cached the NXDOMAIN for min(3600, 900) = 900 s, so the last one, cached shortly before 09:30, can last until about 09:45, plus any runtime cache. Resolvers that were never asked fetched the new record at once.

3. Your name servers are unreachable for 40 minutes. Record A has a TTL of 300 s, record B a TTL of 86,400 s. What happens to each, and what does serve-stale change?

Users of A start failing as copies expire and nearly all fail within five minutes; users of B almost never notice, since few copies expire in 40 minutes. Serve-stale keeps A working for users whose resolver had a copy, but cannot help a cache that never had one.

4. A bug publishes your API’s record with no addresses for 20 minutes. Your negative TTL is 3,600 s and your resolvers serve stale. How long are users affected?

Serve-stale does not apply: the servers answered, and an empty answer is valid. Every cache that fetched during those 20 minutes cached “no addresses” for up to 3,600 s (or the resolver’s cap, one hour in Unbound), so users can fail for up to an hour after the fix. A short SOA minimum keeps this tail short.

5. Four servers behind one name with round-robin DNS. Monitoring shows each server averaging about 25% of traffic, yet one of them falls over every few minutes. How?

The authoritative server decides per resolver cache, a few big resolvers carry most users, and many clients take the first address. At any moment one server can hold several big resolvers’ users, and the hot spot moves every TTL, so averages look even. Size for the peak, or put a load balancer behind each address.

Recap

Section titled “Recap”
  • A DNS answer is a lease you cannot recall. Plan for the longest time any holder might keep it: resolver caps, runtime caches (Java 30 s, Chrome 60 s) and open connections.
  • After a change, copies live for up to the old TTL. Lower it one old TTL ahead, change, drain the old server’s connections, then switch it off.
  • “No” is cached: NXDOMAIN and NODATA for min(SOA TTL, SOA minimum), and resolution failures for 1 s to 5 min. NXDOMAIN covers the subtree, and NSEC proofs cover ranges.
  • In an outage, failures grow with time ÷ TTL; serve-stale hides an unreachable server but never an empty record, which also lingers for the negative TTL after the fix.
  • Resolvers spread queries over all name servers: make each equally strong, and use two providers for critical names. DNS balancing is per resolver and lumpy; failover takes minutes.
  • Answers above 1,232 bytes go to TCP; DNSSEC adds size, lookups and a fail-closed mode.

Where to go next

Section titled “Where to go next”
  • TCP vs UDP

    Why DNS uses UDP first and TCP for big answers, and what each costs.

    Networking & APIs
  • TLS and Encryption in Transit

    Certificates are issued for names: how CAs check that you control a domain, often with a DNS record, and what CAA records restrict.

    Networking & APIs
  • CDNs and edge cachingcoming soon

    The biggest user of DNS steering: GeoDNS, anycast and short TTLs to reach a nearby edge.

    Traffic & edge
  • Load Balancing

    What does the fine-grained work behind the address DNS hands out, in seconds instead of TTLs.

    Traffic & edge
  • Service discoverycoming soon

    Finding services inside your own system, where DNS is one option and a registry is often better.

    Traffic & edge
  • Caching Strategies

    TTL expiry is one caching strategy among several; the same freshness trade-offs appear everywhere.

    Caching

Further reading

Section titled “Further reading”

Specs & official docs

Research & engineering

Explainers

Edit page

Last updated:

Senior Dev Notes

System design, explained like a mentor would.

Written by Uday. Every page is researched, reviewed by a human, and every code example was run before it was published.