Debug From Where the Rejection Happens

A remote node wouldn't authenticate to a relay. From the node's side, everything looked healthy: daemon running, network reachable, handshakes happening — then rejection, retry, rejection, forever.

Five hours and two wrong theories later, someone finally read the relay's own log. One line: "auth event timestamp outside ±60s window."

The node's clock was skewed by more than a minute. It was signing authentication challenges with timestamps from the past. The relay rejected every one — correctly. Re-enabling time sync fixed it in two minutes.

The Client's View of Its Own Failure Is Incomplete

The node could see that it was being rejected. It could not see why — the rejection reason lived on the other side of the connection. So its logs said "auth failed" and retried, and from the node's chair, "auth failed" looks like a credential problem, a membership problem, a network problem. Anything but a clock problem.

That's how you get two confident theories in a row:

• Theory 1: firewall. Fix the egress path. Still failing.
• Theory 2: membership. Re-check admission, re-check the migration. Also wrong.

Both theories fit every fact available on the client side. That's the trap: client-side-only debugging is theorizing in the dark. You're fitting explanations to a symptom instead of reading a cause.

The Rejecting Side Has the Reason

When a distributed system misbehaves, the two halves of the conversation hold different information:

The initiator knows what it sent and that it failed. The rejector knows why it said no.

The debug information you need is almost always on the side doing the rejecting. Not the side feeling the pain. Those are different machines, and habit sends us to the wrong one — the one we're already logged into, the one that's hurting, the one whose logs are easy to reach.

Get to the other side first. Even indirect access beats perfect visibility on the wrong host: a forwarded log, a support ticket transcript, a colleague who can run one command. One line from the rejecting side outweighs an afternoon of client-side archaeology.

Signed Protocols Fail on Time, Not Identity

The clock-skew class of failure deserves its own respect. Anything with signed, time-bound credentials — JWTs, mTLS with short-lived certs, challenge-response protocols — treats a timestamp outside the validity window as invalid, by design.

A perfectly valid credential from "the past" is an invalid credential. The system is working correctly while appearing completely broken. And nothing in the credential itself will tell you — the key is fine, the signature is fine, the membership is fine. Only the clock is wrong, and only the rejector can see it.

Practical corollaries:

• Clock sync is a security dependency. Treat NTP/chrony as part of your auth infrastructure, not housekeeping. Monitor drift like you monitor disk.
• When auth fails "everywhere" for one host, check that host's clock before rotating any keys.
• Persist the fix. A one-time date -s unblocks you today and fails again on the next reboot. Sync daemons or nothing.

The Asymmetric Economics of Logs

Here's what the five hours actually cost, broken down: two wrong fixes implemented and verified, one incorrect root-cause announced (and retracted), one team's attention burned — versus a single log line that was sitting on a disk the entire time.

The economics of debugging are asymmetric. Theories are expensive; the rejecting side's log is nearly free. The discipline is sequencing: pull the server log before forming theory two, not after. Ideally before theory one.

A useful self-check when you're deep in a distributed-systems failure: whose log am I reading right now — the side that's hurting, or the side that's saying no? If those are different machines and you're only on one of them, you don't have evidence yet. You have a hypothesis wearing evidence's clothes.

Takeaway

The node connected two minutes after its clock was fixed. The relay's log had contained the answer since the first failed handshake.

Debug from where the rejection happens, not from where the pain is felt. Signed protocols fail on time before they fail on identity. And the fastest path through your next mystery outage is one log line from the machine that said no.