Server shows offline or zones don't propagate: a ZoneCloud checklist

Most of these cases come down to the network or a security layer between a server and the Controller, rarely the Agent or Node itself. Work down this list in order: each step rules out a layer.

1. Run the built-in check

On the Agent’s server:

zcloudagent -v

It prints the version, the DNS server it detected, and two connectivity checks, the licence server and your Controller:

DNS Backend: PowerDNS
  pdns_server process: running
License Server:
  Host: …
  IP: …
  HTTPS: Reachable (38 ms)
Zonecloud Controller:
  Host: controller.example.net
  IP: 192.0.2.50
  HTTPS: Reachable (12 ms)
  API: Authorized (25 ms)

What each line rules out:

DNS: FAILED: this server can’t resolve the hostname. Go to step 3.
HTTPS: NOT reachable: nothing answers on port 443 from this server. Look at firewalls, the Controller’s web server, and the WAF (step 2).
API: FAILED (...): HTTPS works, but the Controller refuses or doesn’t understand the request. The message in brackets says why (step 4).
… process: NOT running: the local DNS server is down. The Agent has nothing to read zones from.

2. A WAF or security layer on the Controller’s host

This is the most common cause in our tickets. ModSecurity, Imunify360, BitNinja or a CDN in front of the Controller sees an Agent sending zone lists every minute, decides it looks like an attack, and blocks the Agent’s IP.

The giveaways:

The server goes offline at regular intervals: a temporary ban that expires and comes back.
It works for a while after a restart.
The Agent logs 403 or connection resets.

Check the Controller host’s web server access log for the Agent’s IP and the response codes, then the WAF’s own log. Whitelist your Agents’ and Nodes’ IPs in the WAF. They talk to the Controller’s API all day; that’s normal.

3. Name resolution on the server

The Agent resolves the Controller’s hostname like any other program, through /etc/resolv.conf. Two traps:

cloud-init or the hosting panel rewriting /etc/resolv.conf after a reboot, pointing at a resolver that’s broken or blocks lookups.
A short TTL on the Controller’s hostname, combined with a resolver that fails now and then. Each failed lookup looks like an outage. Raise the TTL of the Controller’s A record.

4. Read the error class the Agent logs

journalctl -u zcloudagent -f      # zcloudnode on a Node

Every failed call to the Controller is logged with a class in square brackets and a hint:

Log starts with	Meaning	Usual fix
`[dns] cannot resolve …`	the Controller’s hostname doesn’t resolve	`SERVER_HOSTNAME` in the config, `/etc/resolv.conf` (step 3)
`[tls] certificate error …`	the Controller’s certificate doesn’t validate	wrong hostname, missing chain, or the server’s clock is off
`[connect] … refused connection on port 443`	nothing listens on 443	the Controller’s web server is down
`[connect] connection to … timed out — firewall blocking port 443?`	packets dropped on the way	firewall rules on either side
`[auth] HTTP 401`	the token is missing or invalid	`TOKEN` in the config
`[auth] HTTP 403`	access denied	wrong token, the server’s IP not registered in the Controller, or a WAF (step 2)
`[api] HTTP 404` / `405`	endpoint not found	Agent and Controller versions out of step: update the older one
`[api] HTTP 413`	payload too large	a very large batch of zones: see step 6
`[network] … unexpected EOF` / `… reset`	the connection was cut mid-request	a proxy, a firewall, or MTU / fragmentation on the path
`[timeout] … timed out`	the request started but didn’t finish in time	a slow Controller, a congested link, or MTU on large payloads

5. Servers with more than one IP

The Controller identifies servers by the IP they connect from. On a server with several addresses, the operating system may pick a different source IP from the one registered in the Controller, and the result is 403. Tell the Agent which address to use in /etc/zcloudagent.conf:

BIND_IP=203.0.113.10

6. Thousands of zones

On servers with several thousand zones, the first sync sends a lot of data. Raise the HTTP timeouts in the config (values in seconds):

HTTP_CONNECT_TIMEOUT=15
HTTP_READ_TIMEOUT=120
HTTP_WRITE_TIMEOUT=120

If [network] … unexpected EOF or [timeout] errors appear only on big requests, while small ones work, suspect the MTU on the path: small packets pass, and large ones are dropped somewhere in between.

7. The service won’t start again

After repeated crashes, systemd stops trying. The units allow 5 starts in 60 seconds, then the journal shows:

zcloudagent.service: Start request repeated too quickly.

Fix the cause first (the journal shows it just before), then clear the counter and start again:

systemctl reset-failed zcloudagent.service
systemctl start zcloudagent.service

8. It isn’t ZoneCloud: DNS-level causes

When the Controller shows the zone and the Nodes have it, but the domain still doesn’t resolve, the problem is in DNS itself:

Glue records at the registrar that don’t match your nameservers’ real IPs, or a stray NS record left in the zone.
A nameserver listening only on IPv4. Queries over IPv6 get REFUSED; add listen-on-v6 { any; }; to BIND.
The same zone defined twice on a Node. BIND refuses to start, with a line like this:
```
zones.conf:55681: zone '0.in-addr.arpa': already exists previous definition: /etc/named.rfc1912.zones:41
```
Here a default reverse zone in the system’s own config clashes with one synced from an Agent. Exclude it at the Agent (EXCLUDE_ZONES) or remove it from the system config.
Transfers refused by the primary: Zone transfers fail with NOTAUTH or REFUSED.