Operations and Security

Threat model in one paragraph

NetworkCore expects Redis to be reachable only by trusted servers. If Redis sits on a different host than the Hytale servers (the common case with hosted game servers), enable redis.tls so the connection is encrypted in transit. The shared auth token guards against accidents (a misconfigured staging server talking to prod) and casual mischief (someone with bus access spoofing a server id). It does not protect against an attacker who already has both Redis credentials and the token - at that point they can craft any envelope they want. The defenses in depth: TLS for the wire, the auth token for accidents, and network controls (firewall rules, private Redis interfaces, hosting-provider ACLs) for the actual attacker scenario.

Auth token rollout

NETWORK_AUTH_TOKEN=<long random string>

Pick something long. 32+ chars of randomness is fine. Set it the same way on every server.

When rotating: do a rolling restart. Mid-rotation, servers running the old token will reject messages from servers running the new token (and vice versa). messagebus_rejected_total will tick up, and you'll see warning logs at most once per 10s per server. Finish the rollout fast.

If you want zero rejection windows, you'd need a multi-token scheme where receivers accept either the old or new token during transition. That's not in v1.

What is and isn't encrypted or signed

Redis traffic. Optional TLS (redis.tls in config). Turn it on whenever Redis runs on a different host than the Hytale server. Without TLS, anyone sniffing the network between the Hytale server and Redis sees every payload and the auth token. With TLS, the connection is encrypted end to end. Use verify_peer: true against any production Redis with a real cert.
Messages themselves. No HMAC, no signing. The auth token in the envelope is checked for equality, not authenticity. An attacker with Redis credentials can craft any envelope and stamp it with the token (if they have it). TLS prevents the token from leaking on the wire, but doesn't prevent forgery by anyone who legitimately has Redis access.
The metrics endpoint. No auth at all. Defaults to 127.0.0.1 so only the local machine can hit it. If you bind to a non-loopback address, put a reverse proxy with auth in front.

The metrics endpoint

http://127.0.0.1:9100/metrics by default. Three ways to consume it remotely:

Run Prometheus on the same host. Simplest. Prometheus hits 127.0.0.1:9100 and forwards via its own remote-write to a central store.
SSH tunnel from the scraper host. ssh -L 9100:127.0.0.1:9100 hytale-server, then point Prometheus at 127.0.0.1:9100.
Reverse proxy with auth. Nginx in front of 127.0.0.1:9100, requires HTTP Basic or mTLS. Change metrics.bind to 127.0.0.1 (default) and have nginx be the public face.

Do not change metrics.bind to 0.0.0.0 without one of the above. The endpoint leaks server identity, player counts, JVM internals, plugin state, and roughly what you're doing operationally.

Ops checks at a glance

Question	How to answer
Is this server alive in the network?	`/networkcore status` shows "Network: not connected" if Redis is down
Are other servers seeing me?	On another server: `/networkcore servers` - your id should be listed within a few seconds
Is the metrics endpoint up?	`curl http://127.0.0.1:9100/metrics`
Are servers communicating?	`redis-cli -h <host> MONITOR` and look for PUBLISH activity
Is Redis in the expected state?	`redis-cli -h <host> KEYS 'network:*'` shows registry hashes
Are auth rejects happening?	`messagebus_rejected_total` in `/metrics`; warning logs `Rejected message on ...`
What's the per-server RPC latency?	PromQL: `rate(rpc_request_duration_seconds_sum_seconds[1m]) / rate(rpc_request_duration_seconds_count[1m])`

Common failure modes

"Failed to connect to Redis at host:port" at startup

Redis is unreachable. Check redis.host / redis.port. Check REDIS_PASSWORD. Check whether Redis is actually running. /networkcore status will show "Network: not connected" and the rest of the plugin keeps running with the message bus / registry / RPC services unregistered.

`messagebus_rejected_total` is climbing

One of your servers has a different NETWORK_AUTH_TOKEN than the rest. Find it (the warning log includes the from server id) and fix.

A server isn't appearing in `/networkcore servers` on other hosts

Did its start() complete? Check its log for "Connected to Redis ... auth=on/off".
Is its auth token the same as everyone else's?
redis-cli -h <host> HGETALL network:servers:<that-id> - if empty, it's not heartbeating.
If the hash is there but isn't appearing on other hosts, the receiver has the wrong token or the receiver's Redis is a different instance entirely (check redis.database).

Servers stay listed for ~15s after crash

Expected. Hard crashes don't send the goodbye. The 15s TTL is what removes them.

RPC requests time out

Is anyone actually serving the named service? registerService("<name>") must have been called.
Is the handler slow? Check rpc_request_duration_seconds.
Is the responder server reachable on the bus? If /networkcore servers shows it, yes.

Sizing notes

One Redis instance handles thousands of messages/second easily. The bottleneck is almost always subscriber processing, not Redis.
The MessageBus worker pool is sized 2 core / 8 max. If you're doing heavy fan-in (lots of subscribers all doing real work on the same channel), you may want to bump this. It's a constant in RedisMessageBus right now; promote to config if needed.
Heartbeat interval (5s) and TTL (15s) are constants. Tighter values mean faster failure detection at the cost of more Redis writes. 5s/15s is fine up to hundreds of servers.
The local server-registry cache rescans Redis every 5s using SCAN - safe even with many keys, no KEYS blocking. Scales linearly with server count.

5.9 KiB Raw Permalink Blame History