core-vitals/docs/09_Operations_And_Security.md

Operations and Security

Threat model in one paragraph

NetworkCore assumes Redis is on a private network reachable only by trusted servers. The shared auth token guards against accidents (a misconfigured staging server talking to prod) and casual mischief (someone with bus access spoofing a server id). It does not protect against an attacker who already has Redis credentials - at that point they can read the token off any envelope and start signing their own. For hard isolation, rely on network controls: VPC peering, firewall rules, Redis bound to a private interface.

Auth token rollout

NETWORK_AUTH_TOKEN=<long random string>

Pick something long. 32+ chars of randomness is fine. Set it the same way on every server.

When rotating: do a rolling restart. Mid-rotation, servers running the old token will reject messages from servers running the new token (and vice versa). messagebus_rejected_total will tick up, and you'll see warning logs at most once per 10s per server. Finish the rollout fast.

If you want zero rejection windows, you'd need a multi-token scheme where receivers accept either the old or new token during transition. That's not in v1.

What is not encrypted or signed

• Redis traffic. Plain TCP. Anyone sniffing the wire between Hytale servers and Redis sees every payload and every token. Mitigations: keep Redis on a private network, run it on the same host as the Hytale server, or use SSH tunnels / a VPC. TLS (rediss://) is a future addition.
• Messages themselves. No HMAC, no signing. A token in the envelope is checked for equality, not for authenticity. An attacker on the bus can craft any envelope.
• The metrics endpoint. No auth at all. Defaults to 127.0.0.1 so only the local machine can hit it. If you bind to a non-loopback address, put a reverse proxy with auth in front.

The metrics endpoint

http://127.0.0.1:9100/metrics by default. Three ways to consume it remotely:

1. Run Prometheus on the same host. Simplest. Prometheus hits 127.0.0.1:9100 and forwards via its own remote-write to a central store.
2. SSH tunnel from the scraper host. ssh -L 9100:127.0.0.1:9100 hytale-server, then point Prometheus at 127.0.0.1:9100.
3. Reverse proxy with auth. Nginx in front of 127.0.0.1:9100, requires HTTP Basic or mTLS. Change metrics.bind to 127.0.0.1 (default) and have nginx be the public face.

Do not change metrics.bind to 0.0.0.0 without one of the above. The endpoint leaks server identity, player counts, JVM internals, plugin state, and roughly what you're doing operationally.

Ops checks at a glance

Question	How to answer
Is this server alive in the network?	/networkcore status shows "Network: not connected" if Redis is down
Are other servers seeing me?	On another server: /networkcore servers - your id should be listed within a few seconds
Is the metrics endpoint up?	curl http://127.0.0.1:9100/metrics
Are servers communicating?	redis-cli -h <host> MONITOR and look for PUBLISH activity
Is Redis in the expected state?	redis-cli -h <host> KEYS 'network:*' shows registry hashes
Are auth rejects happening?	messagebus_rejected_total in /metrics; warning logs Rejected message on ...
What's the per-server RPC latency?	PromQL: rate(rpc_request_duration_seconds_sum_seconds[1m]) / rate(rpc_request_duration_seconds_count[1m])

Common failure modes

"Failed to connect to Redis at host:port" at startup

Redis is unreachable. Check redis.host / redis.port. Check REDIS_PASSWORD. Check whether Redis is actually running. /networkcore status will show "Network: not connected" and the rest of the plugin keeps running with the message bus / registry / RPC services unregistered.

messagebus_rejected_total is climbing

One of your servers has a different NETWORK_AUTH_TOKEN than the rest. Find it (the warning log includes the from server id) and fix.

A server isn't appearing in /networkcore servers on other hosts

1. Did its start() complete? Check its log for "Connected to Redis ... auth=on/off".
2. Is its auth token the same as everyone else's?
3. redis-cli -h <host> HGETALL network:servers:<that-id> - if empty, it's not heartbeating.
4. If the hash is there but isn't appearing on other hosts, the receiver has the wrong token or the receiver's Redis is a different instance entirely (check redis.database).

Servers stay listed for ~15s after crash

Expected. Hard crashes don't send the goodbye. The 15s TTL is what removes them.

RPC requests time out

1. Is anyone actually serving the named service? registerService("<name>") must have been called.
2. Is the handler slow? Check rpc_request_duration_seconds.
3. Is the responder server reachable on the bus? If /networkcore servers shows it, yes.

Sizing notes

• One Redis instance handles thousands of messages/second easily. The bottleneck is almost always subscriber processing, not Redis.
• The MessageBus worker pool is sized 2 core / 8 max. If you're doing heavy fan-in (lots of subscribers all doing real work on the same channel), you may want to bump this. It's a constant in RedisMessageBus right now; promote to config if needed.
• Heartbeat interval (5s) and TTL (15s) are constants. Tighter values mean faster failure detection at the cost of more Redis writes. 5s/15s is fine up to hundreds of servers.
• The local server-registry cache rescans Redis every 5s using SCAN - safe even with many keys, no KEYS blocking. Scales linearly with server count.