init

2026-05-26 00:11:45 -04:00
commit 296c233157
41 changed files with 2829 additions and 0 deletions
@@ -0,0 +1,74 @@
+# Operations and Security
+
+## Threat model in one paragraph
+
+NetworkCore assumes Redis is on a private network reachable only by trusted servers. The shared auth token guards against accidents (a misconfigured staging server talking to prod) and casual mischief (someone with bus access spoofing a server id). It does *not* protect against an attacker who already has Redis credentials - at that point they can read the token off any envelope and start signing their own. For hard isolation, rely on network controls: VPC peering, firewall rules, Redis bound to a private interface.
+
+## Auth token rollout
+
+```
+NETWORK_AUTH_TOKEN=<long random string>
+```
+
+Pick something long. 32+ chars of randomness is fine. Set it the same way on every server.
+
+When rotating: do a rolling restart. Mid-rotation, servers running the old token will reject messages from servers running the new token (and vice versa). `messagebus_rejected_total` will tick up, and you'll see warning logs at most once per 10s per server. Finish the rollout fast.
+
+If you want zero rejection windows, you'd need a multi-token scheme where receivers accept either the old or new token during transition. That's not in v1.
+
+## What is *not* encrypted or signed
+
+- **Redis traffic.** Plain TCP. Anyone sniffing the wire between Hytale servers and Redis sees every payload and every token. Mitigations: keep Redis on a private network, run it on the same host as the Hytale server, or use SSH tunnels / a VPC. TLS (rediss://) is a future addition.
+- **Messages themselves.** No HMAC, no signing. A token in the envelope is checked for equality, not for authenticity. An attacker on the bus can craft any envelope.
+- **The metrics endpoint.** No auth at all. Defaults to `127.0.0.1` so only the local machine can hit it. If you bind to a non-loopback address, put a reverse proxy with auth in front.
+
+## The metrics endpoint
+
+`http://127.0.0.1:9100/metrics` by default. Three ways to consume it remotely:
+
+1. **Run Prometheus on the same host.** Simplest. Prometheus hits 127.0.0.1:9100 and forwards via its own remote-write to a central store.
+2. **SSH tunnel from the scraper host.** `ssh -L 9100:127.0.0.1:9100 hytale-server`, then point Prometheus at `127.0.0.1:9100`.
+3. **Reverse proxy with auth.** Nginx in front of `127.0.0.1:9100`, requires HTTP Basic or mTLS. Change `metrics.bind` to `127.0.0.1` (default) and have nginx be the public face.
+
+Do not change `metrics.bind` to `0.0.0.0` without one of the above. The endpoint leaks server identity, player counts, JVM internals, plugin state, and roughly what you're doing operationally.
+
+## Ops checks at a glance
+
+| Question | How to answer |
+|---|---|
+| Is this server alive in the network? | `/networkcore status` shows "Network: not connected" if Redis is down |
+| Are other servers seeing me? | On another server: `/networkcore servers` - your id should be listed within a few seconds |
+| Is the metrics endpoint up? | `curl http://127.0.0.1:9100/metrics` |
+| Are servers communicating? | `redis-cli -h <host> MONITOR` and look for PUBLISH activity |
+| Is Redis in the expected state? | `redis-cli -h <host> KEYS 'network:*'` shows registry hashes |
+| Are auth rejects happening? | `messagebus_rejected_total` in `/metrics`; warning logs `Rejected message on ...` |
+| What's the per-server RPC latency? | PromQL: `rate(rpc_request_duration_seconds_sum_seconds[1m]) / rate(rpc_request_duration_seconds_count[1m])` |
+
+## Common failure modes
+
+### "Failed to connect to Redis at host:port" at startup
+Redis is unreachable. Check `redis.host` / `redis.port`. Check `REDIS_PASSWORD`. Check whether Redis is actually running. `/networkcore status` will show "Network: not connected" and the rest of the plugin keeps running with the message bus / registry / RPC services unregistered.
+
+### `messagebus_rejected_total` is climbing
+One of your servers has a different `NETWORK_AUTH_TOKEN` than the rest. Find it (the warning log includes the `from` server id) and fix.
+
+### A server isn't appearing in `/networkcore servers` on other hosts
+1. Did its `start()` complete? Check its log for "Connected to Redis ... auth=on/off".
+2. Is its auth token the same as everyone else's?
+3. `redis-cli -h <host> HGETALL network:servers:<that-id>` - if empty, it's not heartbeating.
+4. If the hash is there but isn't appearing on other hosts, the receiver has the wrong token or the receiver's Redis is a different instance entirely (check `redis.database`).
+
+### Servers stay listed for ~15s after crash
+Expected. Hard crashes don't send the goodbye. The 15s TTL is what removes them.
+
+### RPC requests time out
+1. Is anyone actually serving the named service? `registerService("<name>")` must have been called.
+2. Is the handler slow? Check `rpc_request_duration_seconds`.
+3. Is the responder server reachable on the bus? If `/networkcore servers` shows it, yes.
+
+## Sizing notes
+
+- One Redis instance handles thousands of messages/second easily. The bottleneck is almost always subscriber processing, not Redis.
+- The MessageBus worker pool is sized 2 core / 8 max. If you're doing heavy fan-in (lots of subscribers all doing real work on the same channel), you may want to bump this. It's a constant in `RedisMessageBus` right now; promote to config if needed.
+- Heartbeat interval (5s) and TTL (15s) are constants. Tighter values mean faster failure detection at the cost of more Redis writes. 5s/15s is fine up to hundreds of servers.
+- The local server-registry cache rescans Redis every 5s using `SCAN` - safe even with many keys, no `KEYS` blocking. Scales linearly with server count.