--- title: "DOTS NetCode at scale" slug: /workflows/dots-netcode-scale/ markdown: /workflows/dots-netcode-scale.md section: netcode tier: 2 summary: "Relevancy, importance, bandwidth budgets and queues that stop a spike." sources: ['dots/netcode-scale.md'] canonical: https://unityworkflowlibrary.dev/workflows/dots-netcode-scale/ read_first: /read-first.md --- > Read first, on every prompt: [non-destructive rules](/read-first.md). # DOTS NetCode at scale > Relevancy, importance, bandwidth budgets and queues that stop a spike. Split from the [Netcode overview](/workflows/networking-overview/) page (which had passed the ~350-line budget) because "does it work" and "does it still work at 1,000 players" are different questions with different rules. Pitfalls and prediction/ghost mechanics stay there; everything about what reaches the wire, and what stops it blowing up, is here. Researched against Netcode for Entities 1.3 documentati. Developer target driving this: one dedicated server holding as many players as possible across many planets, staying a competitive FPS, where players land and build structures anywhere. ## DIAGNOSTICS-GROW-WITH-THE-WORKLOAD (0.3.87, HARD) any feature that adds network traffic, authoritative simulation, resident memory or async/persistence queue pressure extends the existing diagnostics in the same change. Humans and agents need observed evidence to locate faults; source inspection alone cannot establish actual cost or network behavior. - Expose readable purpose, authority and world/connection/transport scope, units, sample interval/age, actual state, useful work, limits, queue/admission/failure effects and measured bytes/packets where the carrier permits attribution. - Keep simulation, replication and persistence budgets separate. Configured rates are not achieved rates, inventory/activity is not useful work, inclusive system scheduling is not completed worker CPU, and shared transport totals cannot become per-feature bytes by relabeling. Mark unknown/stale/disabled explicitly. - Save bounded structured reports and readable summaries with UTC, compiled-build identity, configuration, sampling gaps, outcomes and interpretation limits. Preserve small verification evidence and retained private-capture locations; exclude secrets and sensitive payloads. Avoid unbounded per-tick logging. - Use live or saved diagnostic evidence to establish the symptom and before/after result, then trace the owning source. Verify counters, failure/unavailable states and observer cleanup with the feature. Bound and disclose diagnostics overhead. - Correct interaction freshness and authority govern optimizations. Population claims require adverse-network, distributed and dense workload evidence. A low idle bandwidth figure or another game's capacity never clears that gate. ## The budget model — know it before optimising anything - **SNAPSHOT-TARGET-IS-NOT-A-BANDWIDTH-CAP (0.3.85):** snapshot targets are desired bytes per receiving connection, not a fixed transport packet or bits-per-second limit. NetCode 6.5 can increase the target when even one ghost will not fit. Fragmentation and non-snapshot traffic have separate costs; measure actual packets, bytes and queues. Chunk priority ranks available snapshot content. - **Importance is calculated PER CHUNK, not per entity** (documented explicitly). Your chunk layout IS your bandwidth layout. A shared component (`GhostImportancePerChunkDataType`) is what forces entities into a chunk, and moving an entity between chunks is a structural change - so a scheme that re-chunks entities every frame costs more than it saves. - **Capacity depends on workload.** Measure CPU, memory, traffic and persistent queues separately in distributed, dense and mixed layouts. Dense relevant-pair fan-out may grow quadratically; a linear slope cannot certify a population. - **PROCESS-WIDE-TICK-OVERLOAD-IS-THE-MESHING-TRIGGER (0.3.85 qualification):** a hot encounter can delay other regions sharing one authority tick loop. Measure distant players during a dense fight. Interference is a candidate partition trigger after focused fixes, not evidence that isolation triggers first. Multiple NetCode worlds need not share a clock, though they contend for resources. Meshing may distribute simulation but does not eliminate required visible-pair fan-out, and adds boundary work. ## Choose the next lever from evidence Choose the next lever from measured cost after satisfying interaction correctness. Remove unnecessary disclosure, reduce optional freshness or prune fields only when their dependencies and required observer state remain intact. Distance is not threat: preserve potential first-shot interactions and frame/support state. ## Relevancy is a SECURITY boundary, not a bandwidth trick (HARD) Unity documents `GhostRelevancy` as supporting a server-side, "anti-cheat-capable" fog of war - preventing clients from knowing about entities they should not be able to see. In a competitive FPS this is the difference between shipping a game and shipping a wallhack: **anything replicated to a client is known to that client**, whatever the renderer does with it. Client-side culling is a graphics feature and has no security value whatsoever. - **`GhostRelevancyMode.SetIsRelevant` makes nothing replicate by default**, and you opt entities in per connection via `GhostRelevancySet`. That is the correct default for an open world where no player can ever see everything. `SetIsIrrelevant` is the opposite default and only suits games where seeing the whole world is normal. - `GlobalRelevantQuery` / `DefaultRelevancyQuery` covers "always relevant to everyone" (scoreboards, session state). The per-connection set takes precedence over it. - **The despawn is a misnomer, and it is a gameplay bug waiting to happen.** When a ghost becomes irrelevant, the client is told the entity was DESTROYED. Do not hang death animations, ragdolls, audio or score effects off ghost destruction - drive them from explicit state (an `IsDead` / `IsCorpse` component) or a player walking out of range plays a death sound behind them. - Relevancy also serves ordinary product needs: hidden-information mechanics, per-player NPCs, and pausing replication entirely for a dead client on a long respawn. ### NEAREST-N-CAP-NEEDS-A-REENTRY-HOLD - a ghost that leaves and returns inside the despawn delay is orphaned (0.4.75) A per-connection nearest-N cap re-ranks on every rebuild. When candidates sit at almost equal distance around the cutoff (a pile of bodies, a crowd), a small move by the viewer swaps which ones make the cut. A ghost can leave the relevancy set and come back a tick or two later. The client does not remove an interpolated ghost the moment it leaves: its despawn waits until the interpolation tick passes. A ghost that returns first is spawned again under the same ghost id and spawn tick while the old copy is still in the spawned-ghost map. The client logs "Ghost ID N has already been added to the spawned ghost map", and it keeps one copy that no snapshot will ever despawn. Later it logs "Trying to despawn a ghost ... does not have a ghost component". Measured: about 100 kills in one pile past a 32-body cap left 71 body ghosts on the client against 17 on the server, with 42 duplicate ids and 65 errors. The first error came 4 minutes in, when the pile crossed the cap. - **Hold a capped-out ghost out of that connection's set for longer than the client's despawn delay.** Key the hold by (connection, ghost), and expire it by time. Apply it to every capped kind, never to always-relevant ghosts (the developer's own). Clear it when relevancy is disabled. A 0.5 s hold at 60 Hz with default interpolation measured clean. - **Radius hysteresis does not cover this.** Enter/exit radii guard distance, not rank. Rank churn inside the radius needs its own hold. - **Test both halves.** EditMode: move a rival in and out across the cutoff, then assert the dropped ghost stays out on the next tick and returns after the hold. Play: build a pile past the cap and log both worlds every few seconds. The client count should be min(server, N), with no duplicate ghost ids and no ghost-map errors. After the hold: the server peaked at 51, the client held at the cap with 0 duplicates and 0 errors, and it matched the server in every sample once the pile fell below the cap. - **Cost:** a dropped ghost can take up to the hold to return. A crowd at the cap still trades slots, but each ghost changes membership at most once per hold. Show the held count in the relevancy diagnostics. ## Static optimization is what makes "build anywhere" affordable (HARD) A player-built structure is the worst possible dynamic ghost: it never moves, it persists forever, and the count only ever grows. Left on the default Dynamic optimization mode, a filling world spends its snapshot budget re-confirming that houses are still where they were. - **Set Optimization Mode to Static on anything that changes rarely.** Documented behaviour: a static ghost is **not sent at all while it is not changing** (a changed `GhostField` is still sent regardless of the mode). Dynamic is the default and is tuned for things that change constantly. - Structures must ALSO be relevancy-scoped and residency-streamed. Static optimization stops the re-sends; it does not stop a client being told about ten thousand distant houses once. - Design the built object so its replicated state is genuinely static: ownership, transform and a content hash on the ghost; anything that ticks (doors, lights, damage state) is a separate ghost or a separate enableable component, so one animated door does not un-static an entire building. ## Do not pay serialization cost for data that never changes (HARD) Non-obvious and documented: when a job has WRITE access to a `GhostField`, Netcode must serialize the value and compare it against the last synchronized version to decide whether it changed. **You pay that cost even when the value is identical.** Grant write access only where a write actually happens; a job that reads-but-declares-write silently taxes every tick. **Under Static optimization a declared write does not tax the skip, it CANCELS it.** Read in the send system's source: the static early-out calls `chunk.DidChange(...)` for EVERY ghost component and bails on the first one that moved, and an ECS read-write type handle bumps that chunk version when the handle is ACCESSED, not when a write lands. So one `RefRW` in a per-tick query - one that writes nothing, one that `continue`s on its first line - makes an authored-Static ghost behave exactly like a Dynamic one, with no warning and no error. Flipping the mode on such a ghost measures **zero improvement**, which reads as "static optimization does not work here" and sends the next session looking in the wrong place. Before trusting an authored mode, audit every system that queries the type for access it does not use: `RefRW` -> `RefRO`, and a buffer to a read-only `BufferLookup`, since `SystemAPI.Query>` is read-write and there is no read-only query form. **Replicate the deadline, never the countdown.** A "seconds remaining" field ticked by the server is the same defect wearing data: it changes every tick by construction, so its ghost can never be skipped however motionless the entity is. Replicate the tick it expires ON - written once at spawn, never again - and let the client subtract its own network time. Measured on stationary lootable bodies: ten of them cost 1.1 kB/s at 24 Hz, more than every moving creature in the world put together, and went to zero bytes for the whole time they lay there. Same information, one write. **Audit `UsePreSerialization` against the mode.** A pre-serialized ghost is serialized every tick even when it is sent to nobody - documented - which is precisely the work static optimization exists to skip. Pre-serialization pays for types sent constantly to many connections; on a rarely-changing type the two settings are a contradiction. **Surface the mode NEXT TO the measured bytes.** A traffic readout that names the heaviest ghost type does not say whether being heavy is that type's job. Publish, per type, `GhostCollectionPrefabSerializer.CanBeStaticOptimized` - which reports the RUNTIME demotion a ghost group or a replicated child causes, not the authored setting - so an authored-static type that is silently running dynamic is visible instead of inferred. ## Throttles, and the caveat that makes them lie - NetworkStreamSnapshotTargetSize is desired snapshot bytes per connection. It may grow when serialization cannot fit a ghost. It does not budget RPCs, streaming, retransmissions or aggregate network egress. Inspect the installed sender rather than promising a hard cap. - `GhostSendSystemData.MaxSendEntities` / `MaxSendChunks` cap what enters one snapshot. - MinSendImportance and MaxSendRate influence selection, but do not promise an exact per-observer cadence under contention or structural changes. Measure achieved sends and state age, including first-send pressure. - **The caveat: these are applied per chunk, AFTER a chunk's contents are added.** Real sends exceed the limit - `MaxSendEntities` 100 with two 99-entity chunks sends 198. Treat them as pressure, not as a guarantee, and measure the actual counters. - Distance importance needs the FULL setup or it shows no gain: `GhostImportance` + `GhostDistanceData` on ONE singleton, `GhostConnectionPosition` on every connection before `GhostSendSystem`, and the `MaxSend*` caps alongside it. ## Importance governs SNAPSHOTS ONLY — everything else needs a queue (HARD) This is where "data blows up". Importance scaling and snapshot budgets do NOT govern RPCs, spawn storms or structural changes. Five hundred players placing a block in the same tick is not throttled by anything Netcode does for you: it is 500 RPCs, 500 validations, 500 structural changes and 500 new ghosts whose `FirstSendImportanceMultiplier` makes them jump the queue - so a build spike steals snapshot budget directly from combat, which is the thing that must never degrade. - **Every player-initiated, unbounded action goes through a server-side bounded queue** with a per-tick drain budget, backpressure, and an explicit rejection path the client can render. Never "process all pending this tick". - **Rate-limit per connection**, not just globally, or one client can consume the whole drain budget. - **Batch the structural changes** through an `EntityCommandBuffer` at a defined barrier, or a query-wide operation - never one structural change per request. - The drain budget is a measured number tied to the tick budget, and it is authored in one config singleton alongside the other capacity knobs, not scattered as constants. - Validate BEFORE enqueueing where validation is cheap, so the queue holds only work that will actually happen. ## The client is a read-only replica (HARD) - **Ghost data flows server -> client. The ONLY client -> server state channel is tick-tagged `ICommandData`**, and an RPC that mutates authoritative state without server validation is a vulnerability, not a shortcut. - **Prediction is a latency optimization, never authority.** The server snapshot wins every rollback. A predicted placement is a local guess that must be confirmed or corrected. - **Server-only state never reaches the client prefab.** Use the Server `GhostPrefabType` / prefab bit so it cannot serialize by accident - not a runtime `AddComponent` on the server, which changes the archetype and the ghost hash. - Trust nothing positional from a client: build placement, hit registration and interaction range are validated against the SERVER's world, at the server's tick. ## Prediction cost - **Prediction cost scales with predicted ghost count.** Use prediction switching to opt distant ghosts out - in an open world most ghosts should not be predicted at all. - At high ping (20+ re-simulated frames) the `PhysicsSimulationGroup` inside `PredictedFixedStepSimulationSystemGroup` pays real scheduling overhead per re-simulated step. Forcing physics onto the main thread (a `PhysicsStep` singleton with Multi Threaded off) can be faster than the parallel path there - measure, do not assume the threaded one wins. - A server world is not idle on off frames: it time-slices sending across connections, and you can schedule your own expensive work there via the rate manager's `WillUpdate`. ## Related the [Netcode overview](/workflows/networking-overview/) page for ghost/prediction mechanics and the failure modes, including GHOST-QUANTIZATION-IS-AN-INT32-WORLD-BOUND (a quantized position is an int32 on the wire, so it hard- bounds world size - decisive at planetary scale, replicate a datum-relative offset). the ecs core page for chunk locality and lookup cost. the system blueprint page Shape B for the system shape all of this lives in.