Game state machines explained: FSMs, transitions, and AI behavior

The three building blocks

An FSM has three concepts you already use informally even if you never named them:

• States — mutually exclusive modes with their own update and render behavior. A player controller might have Idle, Run, Jump, Fall, Attack, and Hurt.
• Transitions — directed edges from one state to another, often labeled with the condition that permits the change. Run → Jump when the jump button is pressed and the character is grounded.
• Events — inputs or world signals that the machine evaluates each frame or on notification: button presses, collision contacts, animation completion callbacks, timer expiry, network packets.

The machine holds a single current state. On each tick it runs that state’s update logic, then checks outgoing transitions in priority order. The first transition whose guard passes fires: the old state exits, the new state enters, and the cycle repeats. That enter/update/exit lifecycle is where you hook animation triggers, sound one-shots, and stat changes — keeping side effects out of generic update code.

A minimal player controller example

Consider a platformer character. In pseudocode the FSM might look like this:

enum PlayerState { Idle, Run, Jump, Fall, Attack }

function update(state, input, physics) {
  switch (state) {
    case Idle:
      if (input.jump && physics.grounded) return Jump
      if (input.moveX != 0) return Run
      break
    case Run:
      if (input.jump && physics.grounded) return Jump
      if (input.moveX == 0) return Idle
      if (input.attack) return Attack
      break
    case Jump:
      if (physics.velocityY > 0) return Fall   // apex passed
      break
    case Fall:
      if (physics.grounded) return Idle
      break
    case Attack:
      if (attackAnim.done) return Idle
      break
  }
  return state  // no transition
}

The important detail: movement physics (gravity, velocity integration) can live outside the FSM or inside state-specific update functions. Many engines split locomotion FSM from action FSM so attacking does not fight with falling. That separation prevents the classic bug where you cannot jump because the attack state forgot to check grounded status.

Table-driven FSMs store transitions as data rows (from, event, guard, to, action) instead of a giant switch. Designers can edit CSV or JSON; runtime code walks the table. Unity’s Animator, Unreal’s Animation Blueprints, and Godot’s AnimationTree are visual incarnations of the same idea.

Transition guards and priority

Real FSMs need more than boolean flags. A guard is a predicate evaluated before a transition fires:

• Cooldowns — cannot dodge again for 300 ms after the last dodge.
• Resource checks — attack only if stamina >= 10.
• Animation phase — combo chain only during frames 12–18 of the swing clip.
• External preconditions — ledge-grab only when a forward ray hits a grabbable surface below the hands.

When multiple transitions could fire on the same event, define explicit priority. Higher-priority edges win. Without ordering, Hurt might lose to Attack because both saw the same input frame — producing invincibility bugs players notice immediately.

Global transitions

Some transitions apply from any state: take damage → Hurt, fall into void → Dead, pause menu → Paused. Implement these as a separate pass evaluated before state-local transitions, or as super-state wrappers (see hierarchical FSMs below). Global edges are powerful but dangerous: a poorly scoped → Idle global transition can cancel cutscenes mid-animation.

Hierarchical and stacked state machines

Flat FSMs hit a wall when behavior nests. A character might be Grounded → Running while simultaneously Combat → Blocking. Two independent dimensions multiply states combinatorially if flattened: RunBlocking, JumpBlocking, FallBlocking…

Hierarchical FSMs (HFSM)

A hierarchical machine groups states under parents. The child inherits transitions from the parent; entering a child implicitly enters ancestors. A Combat super-state might define global → Hurt on damage while children LightAttack, HeavyAttack, and Block handle specifics. Exit from any child returns to the parent’s default child (often idle within combat).

State stacks

A pushdown automaton (state stack) pushes temporary modes on top of persistent ones: open inventory pushes Menu over Exploring; closing pops back. Dialogue overlays, photo mode, and emote wheels use stacks so underlying world simulation can pause or continue depending on design. Stacks are simpler than full HFSMs for modal UI but do not model parallel concurrent behavior as cleanly.

Animation and the visual layer

Animation systems are FSMs in disguise. Each clip or blend node is a state; transitions sync to foot contacts, root motion, or cross-fade durations. Mismatch between logic state and animation state causes the infamous sliding-feet bug: logic says Run but the anim still plays Idle because the transition threshold was never met.

Best practice: treat animation as a follower of gameplay FSM, not the source of truth. Gameplay decides Attack; animation receives playAttackClip() on enter. For locomotion blending, 1D blend trees (speed parameter) or 2D blend spaces (speed + direction) interpolate between clips while the FSM stays in a single Locomotion state — reducing transition count without sacrificing visual quality.

Root motion complicates this: animation drives position instead of code. Gate root motion to specific states and disable it during hit reactions so designers do not accidentally walk the player off ledges during stagger animations.

Enemy AI on FSMs

Simple enemies map cleanly to patrol → alert → chase → attack → retreat → dead. Each state runs behavior for one frame inside the broader update loop:

• Patrol — follow waypoints or idle at post; listen for perception events.
• Alert — turn toward noise, play bark animation, start suspicion timer.
• Chase — pathfind toward last known position using A* or navmesh queries each tick or on a staggered schedule.
• Attack — stop movement, wind up, apply hitbox, respect range and line-of-sight guards.
• Retreat — flee when health below threshold or allies call for help.

Perception (vision cones, hearing radius, last-seen position decay) feeds events into the FSM rather than living inside each state. That keeps sensing logic centralized and testable. When many enemies share one graph, store transition tables in data so tuning does not require recompilation.

FSMs vs behavior trees vs utility AI

FSMs excel when modes are few, transitions are well understood, and designers need predictable flow. They struggle when behavior is reactive and priority-driven with many overlapping goals — exactly where other patterns shine.

Behavior trees (BTs)

BTs evaluate a tree of selectors and sequences each tick: try attack if in range, else chase if target visible, else patrol. They handle interruption gracefully — a higher priority subtree preempts without rewiring explicit transition matrices. Most AAA enemy AI uses BTs or hybrids (BT selects goals, FSM executes locomotion phases).

Utility AI

Utility systems score every action (hide, flank, shoot, reload) with curves over context variables (distance, ammo, cover quality). The highest score wins each decision cycle. Excellent for squad tactics and emergent behavior; harder to debug than a drawn FSM diagram.

When to stay with FSMs

• Player controllers and input buffering with clear mode boundaries.
• UI wizards, match flow (lobby → countdown → playing → results).
• Boss phases with scripted pattern sequences.
• Networking: FSMs replicate cleanly as a single enum + timestamp per entity.

A common production path: start with FSMs, extract shared conditions into a blackboard, migrate AI to BTs when transition count exceeds roughly twenty per agent type.

Networking and determinism

In multiplayer, authoritative servers usually own gameplay FSM state. Clients predict locally for responsiveness, then reconcile when the server enum disagrees. Keep states coarse-grained — Jump not JumpFrame17 — to limit bandwidth and desync repair cost.

Pair FSM changes with the same fixed timestep discipline described in our frame timing guide: if simulation uses a fixed dt, transition guards that depend on timers must use simulation time, not wall clock, or replays and rollback netcode will diverge.

For deterministic lockstep or rollback fighters, FSM transitions must depend only on synchronized inputs and simulation state — never on frame-rate-varying raycast hit order unless that order is itself deterministic.

ECS integration

In an entity component system, store StateId and StateTime as components. A StateMachineSystem iterates matching entities, dispatches enter/update/exit to handler tables keyed by state id, and writes the next state back to the component. Animation and physics remain separate systems reading the same state component.

Avoid embedding FSM logic inside render or collision systems — cross-system coupling makes it impossible to run simulation at a different rate from presentation. The ECS pattern shines when dozens of enemy types share one state machine driver but different transition tables loaded from assets.

Common pitfalls

• Transition explosion — every new state adds O(n) edges. Refactor to hierarchy, substates, or BTs before the diagram becomes unreadable.
• Missing exit cleanup — attack state enables hitbox on enter but forgets to disable on exit when interrupted by Hurt. Always pair enter/exit side effects.
• Same-frame double transitions — allow at most one transition per tick unless you explicitly queue events.
• Implicit states — boolean flags outside the FSM recreate hidden modes. If isDashing changes behavior, make Dash a real state.
• Untestable guards — pure functions for guards enable unit tests without spinning up the whole engine.
• Debug visibility — log state changes in development; overlay current state on screen for QA. Most “random” bugs are wrong transition priority.

Practical checklist

1. Draw the diagram before coding — states as nodes, labeled edges.
2. Define enter/update/exit hooks for every state.
3. Centralize global transitions (damage, death, pause).
4. Assign transition priority explicitly.
5. Separate locomotion and action FSMs when combos meet platforming.
6. Drive animation from gameplay state, not the reverse.
7. Store tables in data when designers tune more than engineers code.
8. Log transitions in dev builds — you will need the trace.

Finite state machines are not glamorous, but they are the scaffolding that keeps game logic legible as teams and feature lists grow. Master the flat FSM first; add hierarchy, stacks, or behavior trees only when the graph tells you it is time.

Related reading

• Game loop and frame timing explained — fixed vs variable timestep, input/update/render phases, and deterministic simulation
• Entity component system (ECS) in games explained — storing state components and dispatching FSM systems at scale
• Pathfinding in games explained — chase and patrol states feeding movement from A* and navmesh queries
• Collision detection in games explained — hitbox enable/disable tied to attack and hurt state enter/exit hooks