Server Virtualization Systems · Team 3

High
Availability.

Keeping virtual machines online when hardware fails.

HypervisorsVMware ESXi · Proxmox VE
Duration20 min
IncludesLive demo · Q&A
HOST A HOST B HOST C HOST D

Agenda

What we'll cover.

01 What is High Availability?
02 Core Concepts & Vocabulary
03 HA Architecture
04 vSphere HA on VMware ESXi
05 HA on Proxmox VE
06 Side-by-Side Comparison
07 Do's & Don'ts
08 Live Demo & Q&A

01 · Definition

What is High Availability?

A system designed to keep running — even when a part of it doesn't.

HA removes single points of failure so services stay reachable during hardware failure, maintenance or unexpected outages. In a virtualization context: if a host dies, the VMs it ran are automatically restarted on a surviving host.

FAILED HOST ✕ VM 01 VM 02 VM 03 auto-restart SURVIVING HOST ✓ VM 01 VM 02 VM 03 No admin intervention required

01 · Why it matters

Availability is measured in nines.

Each additional nine cuts allowed downtime by roughly 10×. Production targets typically start at three nines.

99%
3.65 daysdowntime / year
99.9%
8.77 hoursdowntime / year
99.99%
52.6 mindowntime / year
99.999%
5.26 mindowntime / year
HA realistically delivers three to four nines. Five nines requires geo-redundancy, replicated data and a much bigger budget.

02 · Vocabulary

Core concepts of HA.

SPOF
Single Point of Failure
Any component whose failure brings the whole system down. HA's main goal is to eliminate them.
Redundancy
N+1
Spare capacity ready to take over (second host, second NIC, second PSU) so a failure has somewhere to go.
Failover
Automatic recovery
Moving a workload to a healthy node. With HA this is automatic — no admin needed at 3 AM.
Heartbeat
"I am alive"
Periodic signal nodes exchange. If the heartbeat stops, the cluster assumes the node is dead and reacts.
RTO
Recovery Time Objective
How long the service may be down. HA typically delivers an RTO of 1–5 minutes (the time to restart the VM).
RPO
Recovery Point Objective
How much data may be lost. HA alone has RPO ≈ 0 for stateless workloads — but it is not a backup.

03 · Architecture

What a HA cluster needs.

CLUSTER HEARTBEAT Host A Hypervisor · Active running 3 VMs Host B Hypervisor · Active running 2 VMs Host C Hypervisor · Spare N+1 reserve REDUNDANT NETWORK Shared Storage iSCSI · NFS · FC · vSAN · Ceph VM disks live here, not on the hosts
Cluster of hosts

≥ 2 hypervisors (3 recommended for quorum) managed as one logical pool.

Shared storage

VM disks live outside the hosts. A surviving host can read the same files instantly.

Redundant network

Multiple physical paths so a single switch/cable failure doesn't isolate a host.

Heartbeat

Hosts exchange constant "I'm alive" signals. Silence = failure = trigger failover.

VMware ESXi 04 · vSphere HA — Failure detection timeline

How vSphere HA reacts.

Hosts run an HA agent (FDM). One is elected master, the rest are slaves. The master watches the heartbeat — if it goes silent, this is what happens.

T = 0 s
Normal heartbeat
Master receives a network heartbeat from every slave every 1 s.
T ≈ 3 s
Heartbeat lost
Master stops receiving heartbeats from Host A. Suspicion begins.
T ≈ 15 s
Datastore check
Master checks the heartbeat datastores — is the host writing files?
T ≈ 30 s
Declared dead
No network, no datastore heartbeat → host A is officially marked as failed.
T ≈ 90 s
VMs restarted
All VMs of Host A are powered on automatically on the surviving hosts.
Typical RTO ≈ 60–120 s from kernel panic to a running VM on another host. A user might never notice the failover for a web app.
VMware ESXi 04 · Admission Control + our lab

Reserving the safety net.

Admission Control blocks new VMs from powering on if doing so would eat the spare capacity reserved for failover. Without it, failover can fail silently because there's no room left.

Cluster CPU capacity3 hosts · "Tolerate 1 failure"
Cluster memory capacitysame policy
Used by running VMs
Reserved for failover
Free
Our lab configuration
  • 3-host ESXi cluster on HP servers, managed by vCenter
  • Shared storage via iSCSI from TrueNAS (10.0.13.3:3260)
  • Two heartbeat datastores: iSCSI-Datastore-01 + iSCSI-Heartbeat
  • VM Monitoring enabled, Restart Priority = Medium
  • Admission Control set to tolerate 1 host failure
Why two heartbeat datastores? A single one is itself a SPOF. Without redundancy HA can't tell a real failure from a storage glitch.
Proxmox VE 05 · The HA stack

No central controller. Every node equal.

Proxmox HA is built as a layered stack. The lower layers handle communication and consensus; the upper ones decide what to actually run.

ha-manager
CLI & web UI for adding resources (VMs/CTs), HA groups, and priorities
CRM
Cluster Resource Manager — runs on the elected master; decides where each VM should run
LRM
Local Resource Manager — runs on every node; receives orders from CRM, actually starts/stops VMs
pmxcfs
Distributed cluster filesystem mounted at /etc/pve, synchronized across all nodes
Corosync
Cluster engine — sub-second multicast heartbeat, requires < 2 ms latency between nodes
Watchdog
Hardware timer that hard-resets a node which loses quorum — provides physical fencing
Proxmox VE 05 · Quorum & resource states

Why 3 nodes is the minimum.

Proxmox requires (n / 2) + 1 nodes to agree before taking any action. With 2 nodes and 1 failure, no majority — cluster freezes to prevent split-brain.

3-node cluster · 1 dies
1UP
2UP
3
2 ≥ (3/2)+1 = 2 → quorum OK → failover proceeds
2-node cluster · 1 dies
1UP
2
1 < (2/2)+1 = 2 → no quorum → cluster frozen
HA resource states
  • started — HA keeps it running, restarts on failure
  • stopped — HA keeps it off, restarts if someone powers it on
  • ignored — temporarily out of HA scope
  • disabled / error — manual intervention required
HA Groups
  • Priorities per node — higher wins, prefer "stronger" hosts
  • restricted — VM may only run inside this group
  • nofailback — don't bounce back when the preferred node returns
ha-manager add vm:100 --group prod --max_restart 3 --max_relocate 2

06 · Comparison

vSphere HA vs. Proxmox HA.

Feature VMware vSphere HA Proxmox HA
LicenseCommercial · vSphere Standard or higherOpen source · free (optional support sub.)
Central managementvCenter requiredNone — any node's web UI works
Minimum nodes2 (3 recommended)3 (for quorum)
Cluster engineFDM agent + vCenterCorosync + pmxcfs
Failure detectionNetwork HB + datastore HB + isolation pingCorosync HB + hardware watchdog
FencingLogical (isolation response policy)Physical (watchdog hard-reset)
Shared storageRequired (VMFS / NFS / vSAN)Required (Ceph / NFS / iSCSI / ZFS-repl.)
Typical RTO≈ 1 min after failure detected≈ 2 min (watchdog + restart)

07 · Best practices

Do's & Don'ts.

DO Build for resilience
  • Use shared storage — it's not optional for true HA
  • Run at least 3 nodes so quorum survives a single failure
  • Configure ≥ 2 heartbeat datastores (vSphere) / a real fencing device (Proxmox)
  • Reserve capacity with Admission Control — otherwise failover may fail silently
  • Test failover regularly (kill a host, verify RTO)
  • Monitor cluster health and alert on quorum loss
DON'T Common mistakes
  • Don't confuse HA with backup — HA protects against host failure, not data loss
  • Don't run a 2-node Proxmox cluster — no quorum after one node fails
  • Don't skip fencing — risk of split-brain and data corruption
  • Don't oversubscribe — if every host runs at 90 %, failover has nowhere to land
  • Don't use a single network path between cluster nodes
  • Don't put heartbeat datastores on the same physical disk array
Proxmox VE 08 · Live demonstration

Failing over a VM on Proxmox HA.

We force a node to die and watch the watchdog fence it, quorum reassign the VM, and another node start it up — all without touching the web UI.

STEP 01
Baseline

Show demo-vm running on pve-01, registered as HA resource (ha-manager status).

STEP 02
Kill the node

Hard-reset pve-01 via iLO — simulates a sudden hardware failure, no clean shutdown.

STEP 03
Watchdog fences

Corosync loses the node, quorum drops it. After ~60 s the watchdog confirms it's gone.

STEP 04
VM relocated

CRM reassigns to pve-02, LRM starts it. Observed RTO: ~2 min.

Watch for: the Datacenter → HA panel — resource state cycles from started → fence → recovery → started on the new node. No one is clicking anything.

Questions?

Thank you — High Availability, Team 3.

High Availability · Team 3
01 / 14
→ Space next slide
← previous slide
T theme · F fullscreen
1-9 jump to slide · ? toggle help