A chain of trust has to start somewhere
Every system that verifies identities, signatures, or certificates ultimately traces that verification back to some root that isn't itself verified by a further layer — it's accepted as the foundation, not derived from something else. That root's integrity is what everything built on top of it inherits; if the root is compromised, every verification that depends on it becomes meaningless regardless of how carefully the layers above it were built.
A purely software-based root — a key stored in a file, protected only by the operating system's own access controls — is exposed to anything that can compromise the operating system it runs on. It's a reasonable choice for lower-stakes cases, but it inherits every vulnerability of the general-purpose system around it.
What hardware adds
A hardware root of trust keeps the key material inside dedicated hardware specifically built to resist extraction, so that compromising the general-purpose software around it doesn't hand over the key itself — the hardware performs the cryptographic operation and returns a result, without the key ever having to leave in a usable form. This raises the cost of compromise substantially for the specific keys that most need it, at the cost of provisioning something less flexible than a software key that can be copied and deployed anywhere.
The tradeoff is usually resolved by reserving hardware-backed roots for the smallest set of keys whose compromise would matter most — the anchors everything else ultimately depends on — rather than applying it uniformly to every key a system uses.