Here is the thing, in the age old fight between defense and offense, offense tends to always win in the end. We used clubs and spears, so we built armor to protect against that. We then built swords to pierce that armor, so the armor became thicker. Guns took care of that so eventually we built tanks. Anti tank guns took those out. Well you get the point.
Hundreds of years from now there may BE no armor that can stand up against futuristic weaponry. Shields may be the next "solution" in the never ending struggle between offense and defense. So if armor means squat and shields are the only real protection in the future, then it stands to reason that engineers wouldn't bother overengineering our mech's with a fully encased metal armor. It may also explain why the hard driver mode will have no statistical difference, because all that extra encasing is not really any protection at all.
...That's just my theorycrafting based on how the game seems to be heading.
And shields lead to weaponry designed to bypass/disrupt it, which in turn leads to defenses underneath the shields to prevent pilots from dying because when you are on a far away frontier world nothing is more important then the lives of your pilots, if only to save money on trying to replace them.
Further, since so many of the worlds they had found had non-sentient life the majority of creatures they fight are of a physical nature. If the shields go down an armor layer still keeps the pilot alive so that the shields can recharge. Without the armor layer the moment the shields drop is the moment that the pilot gets impaled/their guts ripped out/crushed and now you got to spend money to replace that casualty.
The reason why HDM does not affect stats is because our overall defense mechanically as far as gameplay is concerned acts as if it is
always there. Even when the player would rather see their character instead of them being encased in the armor layer.
Just ask the Israeli military just how important their tank crews are. They only happen to make some of the toughest tanks in the world with the best crew safety on the planet.
It is also the reason why all mech design uses fully encased armor designs even when they have energy shields.
Also I want to point this out:
Swords were made to lacerate targets to induce death by heavy bleeding across a large area (and having a cheap short range weapon, spears were poor at close combat since the majority of spears were long spears for impaling cavalry or making use of a phalanx wall) leading to the creation of armor to prevent that, leading to advances in spear design to pierce that armor (since heavy armor was generally worn by knights who were used as heavy cavalry), as well as improvements to clubs by turning them into maces to dent the heavier armors as an alternative to impair the person wearing the armor by restricting their ability to move/breathe.
Majority of sword design then became either usability/flexibility to slip through the gaps in armor or to use their weight to bludgeon, denting the armor layer and breaking bone. Any edge they have was to help cut through flesh.