To illustrate my point, I took screen captures of the body templates for everything but plus size, and overlapped them, with each layer a different colour to make the different features more prominent. Here’s what I got:

For the four male bodies there’s some differences in shading to indicate musculature or the lack of it, and Soft Male has a tiny bit of shadow at the bottom of the abdomen to indicate a slight paunch. They are otherwise exactly the same shape, which is why, as you observed, it makes no difference when seen in clothes. All the body differences are lost.
The three female bodies have some slight difference in shading in the abdomen, and that’s it. The outline is identical. The bust line is identical, same hips, same waist, same everything. They are almost exactly the same body.
Here is a group of athletes of various sports, showing the different types of bodies each sport demands. It was published around the time of the last Olympics, I think.

And that’s just athletic bodies. As we know, in life, there is way more diversity in body shapes among the general populace.
…and then there’s children and they’re proportioned completely differently to any adults.
Even if you take out the secondary sex characteristics, they’re proportioned differently, which is why I always find the “shrunken adults” used to portray children in Episode stories a trip through the Uncanny Valley. Something’s just not right.
