It depends on the textile. With wearing and use, some materials relax and become softer and looser because the thread is bent, pulled, and twisted through mechanical motion, much the way that crumpling a sheet of notebook paper over and over makes it very soft. Wearing the clothes often contributes more to this than the washing by itself.
However other materials may contain oils, waxes, and other materials naturally present in the fiber such as lanolin. Washing, and especially bleaching, removes these soft, greasy, fatty substances which can make the fibers dry, scratchy, or cause them to begin to fray or lose their water proofness. The primary purpose of fabric softeners, is to help replenish these substances, but it's always a downhill battle short of soaking the clothing in lard or oil.
Lastly, some synthetic cloth can become brittle with use, similar to the way that bending a plastic spoon back and forth in the same spot will cause it to snap. The fabric begins to get creases and folds in it that don't come out with ironing. They don't exactly get scratchy, but they become stiff like folded paper, and don't bend and take the shape of the body as well.
As fibers wear and break and tear, it has an effect of reducing thread count. Thread count is the number one thing for comfort up close to the skin, with higher thread count cloth feeling softer and silky versus lower but thicker thread count like a knitted sweater which can feel rough.
This is one of those extrinsic/intrinsic things. Extrinsic to cloth fibres as fibres, is how they are spun, and woven or knitted. Intrinsic to most fibres is the 'staple' or length and strength of the individual strands which makes up the thread. How hairy it is under a microscope. Along with staple, how much innate strength it has, how hydroscopic or oleophilic it is. Natural fibres tend to be somewhat shorter, and have innate oil or water or other content. Except silk which is much more like unnatural fibre and is strictly speaking a filament. But still has intrinsic water and other things in it.
When you wash something, you alter this balance. Dry cleaning before perchlorate used to mean rolling clothes in fullers earth (diatomaceous .. well.. basically soil. Special soil.) to absorb oily dirt, and judicious brushing. Why did people do that? Because dyes were not colour fast, but also because mechanically agitating natural fibres in water does exactly what you describe: makes some stiff stuff loose (mechanically alters it's weave tension, it's fibre rigidity, it's intrinsic balance of water or oil) and makes other loose stuff stiff (for the same reason, but in reverse. Absorb water? You swell up.)
Those balls of fluff which you clean out of your lint filter? Where do you think they came from? They broke off the fibres of the fabric. It's one definition of microplastics, most people obsess about the gritty stuff in facial scrub. Apparently the fibre trash is really heinously bad too. All that artificial fibre we live in is adding to the mess.
TL;DR: Different materials behave differently when washed and dried. Even things made from the same thing (cotton fabric and denim) will behave differently because of how they are made. In that case, it's caused by the threads being pulled taut by wear, and then relaxing from having water introduced.
Fabrics cone I'm all shapes and sizes and can be made from things 'simple' as cotton, intense as silk, or complicated as plastics. Each have their own uses, but are also going to get tired in different ways. Some don't do well when they get really hot, some don't do well when they get wet, some really like to stay wet and not dry. It all depends on what they are meant to do and how their materials interact.
For some things that like to shrink, it's basically caused by the fibres being able to absorb a ton of water and being able to relax. This causes them to contract/curl up back into their resting position.. Shorter fibres, means shorter threads, and a shrunk shirt. This is much more a problem with organic materials that are pulled/stretched very tight during processesing into a usable thread.
Similar processes can cause materials to be really uncomfortable after a wash, or just when wet, while it can make others really soft. There are a ton of different ways materials interact, even things made from the same thread (cotton fabric and denim) can behave very, very differently just from how they are woven.
Think of the fabric like different types of hair, say your friend has really oily hair and you don't.
You wash your hair about every other day, if not 2 days, while your friend has to wash her hair every day. If you guys switched routines, your hair would become dry and brittle from over washing, because you've stripped the oils out of your hair that keep it soft and elastic. Same thing applies to richly colored fabrics/dyed hair. Hope this helps!