I'm a toxicologist and I study endocrine disruption in a context outside of reproductive health, so I might be able to help, though I generally study thyroid disruption moreso than estrogen.
First of all, a hallmark principal of receptor biology is that different ligands will have different downstream effects on gene expression, even if they bind to and activate the same receptor (binding to the same receptor is based on structure of the ligand, and ligands will also differ in their binding affinity). This is thought to be due to recruitment of different cofactors upon receptor activation that take that receptor to the DNA response element encoded for by that hormone receptor. The different cofactors cause different regions of the response element to be bound to and transcribed.
EDIT: I should also add that these hormone receptors of relevance to this discussion are not just found in reproductive tissues. They're found in most cells of the body, so endrocrine disruptors will also be able to influence other processes, such as neurodevelopment and immunity.
The other issue is that you're citing human studies, presumably in adults, in which most humans probably don't consume enough phytoestrogens to produce a stable biological effect. Most gene expression from steroid receptor signaling comes on hours after receptor binding, and typically disappears within a few days (most receptors have ways to inactivate themselves after being active for a while). If you aren't constantly or itermittantly exposed to those chemicals like something like pthalates or BPA, then you may not have a long term effect.
A 3rd point for consideration is developmental stage of exposure. Most endocrine disruptors will only cause massive and or permanent reproductive toxicity if the exposure happened early in development (such as in utero or early childhood), before cell types have fully matured and differentiated. For example, lead is well known to hinder neurodevelopment and reduce IQ later in life as an adult if the fetus or child is exposed, but an adult exposed to lead will not experience any permanent reduction in IQ. There may be other consequences to endocrine disruption in adults (many of them are also carcinogens or can alter immunity or cause oxidative stress short term), but they won't be as severe as an equivalent developmental exposure.
I would like to clarify a misconception. You stated:
>To what extent does phytoestrogen act like an estrogen-analog in men? To what extent does it act like one in women?
Phytoestrogen should be pluralized as "phytoestrogens", because there are many. Lignans (produced during the digestion of flax seeds) and genistein (an isoflavone from soy) are both phytoestrogens. Other phytoestrogens are found in hops and lavender and tea tree oil. Phytoestrogens are not the only estrogenic substances we come across. All the estrogenic chemicals that are produced outside the body are collectively known as *xenoestrogens*. BPA and pthalates and other such substances are not from plants, but are still estrogen analogs, so they fall under this designation.
The answer to this is that there are two kinds of estrogen receptors (alpha and beta), and that various phytoestrogens may bind to the receptors with more preference for one than the other. However, binding is one thing, and activation is another. Something can bind to a receptor and not activate it, while blocking the hormone that normally binds to the receptor. Consider the fact that [caffeine binds to the adenosine receptor](https://www.wikiwand.com/en/Caffeine
), but doesn't activate it. If caffeine did activate the adenosine receptor to which it binds, it would make us sleepy, but instead, it binds to the receptor without activating it and in so doing, it blocks adenosine from binding to and activating the receptor, keeping us awake. The behavior of phytoestrogens can vary from being analogous to the behavior of caffeine (binding but not activating) to being just like estrogen. Phytoestrogens that bind but either do not activate or only weakly activate a receptor are known as *weak phytoestrogens*. These weak-phytoestrogens will not have the 'feminization' effect, but would rather have the opposite effect—reducing the influence of estrogen. (Tangentially related: See the video summaries of various peer reviewed sources on [the role of flax weak-phytoestrogens in reducing breast cancer risk and mortality](https://nutritionfacts.org/?fwp_search=flax+breast+cancer&fwp_content_type=video
). Also see the video summaries on [the role of soy in reducing breast cancer risk and mortality](https://nutritionfacts.org/video/should-women-at-high-risk-for-breast-cancer-avoid-soy/
).) The beta estrogen receptors appear to be tumor-suppressing. 
All this is to say that it is an oversimplification to think of phytoestrogens as estrogen analogs. The kind of reasoning from this over-simplified concept of how estrogen works leads to erroneous conclusions.
Men who are concerned about the feminization effect of xenoestrogens should not fear soy, but should be concerned about hops and bacon instead. The most potent phytoestrogen we typically come across is found in beer [1.a-d], from the added hops. IPAs and other heavily hopped beers are the worst offenders. Bacon, especially when cooked at typical bacon frying temperatures, produces a cooked meat carcinogen known as PhIP (for short; the chemical name is a mess— 2-Amino-1-methyl-6-phenylimidazo(4,5-b)pyridine) a heterocyclic amine which is also a xenoestrogen as potent as actual estrogen, and is heavily implicated in breast cancer, since it activates receptors on mammary tissue . If manly men avoid soy to avoid feminization but continue to drink beer and eat bacon, they're doing it wrong.
Soy isoflavones are not feminizing, at least not at any realistic rate of exposure from diet. [3.a-b] Ditto with lignans and equol. The same cannot be said about PhIP and strong phytoestrogens from hops. Phytoestrogens from hops actually appear to have a feminizing effect. [1.c, 1.d]
There was a TED talk given years ago by an angiogenesis researcher, William Li, who explained that eating foods that suppress angiogenesis may help prevent cancer cells that already exist within the body from forming tumors by preventing them from developing a blood supply. The talk was titled "[Can we eat to starve cancer?](https://www.ted.com/talks/william_li
)" In the TED talk, he mentioned soy as one of the most potent anti-angiogenic foods, and soy extract, genistein, an isoflavone weak-phytoestrogen, was shown in one of his graphs as being very potently anti-angiogenic. This may be one of the reasons [soy consumption appears to be protective against cancer](https://nutritionfacts.org/video/the-role-of-soy-foods-in-prostate-cancer-prevention-and-treatment/
 [Cheng J, Lee EJ, Madison LD, Lazennec G. *Expression of estrogen receptor beta in prostate carcinoma cells inhibits invasion and proliferation and triggers apoptosis*. FEBS Lett. 2004 May 21;566(1-3):169-72.](https://www.ncbi.nlm.nih.gov/pubmed/15147889
Video summary: [**The Role of Soy Foods in Prostate Cancer Prevention and Treatment**](https://nutritionfacts.org/video/the-role-of-soy-foods-in-prostate-cancer-prevention-and-treatment/
[1.a] [L R Chadwick, G F Pauli, N R Farnsworth. *The pharmacognosy of Humulus lupulus L. (hops) with an emphasis on estrogenic properties*. Phytomedicine. 2006 Jan;13(1-2):119-31.](http://www.ncbi.nlm.nih.gov/pubmed/16360942
[1.b] [S R Milligan, J C Kalita, A Heyerick, H Rong, L De Cooman, D De Keukeleire. *Identification of a potent phytoestrogen in hops (Humulus lupulus L.) and beer*. J Clin Endocrinol Metab. 1999 Jun;84(6):2249-52.](https://www.ncbi.nlm.nih.gov/pubmed/10372741
[1.c] [D H Van Thiel. *Feminization of chronic alcoholic men: a formulation*. Yale J Biol Med. 1979 Mar-Apr;52(2):219-25.](http://www.ncbi.nlm.nih.gov/pubmed/452631
[1.d] [A Galvão-Teles, L Gonçalves, H Carvalho, E Monteiro. *Alterations of testicular morphology in alcoholic disease*. Alcohol Clin Exp Res. 1983 Spring;7(2):144-9.](http://www.ncbi.nlm.nih.gov/pubmed/6346916
Video summary of the above findings: [Part 1: **The Most Potent Phytoestrogen is in Beer**](https://nutritionfacts.org/video/the-most-potent-phytoestrogen-is-in-beer/
[Part 2: **What Are the Effects of the Hops Phytoestrogen in Beer?**](https://nutritionfacts.org/video/what-are-the-effects-of-the-hops-phytoestrogen-in-beer/
 [S. N. Lauber, S. Ali, N. J. Gooderham. *The cooked food derived carcinogen 2-amino-1-methyl-6-phenylimidazo[4,5-b] pyridine is a potent oestrogen: A mechanistic basis for its tissue-specific carcinogenicity*. Carcinogenesis 2004 25(12):2509 - 2517](http://www.ncbi.nlm.nih.gov/pubmed/15319301
Video summary of the above findings: [**Estrogenic cooked-meat carcinogens**](https://nutritionfacts.org/video/estrogenic-cooked-meat-carcinogens/
[3.a] [M Messina, V L Messina. *Exploring the Soyfood Controversy. Nutrition Today*: March/April 2013 - Volume 48 - Issue 2 - p 68–75.](http://journals.lww.com/nutritiontodayonline/Abstract/2013/03000/Exploring_the_Soyfood_Controversy.5.aspx
[3.b] [M Messina. *Soybean isoflavone exposure does not have feminizing effects on men: a critical examination of the clinical evidence*. Fertil Steril. 2010 May 1;93(7):2095-104.](https://www.ncbi.nlm.nih.gov/pubmed/20378106
I am not an expert, but have done research in this realm to satisfy my own curiosity.
My understanding is that there are actually two types of estrogen receptors, alpha and beta, and most phytoestrogens preferentially bind/react with the beta receptor. And it works out that the literature shows this basically gives it the beneficial health promoting effects of estrogen (increased bone density, improved menopause, reduced cancer rate) without the feminizing and health harming effects (increased cancer rate).
Alpha/Beta Estrogen Receptors and Phytoestrogens
This is further supported by the fact that hops has a [phytoestrogen that preferentially binds alpha receptors (paywall)](https://www.sciencedirect.com/science/article/pii/S0960076003000505?via%3Dihub
), and it has been shown [beer may be more breast carcinogenic than other alcohols.](https://www.ncbi.nlm.nih.gov/pubmed/6475912
For men, it may be blocking binding from normal estrogen [(as shown in this prospective soy milk study)](http://cebp.aacrjournals.org/content/10/3/179.long
) which would cause a drop in blood circulating estrogen levels. Or maybe it just doesn't have any feminizing effect because that's the alpha receptor's job.
Also, in my personal experience, people who claim to worry about the feminizing effects of phytoestrogens typically turn a blind eye towards the actual mammalian estrogens in the meat and dairy they eat, which imo, should also be scrutinized just as heavily.