Let's start with some facts.
In the United States, polygamy is illegal. It is illegal in every single state. It's a misdemeanor in some, and a felony in others, but illegal everywhere. Adultery is chargeable with a felony in Wisconsin, and can be served with a life sentence in Michigan. One can also be court-martialed if in the military while committing such an act. Of these two, polygamy is heavily persecuted wherever it occurs, while adultery is only occasionally sentenced under the law, though both are socially unacceptable in this day and age.
Now, what does the government have to do with the relationships of people?
Oh, alright. There can be issues with having too many kids in a polygamist family for the family to support, but that's really the only thing I can think of not covered by the law. I don't think that in and of itself is enough to make this illegal. I mean, what other harms are there? (Apart from every celebrity instantly having 100 wives/husbands, and the rest of American males/females becoming wiveless/husbandless.)
As for adultery, does the government have to make people be faithful to their spouses? (Alright, you've been emotionally hurt, but if you're in a relationship to someone, it's hardly binding. You're a dick if you cheat, but since when does the government regulate douchebags?) I could think of at least a dozen other things money and time allocated to such cases could be better spent on.
Like gambling, if all the people in a polygamist family consent to such a relationship, what right does society have to intervene? (No! It's not fair that you have two spouses! It's not fair. IT'S NOT FAIR!) Likewise, if someone decides to cheat on a significant other, what right does society have to tell them it's illegal to do so? (NO! If you get into a relationship with me, you're NEVER allowed to leave!)
Once again, this is society/government getting into the lives of other people. Polygamy and adultery don't physically harm anyone directly, and establishing any indirect harms are nigh impossible. It's not as if every adulterer punches a random passerby before cheating. One time emotional harm is nothing illegal (though emotional trauma, as is sustained emotional harm, also known as bullying).
I believe that both of these are relics of Christian influence in American law. That is not to say Christian influence does not still exist, nor is it to say that it is acceptable. (Neither is the influence of any major religion, lack thereof, or most self-interested agendas.) I may go into depth about that later on, but that's not the topic at hand. If one looks at (Gasp!) the Christian bible, adultery and polygamy are both quite clearly held in contempt.
To end this spiel, one could apply these same arguments to why gay (otherwise known as GBLTPQetc., or whatever the acronym is now) marriages should be legal across the USA. "Oh, but they offend me!" Tough.
tl;dr Polygamy and adultery are not something in the government's domain. They may be regarded as immoral or unethical by most of society, but they have no direct victims, and should thus not be the concern of government.