The problem is polarization. Rugged indivdiualism is merely a tool - a message - used by right-wing groups to pull people into their fold, to oppose "the other side".
The "other side" - the left - not only rejects individualism, but fully embraces collectivism, which is just as bad as extreme individualism.
It is these two extremes that are toxic. Unfortunately, our political system's very design makes such polarization inevitable.
We need rank choice voting so that third parties can have a chance - we might then see the rise of moderate parties to challenge the two giants and only seek to divide us for their own ends.