There are ways around this. There are solutions. They are ugly solutions, but they are there.

Just from the top of my head ... ghettoize yourself. By ghettoization it doesn't mean lower your living standards, but to make sure that the ingroup cohesion is enforced by living in neighbourhoods where only your ideological kinds exist, and to create strong in group cohesion. Do not dilute your physical concentration but work to increase them, and in that kind of a situation, make more children. It will take time, but it will work.

This is just one solution from the top of my head.