If the bombs had not worked there would have been invasions of mainland Japan and would have resulted in the extinction of the Japanese people and culture.
It would not have been a simple genocide though, the Japanese people were being trained and indoctrinated to fight to the death with any means available... Which would have led to them getting shot as enemy combatants.

EDIT: To add to that, the firebombing campaigns we had been doing destroyed as much or more of the cities we targets as the nukes.