Some areas have a double wall, and that's the real ladder killer. You might get over the first wall with a ladder, and then shimmy down a rope, but now you need something to go over the second wall.

Either it will be too difficult to pull that ladder through the wall, or it will take a while, and by the time they reach the second wall with it, border patrol is there and can nab them.