We did the same growing up, and I never thought anything of it. There were never any tools left behind that I remember and it was just a shell of a house with no door. They were fresh new neighborhoods being built right down the street from us and we wanted to see what they looked like before anyone started moving in. I still don’t think there’s necessarily anything wrong with it. It may technically be trespassing, but there was no door, they were still owned by the building company at that point and there were no “keep out” signs.