I've noticed if you have post links collapsed by default, and as you are scrolling, expanding links to view the images/tweets/etc, when you click into comments on a particular post, then go back, those images collapse again and while your position stays the same, I believe the posts are collapsed and take up less "space" above and so now you are now viewing like 5-10 posts further down than you would expect.