The only time I've ever had a new site jump around the results was when I put affiliate links on it a couple of weeks after creating it. It was all over the place. I removed the site-wide aff links and the homepage aff links and it settled within a couple of weeks. .
Which may have happened anway...
From this thread so far it has been mentioned that certain links are not passing the value they once did - whether they be spammy anchor text links, paid directory links, reciprocals, manipulative internals and so on...
I don't think it makes any odds what your site does or does not do - if you are not getting natural links then you're going to suffer the death of a thousand penquins.