What if instead of the Dursleys being creul to Harry Potter they told him about how bad Hogwarts was (a lie, of course) and brainwashed him into hating it? Then when Hagrid showed up to take Harry to hogwarts he would not have wanted to go, and I don't think anyone could have forced him to go if he didn't want to but then again Albus Dumbledore probably would have straightened things out. If need be you could even use a Confundous charm to make Harry give Hogwarts a try of course I think he'll have liked it and stayed but it would probably have taken longer in the story and made the reader not like Hogwarts either.
What If Dumbledore had accepted the job as Minister of Magic? Someone else would of had to take over Hogwarts, possibly Minerva McGonnagal, but when Voldermort rose Dumbledore would have made much better descions than good ol' Cornelius Fudge did.