Yes, we've seen them lock Harry up, scream at him and  overall they just don't look like pleasant neighbors.
but just think about it... Would such horrible people, that seem to know about the wizarding world (atleast one does) and hate it that much, take in a baby that was left on their doorstep with a note? That doesn't sound like them to me. I would think they would have dropped the baby off at the nearest orphanage to get rid off him.

It simply doesn't match the way they treat Harry when we fall into the story ten years later. They are horrible to him. Locking him in his bedroom, which is a small cupboard under the stairs! They barely fed him and didn't hide the fact they loath him. But what has happened to make them behave like that? What made them hate Harry so?

To answer that question I need you to let go of the Dursleys for a little while and jump ahead with me. During his time at Hogwarts we've seen Harrys journey to destroy he-who-must-not-be-named and the particles of his soul. 

That's where we find the key. The reason why I think the Dursleys aren't really horrible people. They might have been less pleasant company than others, but something made it worse.

The first clue was dropped in Harry Potter and the chamber of secrets. Ginny was badly influenced by Tom Riddles diary. She might not remember and didn't do it out of free will, but she was acting out of character.

 Later, we saw Ron being affected by a horcrux in the Deadly Hallows. Turning on his friends, being well...depressed and angry. Which got me thinking. If a short period of time can have that much effect, than maybe, just maybe, the Dursleys would be effected too. How you say? Well surely most of you remember that Harry himself was a horcrux. So the Dursleys have lived 10 years near a horcrux. I think even the nicest person could become bitter living near such a bad influence. 

Some might say it's odd the Dursleys are effected that badly. But others who were near Harry for a long time seemed ok. Well dearies, I believe a horcrux can find your deepest emotions. Negative emotions. Ron was in love with Hermione and was affraid he wasn't good enough for her. So that's what the horcrux played at. His friends are mostly joyful and feel a form of love for Harry and we all know love is the strongest magic of all!

The Dursleys had negative feelings for Harrys parents. Petunia was jealous of Lilly and  Vernon hated James, they had been on bad terms. To be fair Harry looks like James. But I for one can not and will not believe someone could hate your parents so much they would treat you as bad as they did. I believe their hate was fuelled by Voldemorts soul living inside of Harry.

So what do you think? Could the horcrux have been an influence? Or do you believe the Dursleys were simply horrible people?