It is not even a matter of your neighbors doing anything, all though that can be an issue. A savy teenager next door could certain cause a lot of problems for you by using your network to do any number of things.

You also have to be concerned about people who roam neighborhoods with laptops in search of open networks. Even secure networks with encryption are vulnerable, but the more precautions and steps you take, the less likelyhood you will have of keeping someone out of your network.