Last summer, in the wake of the Gaza war, some youths threw a Molotov cocktail at a synagogue. The judge has now ruled that there was no antisemitic background to the act. Several people have protested that any act against a synagogue should be classified as antisemitism (e.g., Leonid Goldberg: Fehlurteil in Wuppertal [German]). In this case I agree that the background was probably antisemitic. If the attackers really only wanted to protest against Israel’s war, why would they chose a Jewish target? It is antisemitism to hold all Jews responsible for the politics of Israel.
But I also think that it is dangerous to generalize too much. Every case should be judged based on the actual facts. Not every act against a Jew or Jewish property is automatically antisemitic. A certain portion is just random criminality. If you shout "antisemitism" too often, you are not going to be believed when it really matters.
Two examples to make my point. First, an example from my community. One Saturday morning the congregation arrived to see a small wall in front of the synagogue destroyed. Antisemitic vandalism was of course the first thought, but as it turned out it was just drunken party goers who happened to walk by and destroy random objects all along the street. As a second example, take the process against Nechemya Weberman, an orthodox Jew and convicted pedophile. Many people shouted antisemitism. Is it antisemitism to punish wrongdoings if the offender is a Jew? No, it is not, every criminal should be punished, irrespective of his/her religion.
So is it really such a strange question to ask whether vandalizing a synagogue is antisemitic? No. Antisemitic motives should rank high on the list of possible reasons (as should anti-islamic feelings for mosque, anti-abortionists for an abortion clinic, etc), but the actual reasons may be different.