Let's take a look at the things you should be concerned about that are directly related to your site's security.
Your site could take a big reputation hit if users see a defaced web page instead of your normal site. Security conscious visitors of your site may decide that is no longer safe to visit. And if you actually lose control of your users Personally Identifiable Information(PII), then your credibility with your users will will really suffer.
2. Block Lists:
A relatively newer threat to web servers is the notion of server side malware. This is where an attacker gains control of your server, but does not leave any notice by damage, exfiltration of data, or defacement. Their new goal, once they have control of your server, is to hide there and ambush your unsuspecting visitors. This may be accomplished by sending visitors malware or by redirecting them to a malicious site.
Many services, including google's safe browsing, keep track of sites that are distributing malware and viruses and warn users before they connect or just block the connection. If you wind up on one of these lists, it can take some time to have your site removed effectively making your site down to some users while you are on the naughty list.
3. Data Loss:
If you have ANY data on your server you don't want the whole world to have or that you can't live without and only have one copy, then you should be concerned about data loss. (You are doing regular backups right?)
4. Time and Money: If your site is breached, you can guarantee, the cleanup will cost you some time and money. How much will depend on the depth of penetration by the attackers and what, if any, data was lost or stolen.
Unfortunately, it is a sad truth that if you have a site on the Internet, one or more of these concerns may be something that keeps you awake at night.