City living comes with a lot of challenges. Between safety concerns, the sheer expense, and the plethora of old housing stock, there can be a lot of frustrations in the urban lifestyle. That also translates into a lot of reasons an apartment in a high-rise building might be appealing: The security of a doorman, the spectacular city views from an upper floor, and the potential for modern amenities to name just a few.

But that luxurious image of high-rise living hides some serious downsides—downsides that most people don’t figure out until they’re actually living there. Having a home hundreds of feet in the air might seem like an exciting, glamorous way to live, but there are some potential downsides you should consider before you get in too deep.

A ton of neighbors means a lot of small talk

This one seems obvious—of course a giant building has more people in it—but it’s worth thinking about. If you’ve never lived the high-rise lifestyle before, you might be in for a shock. A single-family home typically has only a few next-door neighbors—one on either side, perhaps one behind the lot (though likely buffered by an outdoor space) and then the folks across the street. Even on a busy street or in a condo situation, you probably have a small number of neighbors you have to deal with.

In a high-rise, the median number of neighbors is around 274. That’s a lot of people you will have to interact with on both a polite good-morning level and on a more substantive level when dealing with building issues.

You’re reliant on elevators, which adds time and complications

When you live several dozen floors above the street, you’re totally reliant on elevators to get to and from your apartment. This certainly has a lot of advantages over living in a walk-up, but elevator living has its own lush subcategory of downsides:

Breakdowns. As noted in this old Reddit thread, elevators can be delicate machines that break down regularly—even in nice high-rise buildings. That can leave you waiting a long time for the next elevator, or carrying your groceries up multiple flights of stairs as you wonder how long it would take anyone to notice you were missing if you had a sudden heart attack.

Crowds. Remember all those neighbors I mentioned? Since the elevators are the one way up and down in your building, you should steel yourself for endless encounters and a lot of polite chit-chat as you wait for and ride in elevators with other people.

Delays. Elevators aren’t all the same, and some can be infuriatingly slow, resulting in a lot of lost time. If it takes five minutes to get from your apartment to the lobby (and vice versa), you’ll need to build significant elevator time into every plan you make. This makes it especially challenging to take your dog out for a quick midday pee break—rather than open the door and walk down a flight of stairs, you may have to wait a few minutes for an elevator to arrive. And if your dog (or the dog on the elevator) isn’t particularly friendly, you may have to let it pass you by and catch the next one.

If the water pressure goes down, you might be waiting a while

Typically, city water pressure is sufficient to push water up a few floors (in New York City, for example, standard city water can make it to the sixth floor without a problem). After that, it requires a pump and storage system to get water to storage tanks that then disperse it to the apartments.

That’s fine when everything is working, but when pumps fail or water pipes leak in the building, it can be disastrous, because so much of that infrastructure is outside your control. Floods from broken water pipes can impact multiple apartments, creating an insurance mess. Unlike a smaller building, where leaks can be quickly isolated and dealt with, fixing leaks in high-rises can be a major undertaking that can further impact your water pressure—and sanity.

The creepy, wind-fueled shakes

High-rise buildings are designed to sway slightly. If they were completely rigid, high winds and storms would affect them much more powerfully.

So a little bit of movement is normal—but it can also be kind of terrifying. In this video, you can clearly hear what can only be described as alarming creaking noises in a high-rise apartment while a storm batters the building. High-rises can be noisy on the best of days—with “creaking, banging, and clicking noises” being pretty common—but in a storm you’ll have to get used to that sense that the whole place is on the verge of collapse (it probably isn’t, it will just sound that way). If you’re not sure you could sleep peacefully through that, a high-rise might not be for you.