Not especially. Holding a wedding on the actual holiday may be considered inappropriate by some, yet both are three-day weddings for most people, save medical professionals and those in the service industry. Having a wedding over Memorial or Labor Day weekend is subsequently a way to ensure most of your guests attend because the have the extra day off. Summer weddings can be problematic because of extreme weather, however both weekends are usually not that hot depending on where you live….unless you live in New Orleans, in which case God help you because it starts getting crazy-hot at the end of March.
Whatever else, it is your wedding, so if you want to schedule it for either weekend, go for it. You have your reasons for choosing your date, for example Memorial weekend could be the anniversary of the day you met or when you were proposed to. Being invited to a wedding is an honor, so any nay-saying family and friends can stick that in their pipes and smoke it. You, your partner, and mostly everyone else in attendance will get an extra day to recover after partying like crazy all weekend. Have fun and be safe!