LVMH just sent out an email to Wharton for recruitment. I'm pretty sure they also hire elsewhere. I'll be surprised if the world's largest luxury group doesn't also have a large need for talent, and so for them to only recruit at CBS and HBS would strike me as a bit odd.
That being said, there are obviously pros and cons to both decisions. All of us can only tell you what we would have chosen. We don't know your own background, your own skills, strengths, weaknesses, and so on. Even if we did have all this, we can only apply that to our own decision making process- not yours.
Personally, LVMH is good because industry experience is important. However if I truly wanted to go into general management straight away and thought that was more important than industry, then I would go to business school. I don't know enough about the luxury industry- do they tend to hire insiders only? Do they instead prefer to look outside, to functional specialists, for instance hiring mainly consulting company alums for their strategy practice? In addition, there are other resources at business school. At Wharton we have a Fashion and Luxury Club that sends people to L'oreal, Estee Lauder, LVMH, and many other companies. I'm also now taking a class with a former fashion designer who now owns her own marketing firm and contracts with JPG, cK, Armani, and all the other major fashion houses.
There are multiple paths to success. Which one suits you most depends on your own needs right now, and on the fashion industry's recruitment policies. Both of these I have no access to. However, the fact that you're getting a fashion offer without prior experience might suggest to me that they are open to recruiting from outside the industry. Whether they're open to moving their internal auditors to a more general role is however something I have no info about.