If you write what you WANT to write, you shouldn't give a damn what people think about it.
At least that's the ideal. Ideals won't pay the rent, though...
If you like to write any kind of "genre" fiction (science fiction, fantasy, horror, romance, crime/detective), you will find people who like those "genres"... and you will also encounter people who dislike your fiction simply because it carries a "genre" label.
Of course, tastes differ. I don't read romance novels (it's a guy thing -- you wouldn't understand)... and resent research seems to prove big differences in men's and women's literary tastes. You can't fight biology. I have no issue with that.
But despite differences in taste, I think "genre" labels are mostly a nuisance. They obscure just as much they illuminate. They perpetuate prejudices, and are often used as an excuse for shoddy workmanship.
Here's the skinny, dudes and dudettes: genre labels exist only to help readers and booksellers sort books by label. There is no inherent quality in the label itself!
It's a familiar complaint among science-fiction fans: as soon as a "mainstream" writer has success with a "genre" novel, the critics will pretend it's not "genre" fiction at all:
"If it's good, it can't be SF. If it's SF, it can't be any good."
But the fans are ALSO wrong. Let's look at genre prejudice from another angle.
There are "genre" fans who uncritically devour any crap that carries their favorite "label" -- be it SF, fantasy, crime, romance or "literary" fiction.
I find that totally unacceptable. Didn't I just tell you that there is no inherent quality in the label itself?
So I say: You cannot truly appreciate quality in fiction, unless you learn to rise above the idea of "genre labels". Someone tells you a book "belongs" to a genre. Why not simply ignore the label and ask: "Well, is the book any good?"
"Genre" fans, critics AND writers should stop being so narrow-minded: Genre is not important. Quality is. If you write pap, it's pap no matter what label you put on it.
Just because I write stories about time-travel, space aliens and supernatural terrors doesn't mean I want to get chummy with every person who writes lousy stories about time-travel, space aliens and supernatural terrors.
The same goes for "literary" fiction (just another "genre" label). I don't care if you won the Booker Prize: VERNON GOD LITTLE bored me and I couldn't drag myself through the entire first chapter.
But let's be nice. Tastes change, don't they? People change. They grow up. Our minds become more refined, more mature. Our favorite "themes" may not change much, but with age we require more depth, more realism... and above all, better grammar.