According to Friedrich Nietschze's theory in "The Birth of Tragedy", theatre emerged from Greek religious rituals. And even today, over 2,000 years later, the power of ritual casts a long shadow over the stories we read, hear and see.
Now, Nietschze is complicated stuff and I want to keep this blog as brief as possible (and don't get me started on much I loathe Wagner), but let's put it like this...
When you watch, say, a professional wrestling match (which I do sometimes), you'll notice how predictable these events are.
The pattern in a pro-wrestling match never changes:
-The "good" wrestler enters, greets the audience, projects his persona.
-The "evil" wrestler enters, sneers, offends the audience, projects arrogance.
-They fight; the "evil" wrestler cheats - and may win the first round;
-...but eventually the "hero" wrestler "reaches deeper down than ever before", beats the evil wrestler senseless, and wins.
Most popular fiction follows a similar pattern. You think James Bond movies are repetitive? Ian Fleming's James Bond novels were just as repetitive - he knew the winning formula and didn't change it one bit - and readers loved them that way.
Most readers have strict expectations and don't want to feel cheated. The ritual is supposed to be followed.
Good triumphs over evil.
Society is preserved.
A sequel is promised.
The Heroine marries the Hero at the end.
The detective Solves the Case.
The Dark Lord drops off a cliff.
The White Middle-Class Guy prevails, and the Darkies are Kept Down.
Would it kill the fans if, just once, the heroine said at the end of a paperback-romance like Her Outsourcing Prince: "You may be a rich alpha male, but I don't want to marry you. You have some personal issues that all the love in the world couldn't change. Bye... - The End" ...?
As it happens, there are successful exceptions to the rule: Gone With The Wind ends in divorce.
Writers should be aware of the rituals. If you follow them too rigidly, the story becomes mundane. There are always ways to put a fresh spin on the expected ending.
In the movie industry, films are test-screened with alternate endings. Maybe writers could also try to write several alternate endings, and "test-screen" them?