I think my favorite use of VoiceOver is probably in Amelie, where the narrator sets the tone/style of the film and reflects the character of Amelie brilliantly. The voice over, to me, was almost an internal dialogue Amelie might be having.

The film begins with this piece of Voice Over:
http://www.imdb.com/title/tt0211915/quotes

Narrator: On September 3rd 1973, at 6:28pm and 32 seconds, a bluebottle fly capable of 14,670 wing beats a minute landed on Rue St Vincent, Montmartre. At the same moment, on a restaurant terrace nearby, the wind magically made two glasses dance unseen on a tablecloth. Meanwhile, in a 5th-floor flat, 28 Avenue Trudaine, Paris 9, returning from his best friend's funeral, Eugène Colère erased his name from his address book. At the same moment, a sperm with one X chromosome, belonging to Raphaël Poulain, made a dash for an egg in his wife Amandine. Nine months later, Amélie Poulain was born.

Youtube clip:
http://www.youtube.com/watch?v=2T9dUBO4pv0

Fight Club

Every single line of V.O.

Yeah, "Fight Club" seconded.

I thought About A Boy did a great job with it. 2 narrators used throughout, practically in every scene there's some VO, but it never wears thin. And of course the Payne/Taylor movies.

Jerry Maguire.

Election makes fine use of VO, and does it with multiple characters (3, I believe) ... it works really well.

Terrence Malick loves using V.O., which tends to work (for the most part) in his films because it's used for are interior monologues instead of pure narration.

From The Thin Red Line:

"Where is it that we were together? Who were you that I lived with? The brother. The friend. Darkness, light. Strife and love. Are they the workings of one mind? The features of the same face? Oh, my soul. Let me be in you now. Look out through my eyes. Look out at the things you made. All things shining."

My favorite V.O. is the opening narration in The Third Man; we never hear from the nameless speaker again:

"I never knew the old Vienna before the war with its Strauss music, its glamor and easy charm. Constantinople suited me better.

I really got to know it in the classic period of the black market. We'd run anything if people wanted it enough and had the money to pay. Of course a situation like that does tempt amateurs... But, well, you know, they can't stay the course like a professional."

I was also thinking of A Christmas Story but there are so many good lines of dialogue to choose from. My personal favorite:

"In the heat of battle my father wove a tapestry of obscenities that as far as we know is still hanging in space over Lake Michigan."

A CLOCKWORK ORANGE

http://www.youtube.com/watch?v=rCFeMg26_wk

"There was me, that is Alex, and my three droogs, that is Pete, Georgie, and Dim, and we sat in the Korova Milkbar trying to make up our rassoodocks what to do with the evening. The Korova milkbar sold milk-plus, milk plus vellocet or synthemesc or drencrom, which is what we were drinking. This would sharpen you up and make you ready for a bit of the old ultra-violence."

and

TAXI DRIVER

http://www.youtube.com/watch?v=y8LH8rWFmV8

(vo at 4:45)

May 10. Thank God for rain, which has helped wash away the garbage and the trash off the sidewalks. I'm working long hours now , 6 in the afternoon to 6 in the morning, sometimes even 8 in the morning, 6 days a week, sometimes 7 days a week. It's a long hustle, but it keeps me real busy. I can take in 3, 350 a week, sometimes even more when I take it off the meter. All the animals come out at night - whores, skunk pussies, buggers, queens, fairies, dopers, junkies, sick, venal. Someday a real rain will come and wash all this scum off the streets. I go all over. I take people to the Bronx, Brooklyn, I take 'em to Harlem. I don't care. Don't make no difference to me. It does to some. Some won't even take spooks. Don't make no difference to me...Each night when I return the cab to the garage I have to clean the cum off the backseat. Some nights I clean off the blood."