We're always taught that even the Nemesis character should not be ultimate evil - that they should have something charming or admirable about them, or we see them in a humanizing moment, to make them more well-rounded and interesting. How come Michael Myers (and many of the other horror villains) defy this and still work so well? Is it something inherent in the genre?

Unless when the story is about a character who is ultimate evil. More from the Wikipedia page on Michael Myers:

A common characterization is that Michael Myers is evil. John Carpenter has described the character as "almost a supernatural force - a force of nature. An evil force that's loose," a force that is "unkillable" Professor Nicholas Rogers elaborates, "Myers is depicted as a mythic, elusive bogeyman, one of superhuman strength who cannot be killed by bullets, stab wounds, or fire." Carpenter's inspiration for the "evil" that Michael would embody came when he was in college. While on a class trip at a mental institution in Kentucky, Carpenter visited "the most serious, mentally ill patients". Among those patients was a young boy around twelve to thirteen years-old. The boy gave this "schizophrenic stare", "a real evil stare", which Carpenter found "unsettling", "creepy", and "completely insane". Carpenter's experience would inspire the characterization Loomis would give of Michael to Sheriff Brackett in the original film. Debra Hill has stated the scene where Michael kills the Wallace's German Shepherd was done to illustrate how he is "really evil and deadly".