Did he really challenge Red? I believe what stirred Red from that path was the promise he wanted to keep to Andy - to go up to that hayfield in Buxton and find the tree... the rock... and hope. I never thought of him as a mentor so much, but I can definitely see it the way you conveyed it. My reading had always been he's more of a negative reflection, someone who represents this is what happens if you decide to take the same path when you come to the fork in the road. As such, Brooks is something of a forewarning etched into the minds of the audience, also used as something of a ticking clock. Once we see Red out of prison, working the same job, staying at the same place... we subconsciously feel unease due to Brooks' outcome and subsequently realize what the next potential recourse may be. Had the story not taken the slight detour to follow Brooks' hopelessness once outside prison, the scene with Red would have had no context, no subtext, no meaning. The fact that Brooks was such a genial, likable and nice character made him all the more dynamic as a character because we fully understand his motivations and fears.

TBBB, here's my take. We can look at Shawshank as a Dual Protagonist story: Both Andy and Red have their own metamorphosis and narrative arcs. Red is a Mentor to Andy -- "a guy who can get things," who knows his way around prison, a sounding board to provide the 'wisdom' of prison life. Andy is an Attractor to Red -- he manages to keep the tiny flicker of hope alive inside Red for two decades through his [Andy's] persistence [suds on the roof, Mozart, the library, etc]. Per the latter, Andy provides a 'light' dynamic to Brook's 'dark' energy to Red. Brooks, as Red explains, killed himself because he was "institutionalized." Red perceives that he himself has also suffered the same fate. So that when he leaves prison, goes to live in the same boarding house as Brooks, works at the same job as Brooks, and confronts the same confusing experience of freedom as Brooks, he has a choice: One path Brooks created which leads to committing a crime and going back to prison, or suicide; or the other path which leads to that tree under which Red will find a letter and some cash put there by Andy. That's why I look at Brooks as a 'dark' Mentor, presenting wisdom of a sort to Red and as such creating a moment of tension toward the very end of the movie: which way will Red go. That he chooses to reject Brooks is because of a "promise" he made to a friend: Andy.