Race and Gender

I am going to violate my own rule here. I try to change a story just enough to make sure no one for sure knows who it is about, but I can't think of a way to change this one enough that is still holds meaning for me. So here goes:
I knew an incredibly talented and creative gardener once. This gardener was a man. He was a black man. I met him because I did some work for a delightful woman in his neighborhood and admired her wildflowers, expressing my desire to specialize in native landscaping some day. She told me I simply must stop by his 'natural' garden and he would love to meet me and just tell him she sent me. She gave me directions.
I drove straight there, before I lost my nerve. He was not there, but his son was, who gave me his work phone number. I called from my cell phone before I lost my nerve. We talked gardening and plants and people and more for about an hour before we realized how much time had passed. There was a bit of an embarrassed sharing of mutual "My, I seem to have gone on a bit there's" before he invited me to stop in on a day he'd be there. I did follow through and he gave me a magnificently interesting tour.
Months later, I saw him at a craft sale, working the booth of a garden club of a nearby town. I asked if our town had a garden club. He said it had at one point, but it didn't have room for the likes of a gardener like him. I asked what he meant by that. He asked if I had ever been to a garden club meeting. I answered that I had only been to meetings of the Wild Ones, a group formed to promote natural landscaping. He told me that at the time he was getting into gardening, in the 70's, most of the ones he found, including the one in our town, were mainly women's social clubs, having room for neither a man, nor a black man. I was shocked. Surely not in our own town, did race and gender matter in a club formed around a hobby like that. He gave me one of his 'looks' and I let it go. I had other gardening friends in town and now and then we got together to shop for plants or to go on an excursion. I got the idea that there were enough of us to form a new garden club, one that would welcome all races and both genders and would certainly have room for my friend. I put some notices in the paper, picked a date, found a meeting room, and we started. I assumed my friend would have seen the notices and that he would be there. He was not. I knew some of the members knew him, so I was certain someone would mention it to him and he'd show up. Finally, after several months, it occurred to me that my friend was not going to come to the garden club I had started for him. I worked up my nerve and called him. I said "Did you know we have a garden club in town now?" He said "Yes, I read about that. Some people mentioned it to me too." I said "Why haven't you come? You should come. You know most of the people and they love you and your garden and we'd have a blast." He said "That club wouldn't have me once, so I am not joining it now. On principle." I argued "It's a NEW club and new people. It is NOT the club that would not have you." He refused. I was hurt and angry. But so was he. His was a longer, deeper, 'principled' kind of hurt. Could I fault him for holding a grudge, even if it was misplaced?
The club ran along smoothly for a few years, they having made me president and me having recruited really good officers like a program chair who found speakers and a press secretary who got the information from the program chair and got notices in the newspaper and a secretary who made sure both did their jobs on time. Then one summer, we went to members' gardens for our programs and after touring the garden of a resourceful and inventive and interesting member of the club, we went inside for refreshments. I was astonished to find there was a Confederate flag, a very large one, on the sloped ceiling of his living room. He and his brother collected Confederate memorabilia and were Confederate history buffs. Someone asked me when he was out of the room, "Doesn't that mean he is racist?" I seriously doubted it, but I also seriously doubted that he understood that most people saw it that way. And I was pretty sure the friend for whom I had started the club would see it that way.
A few months later, the man whose garden we had toured volunteered to take over from me as president, and it would not have looked right for me to refuse. He never got around to calling another meeting. The club ceased to be. I let it go, planning on re-rallying the club after a while. I got busy. It never happened.
The garden club I started, motivated by wanting to make a place for my black friend, died in the hands of another member who displays a Confederate flag proudly. All parts of this are things I will never quite understand, and my failure to make my friend feel comfortable in our club and the demise of the club rank high on my list of personal failures.