I've been meaning to reply to this post for a few weeks now, but have been too busy in the garden!
Purple, pink, and blue are my favorite color theme in my garden. Almost every time I try to branch out, I regret it because other colors seem to me to clash. I especially find oranges and true reds troublesome in my garden. Though I know some people find those combinations pleasing, and I've seen beautiful examples. Just not for me so much.
Sarah Raven has even helped me appreciate orange in combination with pinks and purples and the acid green-yellow of the Alchemilla mollis!

Though, again, not in my garden.
However, I do have a section for the hot colors, and I'm happy with them there. Just not near the cool ones.
I live near Winterthur Gardens, and Henry Francis duPont who created them, was passionate about a yellow/purple combo. I balk a bit at it in theory because, like you, I generally don't care for what I consider to be harsh or glaring colors, especially some yellows. And the softer yellow I prefer seems to be rare in flowers.
However, I do find in the early Spring I'm far more open-minded and find just about any color a welcome relief from the winter grays and browns.
So, even the forsythia doesn't offend too deeply. In fact, at Winterthur, the combination of the glaring yellow forsythia and pinkish-purple redbuds is stunning. The earlier-blooming combo of corylopsis and a pinkish-purple rhododendron is even more to my liking, with a softer, buttery, and to my eye not lemony, yellow.
I've put up a post at Houzz to illustrate, if you'd care to have a look, it's here. (You don't have to be a member to see the posts.)