Maybe it's the wrong type of wind?
Today's Horse Racing
I've never ridden a horse at speed (did some light cantering) but I'm pretty sure horses aren't too affected and I doubt professional jockeys would have much difficulty balancing. When Cheltenham had to be postponed in the naughties the concern was entirely around the Guinness tent being blown away and racegoers being left without Guinness, as well as the obvious safety concern.
