I live in Kansas and I see people flying the Confederate flag. Kansas was never part of the Confederacy nor would the people who fought and died to keep Kansas from allowing slavery be okay with you flying that flag. It also occurs to me that the people who fly this flag would never, ever, ever fly flags of other countries where their ancestors immigrated from. My ancestors came to Connecticut from Britain, migrated to North Carolina and Tennessee before ultimately winding up in the Little Dixie area of Missouri. Due to those last three places, there is a huge possibility that distant relatives of mine fought for the Confederacy and owned slaves so I think I could get away with flying the Confederate flag.
But I won't because why would I fly another country's flag? If you're not willing to fly the British flag which is more than likely where your ancestors came from, then why fly the Confederate flag? Also, if you have to explain why your flag isn't considered hate speech then maybe you need to get a new icon for your heritage.