Was it really sexual assault?

Yes, it was sexual assault (you're a minor) and possibly other charges depending on what state you are in and the applicable state statute. You're supposed to be confused, you're 14. He and your sister should know better and should be held accountable for their actions, ie, contributing to the delinquency of a minor, (allowing/participating drinking and smoking weed), and perpetually setting the stage for a situation for corruption of a minor.