According to Vets, Your Cat's Ears Can Go Back For Any Number of Reasons Besides Fear

As it turns out, pulled back ears are just one of the many ways that cats get across … fear and anxiety paired with repulsion, frustration, and lastly, pain.
Source: back pain