A long time ago, unicorns were not the only ponies to do magic. Members of the other races also had such power, focused in a different way. Using the magic ambient within the environment, they worked spells with Runes to alter the world around them.

They were called Weavers.

Fearing their power, immoral nobles took advantage of Celestia's bereavment over Luna's banishment to all but exterminate these ponies.

Now, the last survivors, living in hiding for their own survival, seek to keep their secret in their new home in Ponyville.

...unfortunately, the first pony they meet is Pinkie Pie. So much for that.

Cover art isn't mine