The castle was the residence of the Counts of Conversano for almost seven centuries, ever since Norman times. Yet his story is much older, probably already at the time of Gothic War (535–554) on the same place there was a defense building, which embraced a part of the megalithic walls of the ancient city of Norba.