William II, the 3rd Lord of Bramber, married Bertha, daughter of Miles of Gloucester, who had four brothers who were expected to inherit. But they all died childless and the inheritance was split between the daughters. William and Bertha got Brecon and Abergavenny, which gave them a vast block of territory in the Middle March of Wales. William was close to King Stephen and was in the escort for the defeated Empress Maud. He served King Henry II and was part of many of the king's expeditions in France. He was appointed sheriff of Hereford in 1173.i
William's son, William III, was very anti-Welsh, and it caused a lot of trouble. It may have led to war, but Rhys ap Gruffydd, the ruler of Deheubarth, submitted to Henry II as a vassal and was appointed Justiciar of South Wales. This kept the Welsh in check, until Henry II's death in 1189.
Abergavenny Castle (Wikimedia)
This resulted in outrage and hostility from the Welsh, whom the kings were always trying to pacify. They named William 'The Ogre of Abergavenny'. Gerald of Wales emphasised his subsequent great piety and generosity to the priories of Abergavenny and Brecon, presumably in an attempt to atone for his crime. Seven years later Seisyllt's surviving sons took their revenge by burning Abergavenny Castle down. The keep survived and William III built a new castleiii.