Shortly after his accession William Rufus gave Northallerton, with the lands adjacent, to the see of Durham, and, under the patronage of the bishops of that diocese, it grew in importance, and became an episcopal residence. In 1130 a castle was built on the west side of the town adjacent to North Beck by Bishop Rufus and was expanded in 1142 after William Cumin seized the Bishopric of Durham in 1141. The castle was further expanded in 1173 by Hugh Pudsey and garrisoned by a group of Flemish soldiers, an act which enraged King Henry II who ordered that it be razed to the ground in 1177. A more substantial fortified palace surrounded by a moat was built on the same site in 1199 replacing the traditional motte-and-bailey castle. The palace became an important administrative centre for the bishops' lands in Yorkshire and served as a major residence for the bishops and their staff. The palace lay on the main road from York to Durham and was a regular stopping place for royalty and other dignitaries. The palace fell into ruin by 1658 and the site is now a cemetery.
.