The original fortress, known as Gyel-khar-tse was attributed to Pelkhor-tsen, son of the anti-Buddhist king Langdharma, who probably reigned from 838 to 841 CE. The present walls were supposedly built in 1268, after the rise in power of the Sakyapa sect.
A large palace was built in 1365 by a local prince, Phakpa Pelzangpo (1318–1370), who had found favour campaigning for the Sakyapas in the south. He also brought a famous Buddhist teacher, Buton Rinchendrub of Zhalu, to live in a temple there.
Later in the 14th century Phakpa Pelzangpo's son, Kungpa Phakpa (1357–1412), expanded the Gyantse complex and moved the royal residence here from the palace and fort his father had built at the entrance to the Gyantse valley. He also built Samphel Rinchenling, the first hilltop temple, beside the castle. Although the walls are mostly ruined, they still contain some 14th-century murals in Newari style as well as in the Gyantse style which grew from it
