Initially the island was uninhabited, there was only the occasional fisherman from the neighbouring islands, looking for shelter in a storm or just resting before continuing on his journey.
It would appear from old maps and descriptions that this island was known by European cartographers and mariners as "Pulo Bardia". The old maps show a chain of three islands aligned north-south and lying off the east coast of the Malay Peninsula. The most northerly and smallest of these islands is marked P. Bardia - the name sustained until the early 1900s. The best map example is a map by John Thornton from "The English Pilot, the Third Book", dated 1701 but the specific map of the Gulf of Siam is dated around 1677. Also see maps of the East Indies by William Dampier c1697. By modern standards of accuracy, the islands are pooly placed on early maps. 17th century marine navigation and cartography used the 'backstaff' which, in this area, was accurate to one degree of longitude or around 60 nautical miles.
Page 383 of 'The Edinburgh Gazetteer, or Geographical Dictionary' (1822) also mentions the island and provides a geographical position. In his 1852 book titled "Narrative of a residence at the capital of the Kingdom of Siam" by Frederick Arthur Neale he describes the people and wildlife of Bardia. According to the account there were farms and even cows in a village on the bay lying to the west side of the island. The book includes a fanciful illustration of 'Bardia' showing huts and palm trees.
On June 18, 1899 King Chulalongkorn visited Ko Tao and left as evidence his monogram on a huge boulder at Jor Por Ror bay next to Sairee Beach. This place is still worshiped today.
In 1933 the island started to be used as a political prison. In 1947 Khuang Abhaiwongse, prime minister at that time, pleaded and received a royal pardon for all prisoners on the island. Everybody was taken to the shore of Surat Thani and Ko Tao was abandoned again.