The oldest archaeological findings from the area date back to the 6th millennium B.C. At the end of the 12th century A.D., the Lehola hill fort of ancient Estonians was built at a location (now known as Lõhavere) approximately two kilometers north-east of the present town border. The hill fort was the centre of the northernmost district (Estonian: kihelkond) in the historical Estonian county of Sakala and also one of the centers of Estonians' fight against the conquest of German Sword Brethren in the 13th century. Not much except the hill remains of it but the place (known in Estonian also as Lembitu linnamägi) is still visited as a tourist attraction. 2/3 of the hill-fort have been excavated archaeologically.
The locality of Suure-Jaani was initially called Wallola (or "Valle"), first recorded in 1423(?). The town began to develop around the (now Lutheran) Church of Great Saint John the Evangelist (Groß Sankt Johannis in German), built before the year 1300. There is a Lutheran cemetery near the church. The church was severely damaged during the Livonian War (1558–1582) and the Great Northern War (1700–1721), particularly the latter (the church was burnt down in 1703 and the restoration was completed only in 1767). The present name, "Suure-Jaani" (literally meaning "Great John's"), which came into widespread use for the locality only in the 1880s, when it developed into a village in its own right, is an Estonian derivative from the German name of the church. Until then, the name "Suure-Jaani" was used only for the church as well as its immediate surroundings (the church estate), while other areas of the present town were separate villages: Valula (or Wallola; the southern part of the present town of Suure-Jaani), Nuutre (the western part) and Päraküla (or Perraküla; the north-western part; one part of the traditional Päraküla still being an independent village).
During centuries, Suure-Jaani was one of the centers of Suure-Jaani parish
