Hartest is situated in the area known as High Suffolk, which is characterised by heavy loam and clay soils, originally covered by woodland and forest.

Indeed the name `Hartest` derives from the wooded nature of the landscape. Earlier spellings such as Hertest (c1050), Herterst (c1086), Herthyrst (c1095) & Herthest (1200) contain two elements meaning `hart` or stag and `wood`. The name Hartest therefore originally meant Harts Wood.

The village proper is clustered around the central green. This was originally larger and it appears the eastern edge was formerly on the other side of the river, and that the tightly packed houses along the present eastern edge are later encroachments. A timber-framed Wealden house, between `Hunters` and `Green Farm`, dates from c1470, which suggests that the encroachment is an early change to the village.

At the south side of the Green, at the lower end, near the fording point of the river, stands the church, which was already there by 1086; the Hall or Manor House, now the Crown Inn, higher up the slope, and the rectory to the south of the church.

On the western side the Green originally tapered into Somerton Road, as it still does on the North side, into Workhouse Hill.

Twenty-four farms were recorded in the 1841 census. The names of three of them can be traced back at least to 1327, when Henry Waryn, Robert the Long and Robert of Monte (mount or hill) all appear among the list of tax payers.