The first workhouse was in the churchyard. In 1836 a new union workhouse was built at a site to the south of Swaffham. It cost £7000 and could accommodate up to 405 inmates.
The architect, William J Donthorne, who lived in Swaffham, also designed other workhouses in Norfolk. His design was closely based on the hexagonal design published by the Poor Law Commissioners in 1835.
There were three accommodation wings for the various classes of inmates with segregated exercise yards between the wings.