The work went ahead in Staffordshire despite workers knowing asbestos existed there, inspectors say.