Like water spilled on the ground, which cannot be recovered, so we must die. But God does not take away life; instead, he devises ways so that a banished person may not remain estranged from him. 2 Sam 14:14
I believe that unhappy workers count as unhappy workers even when they're working a building. You do have unhappy workers, they just are in use right now. So yes, you should disband/destroy in order to reduce your pop. Had this exact thing happen to me the other day online, and yes it sucks.
The sad faces on the board are the discontent workers. With unused workers, you would cover them up to prevent uprisings, but they are still discontent.