Villagers have said a proposal to create a gypsy and traveller site in their community is causing them to become “depressed”.