I leave them alone unless they build on my house or sheds that I'm in and out of. I've been stung twice checking the nest box for eggs.
The chicken kind.
Yeah, I know they're beneficial, and we have very few flies out there so they must be eating a lot of them, but I'm so tired of sitting out in the yard and having them do fly-bys and touch and goes off my head. They are literally everywhere, on all the plants, crawling on the grass, buzzing in my breathing space...
I just found out that the nests are used for patches for black powder guns. Thought that was interesting.