Most ants are sort of neutral for the garden, except if they've set up a giant ant metropolis amongst the roots of an important plant in which case SOMETIMES the plant suffers from drought etc because the ants have sort of mined all the soil away from its roots. (Usually the plant will be half-buried in the ant nest when this happens, but the ants have made the nest so porous that the roots aren't really IN much of anything at all, you know?)
There are exceptions, depending on type of ant, but in general I would not personally worry about the ants unless the plants are clearly suffering from the ants in particular. And if they ARE, good luck doing anything about the ants without also killing the plants

-- ant poisons often don't work real well in the garden (when they do, it involves long-term use) and the other usual solutions like digging the ants up or pouring boiling water on the nest are, obviously, not without disadvantages if the ants are nesting in the roots of a plant you wanted to save

DE will do *some* good, sometimes - but is also arguably not so good for the rest of your garden, since it kills ANY insect or other arthropod-type thing.
My best suggestion would be to put up with the ants until the harvest gets to a point where you can pull the plants out with a clear conscience and then go after the ants with the big guns - like canning kettles of boiling water, and/or a shovel.
Good luck, have fun,
Pat