
... Blueberries are a great choice. I have had some trouble adjusting the pH to get good production but my bushes look good.
I like the Nanking Bush Cherry ... very nice blooms in spring and the bush has a nice appearance all season.
Mildly tart small cherries... birds and chipmunks love them also
The
Highbush Cranberry - Virburnum trilobum - gets a bit large but they are beautiful! Mine grew fast, blooms by the second year.
They are not a true cranberry but taste very similar .. and wonderful fall leaf color!
This one looks interesting ...
Ben Lear Cranberry ... if you can grow it....?
Another one I've been meaning to check out is the
Juneberry ...
I think Rhubarb is attractive

... and Garlic Chive blooms.
Hyssop officinalis has been winter hardy for me for several years although I usually see it rated zone 6 or 7. Long blooming nice little bush.
How about a pole or trellis for some climbing Red Malabar Spinach or Scarlet Runner Beans...
Sweet pickle peppers are fun, or many of the ornamental peppers are edible and attractive, but some can get REAL hot.
Great idea to have a munching garden just outside the door ... very fun! Good luck!