Ah, looks like you know your problem. Don't worry too much, it may take a few years, but eventually you can build up your soil just as good as your flower beds!
In our area buckwheat dies back without frost. Ours has never lived to frost. We grow it in the summer in our rotational cover crop beds. (1 of every 4 in the rotational blocks.)