I got a chuckle from that image of crawling and digging in a perennial bed Lavender, been there and done that. I won't say those are pleasant memories. That's thorny blackberries that got established in a landscaping bed, probably by birds dropping a seed and me being slow to dig it out. I'm still working on that but I'm getting closer. Perseverance is the key, plus never let it go to seed or get established.
I've avoided planting some things because I heard they were invasive, usually from people on this forum. Thanks for the warnings. The only invasives I have that I want, blackberries and mint, are planted where I can control them by mowing. I had a hybrid poplar to deal with, kept sending up shoots all over after I cut it down. Brush-B-Gone finally took care of that but it took several sprayings as sprouts just kept popping up. Perseverance! Johnson grass is in my main garden too. Talk about a mass of tubers if you don't get it when it first sprouts. I'll be dealing with that for a while. A neighbor has let burdock get established in his pasture next to me. It's a place that is hard to mow, but it keeps invading me. Constant mowing and judicious use of Glysophate are required. Sometimes my Glysophate might drift onto his side. He doesn't complain.
Then you have the constant invasive here, Bermuda grass. It's great in a lawn, hayfield, or pasture but a real problem in flowers and veggies. In non-food areas like my isolated iris beds I'll use a grass killer but anywhere close to things I eat I have to think of barriers and do a lot of digging.
In my opinion invasive plants are noting to take lightly. Be careful and don't let them through the door to start with. Especially don't invite them in.