The first I'll mention is Purple Loosestrife (Lythrum salicaria). It was introduced first in the 1800's and became a garden ornamental as well as a medicinal. Then it started plans for world domination. The edges of waterways become tinged with a purple haze of flowers when it blooms in mid-summer. Looks lovely, but the plant chokes out all other plant life around it. Cold weather doesn't seem to slow it down much as I've viewed it happily growing in Maine and along waterways in the Green Mountains of Vermont. There are chemical controls, of course, but using them near water is especially risky. Biological controls are also available and they do appear somewhat successful. Given the massive number of acres now invaded it will take a great deal of effort to reign this plant in. This perennial was once a recommended garden plant and it is found in gardening books of the past (including one of the Victory Garden books; Crockett's Flower Garden, from 1981).
Another plant that seems to be attempting a hostile takeover is Tansy (Tanacetum vulgare). Once planted as a useful medicinal and scented herb, it managed an escape and now turns up in many places it doesn't belong. The fern like foliage is thick and lush and the yellow button flowers are attractive. Sadly, it's tenacious, and much like the roadside daylily, it grows along roadsides and in areas where soil is not tilled. In fact, tilling the soil where Tansy shows up is one method of control. The foliage has a distinct spicy and herby scent, reminiscent of mint combined with Sweet Fern (Comptonia peregrina). The plant is poisonous to livestock and some people react badly to it if ingested. It was used as a culinary herb, an abortificant, packing material to help prevent meat spoilage, insecticide, and as a coffin stuffer to help prevent unwanted smells (according to Wikipedia). So here's a perennial that has potential use in the herb garden but just likes to wander a bit too much. Although not as invasive as Loosestrife, it does tend to crop up all over the place.
Recommended Plant: Alchemilla molllis, or Lady's Mantle. It grows as a low mound of folded leaves, similar to a cloak or mantle from wardrobe collections of years past. The greenish flowers mix well with other colors and the overall habit of the plant is similar to a ground cover. Perhaps the best aspect of the plant are the leaves. When wet, water droplets collect and bead up on the leaves. The plant is tough and will grow along walkways with little trouble. Unlike the other plants mentioned in the post, it's not especially invasive although in some situations it will self sow readily (personally, I've never found this to be the case).