You can add them if you like. How large are the containers you will be using?
Also, you would need food for them (compost, avocado, ect.) some kind of organic matter. Also you would want to have a mulch layer at that point to retain moisture. This can make a home for pests. I always encourage ppl to go the living soils route but it's a hard road to hoe at first. After your first and or second cycles it gets ssswwwweeeeettt.
Also, be aware that there are different types of worms, they live at different depths of the soil column (don't buy night crawlers for a 3 gallon pot). If you buy red wrigglers (trout worm type) they like to eat leaves and duff. they don't really live IN the soil.
If you wanna throw bait worms at your plants that's fine but you can bring soil-born pests (microscopic) to the party.
In short, If you really care about your plants, do some research into worms in cannabis. Anything worth doing, is worth doing right.
Especially if it yields sweets flowers from God . . .