Possible sources of texture problems in my experience:
* Prep issues - not filing off bumps and mould lines and so on. Sanding or buffing the surface can help, but really that's more for display contest level results. Giving the mini a quick scrub with soap and water gets off any mould release or skin grease that might be on it and helps the primer adhere better.
* Priming technique - if you hold the spray primer a little too far away from the mini, droplets can dry on the way over and cause a dusty or bumpy surface. Spraying in a dusty environment also isn't great.
* Primer - as mentioned, the primer itself can be not the greatest or not suitable for minis. Lots of people use Duplicolor sandable, I believe, so you're probably fine there.
* Paint - if you use twist off lid paint pots, particularly if you paint from the pot/lid, you can get grit and fibers in the paint pretty easily. Either from the environment, or from what dries up in the thread of the cap and around the edges and falls in. Generally these are larger chunks, though. Paint can go off if it's exposed to temperature extremes, and some paint colours/brands have a bit of graininess. If it's an all over the mini issue, though, it's more likely a prep or priming thing.
* How you use the paint - if you use a wet palette you leave open a lot and don't change the paper on very often and you have dust and cats and so on, you can get a lot of gunk in your paint. I had problems in the past with over-vigourous stirring on a wet palette rucking up fibers from the paper that made for a noticeably bumpy surface when I took pictures. Also consider what you're drying your brush on, it might be a type of cloth or paper towel that has a lot of fibers.