For holes in clothing, or ripped and frayed edges I would work in layers of putty.
Sculpt the initial form of the cloth, let dry. Add a new, thin layer over that and allow it to set up to a workable consistency. Once you can work it, push it around until you get the effect you want.
For holes, just leave the underlying surface showing through. For tattered edges, draw out the edges with a pointy tool, or tool with a thin edge, like a dull exacto blade. Once you get the little tendrils of cloth dragged out, begin to shape them, trim them and smooth them into place.
Simply put.
1. Sculpt the form.
2. Add thin layer of putty on top.
3. Sculpt holes and frayed edges on thin layer.