Personally, I'd rather fight four Giants any day over an army of 8 Raiders. At least Giants are worth points. But yeah, each army has a problem unit, but I (as a Stark player) find that the Lannisters have unique ways of solving various problems, you just gotta get creative.
As I'm sure Oakwolf can testify, I hate the over use of Umber units in Starks, but they are not difficult to deal with, you just have to get creative. Same with Freefolk Giants, Bolton Flayed Men, or Nightswatch anything.
One of the rudest thinks a Stark player can do to you is charge you with Umber Greataxes, led by a Sworn Sword Captain. Missed saves generate extra hits, plus Vulnerable, your pretty much dead. How to counter this? Out manuver them, or weakness tokens. Giants swing a single dice and kill a crap tone of dudes? Weakness may help, unlikely but better than nothing, otherwise just hold out. Flayed Men are terrifying on a charge, but they are just like any other calvery, just slower. Nightswatch are no issue, Jon Snow and his NCU's are the issue.
The big issue with Lannisters against these groups tend to be how you approach them. And my favorite answer to this is usually Tywin Lannisters: Lord of Casterly Rock. Can he prevent calvery or Giants from having !ultiple wounds? No, but he can slower their roll like a mud slide, and spitting out weakened tokens with Pycel makes it all the better.
So have fun with that bit of information.