I think the funniest thing here is that Big Black only have two studio albums. Both Atomizer and Songs About Fucking are essential. The former head been rereleased with the Heartbeat single as The Rich Man's 8-track Tape.
The three early EPs were collected as The Hammer Party and would be the next place to go.
They also have two live albums; I've only heard Pigpile but it hasn't done much for me.