The ultimate test of a brush's quality is how well it picks up and releases paint. Unfortunately, you can't check that in the hardware-store aisle, but here are some other things you can look for.
Tapered Tips Look at the brush in profile. The tips should come to a tapered point—this gives you more control because they form into a narrow line as you press the brush against the work.
Full Body The cardboard or wood spacer creates a void that, on good brushes, holds a reservoir of paint. On cheap brushes, thick spacers make up for a deficit of bristles. Pinch the bristles between your fingers near their base. If they feel thin, you'll be dipping that brush a lot.
Firm Construction Tug on the bristles. If they come out in your hand, they'll come out in the paint. Ferrules, the metal bands that help hold the bristles in place, should be nailed through the handle and made of rust-resistant stainless or plated steel or copper.
Soft Bristles Touch the tips to your cheek or the back of your hand. Cheap brushes feel stiff and coarse; the feathery, soft ones will apply paint smoothly.
Price A quality brush, one that's made by hand, runs $10 and up.