I think most people could probably get away with owning just a mild product (#9, SMR, etc) and a rubbing compound or something close. I wouldn't bother personally if you already own rubbing compound (fine cut I hope?).
Sometimes the only real diff between the two is the packaging. Scratch removers are usually used by hand to fix small spots needing repair. Rubbing compounds are usually used all over the car to restore dull paint. But the contents of the products are about the same.