The state attorney general is investigating Compass after it closed a $1.6 billion acquisition of a rival brokerage firm.