"It's fine, I don't mind," Aaron said. "They're not a secret. It just doesn't come up in conversation a lot. Especially since they're not really a part of my life - or I of theirs."
Other options:
no subject
"It's fine, I don't mind," Aaron said. "They're not a secret. It just doesn't come up in conversation a lot. Especially since they're not really a part of my life - or I of theirs."