It's a long story. [As to why they weren't living together and why Subaru didn't remember anything.]

This is a dreamworld called Somarium. Somehow we got separated while we were traveling, and I arrived here first.But that's really further in the future from where you come from.

We don't live together because I live with someone that I've come to love. And your older self didn't want to get in the middle of that and decided to live here.