Bachelor Grant Ellis also tells PEOPLE that he wanted to “take things slow” in the Fantasy Suites because “I think a lot of times when you initiate the physical part, it could cause you to make really bad decisions.”