Time travel as depicted in movies like Back to the Future would only be possible if the multiverse theory held to be true. As that is time travel from the perspective and reference point of only one person, they would need to be able to transcend parallel universes in order to reach different points in time. Time travel within a single universe would cause a hell of a lot of problems, especially in travelling back in time. For example, if tomorrow I invented a time machine and decided to travel back to say 1990 (two years before I was born, and when my two sisters were infants) and then came knocking on my parents' doorstep, first of all they would think I was a lunatic and probably call the police if I didn't leave, but also they would have remembered this happening in the present and then realised as I got older that I looked exactly like the guy that knocked on their door 26 years earlier. Me deciding to time travel tomorrow would have been determined before I'd decided it by the encounter with my parents that I couldn't control because I hadn't been born yet, but at the same time that encounter was determined by me making the time machine. I probably haven't explained that very well, the same thing happens in the Harry Potter and the Prisoner of Azkaban. From within the confines of a single universe (and a single body) it doesn't make any sense and kind of destroys the laws of physics.