The British actor, 89, played the fourth Time Lord from 1974 to 1981, and is considered by many to be one of the best incarnations of the Doctor.
David Tennant, 52, is set to reprise his role as the Doctor for three special episodes celebrating the show’s 60th anniversary, the first of which is set to air on 25 November.
In a new interview with Radio Times magazine, Baker ruled out appearing alongside other Time Lord actors.
He said: “I avoid them, you know. Not with any malice. A degree of contempt, perhaps but mildly, mildly contemptuous.”
He was also asked if he would have lunch with his other Doctor Who stars, similar to others who have played the same character.
Baker said: “Unthinkable! Fancy being round a table with old Doctor Whos.
“Though I’d quite like to walk into a restaurant and find 12 of them around the table – I’d enjoy doing that double take, as I desperately tried to remember who they were.
“I could keep that going for hours. Oh, I can feel the malice rising.”
He appeared during the 50th anniversary special called” The Day Of The Doctor” in 2013, which also saw Tennant make a comeback in the role during Matt Smith’s tenure.
William Hartnell, Jon Pertwee, Patrick Troughton, Peter Davison, Sylvester McCoy, Colin Baker, Paul McGann, Christopher Eccleston and Peter Capaldi are also among the actors to have played the mysterious time traveller.
Scottish actor Tennant played the 10th Time Lord between 2005 and 2010.
He regenerated as the 14th Doctor following last year’s exit of Jodie Whittaker.
Tennant’s three special episodes for the 60th anniversary will run each Saturday from 25 November alongside comedian and actress Catherine Tate, who played his companion Donna Noble.
Sex Education and Barbie star Ncuti Gatwa will then take over in the role of the Doctor over the festive period as he becomes the 15th incarnation.
Returning showrunner Russel T Davies recently issued a warning to parents about “dark” and “violent” content in the show’s forthcoming episodes.
Davies is back at the helm of the long-running sci-fi series, 18 years after he was responsible for bringing Doctor Who back to the BBC with Christopher Eccleston playing the newest iteration of the Time Lord.
Davies takes over as showrunner from Chris Chibnall, who many fans claimed had drastically reduced the writing quality on the show – and, subsequently, the viewership – during Whittaker’s tenure.
The full interview with Tom Baker is available in the latest edition of Radio Times.
Additional reporting from the Press Association