The two stars worked together 22 years ago on the Oscar-winning period drama, Restoration.
Now the 57-year-old Brit has revealed how he believes the Iron Man actor had a problem with him.
Speaking with People, Grant said: “[He] hated me. We did a thing called Restoration, and he took one look at me and wanted to kill me.”
“I don’t know. I was so hurt.”
However Downey Jr has since reached out on Twitter to Grant, to make up.
The 52-year-old American tweeted: “A lot has happened over two decades! I respect how Mr. Grant has matured as an artist & voice against violations of privacy.
“Let’s break bread together soon @HackedOffHugh! #burythehatchet2018.”
Grant replied: “Thanks, @RobertDowneyJr. Nice way to kick off the year. And yes, if you’re in London come by and break bread.
“Won’t be easy as my 5 year old bakes it, but you seem strong. Respect. #burythehatchet2018.”
Grant has been nominated for a 2018 BAFTA in the Best Supporting Actor category.
The star played the villain Phoenix Buchanan, a washed up actor, in Paddington 2.
Meanwhile Downey Jr has wrapped on Avengers 4, which will be released in 2019.
