The TV chef was cooking a fennel chocolate cake during the ITV show when James joked about his abilities – saying he had “never lost it.”
Galton quickly poked fun at the television presenter, telling the 45-year-old that he “never had it”.
The former Strictly Come Dancing star hit back at the remark, saying: “How very dare you.”
James proceeded to jokily hit the guest chef with a tea towel as he reached for a sieve.
The Yorkshire-born star was joined by guests Amanda Lamb and Jonathan Phang, who giggled at the awkward incident.
Galton teased James while he produced a pre-made cake from the oven, to which Amanda gushed: “Oh, you are good.”
TV actor Jonathan added: “Smooth.”
While James took the comment in jest, his guest chef was quick to apologise.
Galton, 55, has appeared on the Saturday morning show twice before, last year.
Last month, James suffered an epic blunder when he accidentally melted a £10 note in front of Eastenders’ Adam Woodyatt.
James Martin’s Saturday Morning returns next Saturday on ITV at 9.30am.