The comedian and his professional partner Amy Dowden were eliminated after all three judges chose to save Simon Rimmer and Karen Clifton following the dreaded dance-off.
Both couples took to the dancefloor for a second time in a bid to progress to the next round of the competition but after making a mistake, the panel decided it was time for Brain to depart.
Explaining his decision, Craig Revel Horwood said: “Both couples actually made mistakes in that, one fell-over and then the other one blanked before they went into a back-drop so it made it a really, really difficult choice.
“But the couple I am going to save just for that little bit of swivel is Simon and Karen.”
Darcey Bussell added: “This is incredibly difficult with these two fabulous men and it’s not easy at all to choose between them.”
However, she noted Simon and Karen gave a more “finished performance”, with head judge Shirley Ballas agreeing.
Speaking to host Tess Daly about his time on the show, Brian made a shocking confession to his partner, revealing he would miss Aston Merrygold the most.
“You know what, there has been someone by my side all the way through this journey who has been just there every moment and that is… Aston!,” he admitted as Tess shook her head. “I love him and I am really going to miss him.”
But he added to Amy: “I love this lady and I want her to be on this show for ever and ever because I want everyone to know how special this lady is because she is a star and I love her to bits,” he said.
Strictly Come Dancing: It Takes Two airs Monday on BBC Two at 6.30pm.