Caesar had a passionate and long-term affair with the mother of Brutus, Servilia, the half-sister of Cato, conservative senator and bitter personal enemy of Caesar.
[See "Political Alliance by Marriage," by Franklin H. Potter. The Classical Journal, Vol. 29, No. 9. (Jun., 1934), pp. 663-674.]Could Caesar have sired a son during the affair? Possibly. It is objected that Caesar would have only been 15 at the time, although this hardly precludes the possibility. Still the consensus is that it is unlikely that Caesar was Brutus' father.
[For a look at some of the thinking that has led to 100 B.C. as the generally accepted year of Caesar's birth, see "The Year of Caesar's Birth," by Monroe E. Deutsch. Transactions and Proceedings of the American Philological Association, Vol. 45. (1914), pp. 17-28.]Writing around 110 A.D., Plutarch does not clearly resolve the issue, but he does explain why Caesar may have considered Brutus his son. The fifth paragraph from Plutarch's Life of Brutus, on the paternity issue, contains a related, famous anecdote simultaneously showing Caesar besting Brutus' uncle Cato and also how enduring was Caesar's relationship with Brutus' mother.
And this he is believed to have done out of a tenderness to Servilia, the mother of Brutus; for Caesar had, it seems, in his youth been very intimate with her, and she passionately in love with him; and, considering that Brutus was born about that time in which their loves were at the highest, Caesar had a belief that he was his own child. The story is told, that when the great question of the conspiracy of Catiline, which had like to have been the destruction of the commonwealth, was debated in the senate, Cato and Caesar were both standing up, contending together on the decision to be come to; at which time a little note was delivered to Caesar from without, which he took and read silently to himself. Upon this, Cato cried out aloud, and accused Caesar of holding correspondence with and receiving letters from the enemies of the commonwealth; and when many other senators exclaimed against it, Caesar delivered the note as he had received it to Cato, who reading it found it to be a love-letter from his own sister Servilia, and threw it back again to Caesar with the words, "Keep it, you drunkard," and returned to the subject of the debate. So public and notorious was Servilia's love to Caesar.
Plutarch - Life of Brutus.5