The date of the birth of Christ was fixed in the fourth century. Before then, various dates were assigned to Christmas, including the 6th of January, the 25th of March and the 25th of December. The 1911 Encyclopedia says that the earliest reference to the date of December 25 comes from Theophilus of Antioch (A.D. 171-183). Hippolytus (c. 202) comments that Jesus was born in Bethlehem on that December date, a Wednesday, in the 42d year of the reign of Augustus. In 245 Origen objected to having a birthday celebration for Jesus.

The first certain mention of Dec. 25 is in a Latin chronographer of A.D. 354, first published entire by Mommsen.' It runs thus in English: "Year 1 after Christ, in the consulate of Caesar and Paulus, the Lord Jesus Christ was born on the 25th of December, a Friday and 15th day of the new moon." Here again no festal celebration of the day is attested.
Christmas - 1911 Encyclopedia

According to the Catholic Encyclopedia, in the fifth century, circus games were outlawed on December 25, by the Codex of Theodosius. It wasn't until the sixth century that Christmas became a legal holiday -- work was banned -- according to the Codex of Justinian. Later that century, Christmas became a holy period running from the 25th of December through Epiphany, preceded by a fast throughout Advent, that period in which modern children daily open little paper doors.
Latin Christmas Carols