The Short Answer
The short answer to the question of what distinguished the ancient Roman freedman or freedwoman from the free born is the stigma, shame, or the macula servitutis ('stain of slavery'), as King's College's Henrik Mouritsen describes it in The Freedman in the Roman World, that never left the slave or ex-slave.
Over-generalizing about the citizens of ancient Rome, you may find yourself describing a tripartite wealth and status system. You might describe the patricians as the wealthy, upper class, the plebeians as the lower class, and the landless humiles -- basically the proletariat -- as the lowest of the freeborn low, those considered too poor to enter the military service whose only purpose for the Roman state was to bear children. Also considered humiles and generally lumped with the proletariat for voting purposes were the freedmen. Beneath these were the slaves, by definition, non-citizens. Such a generalization might possibly apply to the earliest years of the Roman Republic reasonably well, but even by the middle of the fifth century B.C., the time of the 12 Tables, it wasn't so accurate. Léon Pol Homo says that the number of patrician gentes dwindled from 73 to 20 by the year 210 B.C., at the same time the ranks of the plebeians swelled -- among other ways, through the expansion of Roman territory and the granting of citizenship rights to people who then became Roman plebeians (Wiseman). In addition to the gradual class shifts over time, starting with the great military leader, 7-time consul, and uncle of Julius Caesar (100-44 B.C.), Gaius Marius (157-86 B.C.), men of the proletariat class -- far from being excluded from military service -- joined the army in large numbers as a way to earn a living. Besides, according to Rosenstein (Ohio State history professor specializing in the Roman Republic and early Empire), the proletariat had already been manning the Roman fleets.
By the time of Caesar, many plebeians were wealthier than patricians. Marius is a case in point. Caesar's family was old, patrician, and in need of funds. Marius, probably an equestrian, brought wealth into the marriage with Caesar's aunt. Patricians might give up their status by being formally adopted by plebeians so that they could attain prestigious public offices denied the patricians. [See Clodius Pulcher.]
A further trouble with this linear view is that among the slaves and the recent slaves, you could find extremely wealthy members. Wealth wasn't dictated by rank. Such was the premise of the Satyricon in the portrayal of the ostentatious, nouveau riche, tasteless Trimalchio.
Distinctions Between Freeborn and Freedman or Freedwoman
Wealth aside, to the ancient Romans, Rome held social, class-based differences. One big difference was between a person who was freeborn and someone who was born a slave and later freed. Being a slave (servus meant being subject to the will of the master (dominus). A slave might, for instance, be raped or beaten and there was nothing he or she could do about it. During the Republic and first few Roman emperors, a slave could be forcibly separated from his mate and children.
"A Constitution of Claudius enacted that if a man exposed his slaves, who were infirm, they should become free; and the Constitution also declared that if they were put to death, the act should be murder (Suet. Claud. 25). It was also enacted (Cod. 3 tit. 38 s11) that in sales or division of property, slaves, such as husband and wife, parents and children, brothers and sisters, should not be separated."
William Smith Dictionary 'Servus' entry
A slave could be killed.
"The original power of life and death over a slave .. was limited by a constitution of Antoninus, which enacted that if a man put his slave to death without sufficient reason (sine causa), he was liable to the same penalty as if he had killed another man's slave."
Free Romans didn't have to put up with such behavior at the hands of outsiders -- ordinarily. It would have been too degrading. Anecdotes from Suetonius about the extraordinary and aberrant behavior of Caligula give an indication of how demeaning such treatment could be: XXVI:
"Nor was he more mild or respectful in his behaviour towards the senate. Some who had borne the (270) highest offices in the government, he suffered to run by his litter in their togas for several miles together, and to attend him at supper, sometimes at the head of his couch, sometimes at his feet, with napkins.
In the spectacles of gladiators, sometimes, when the sun was violently hot, he would order the curtains, which covered the amphitheatre, to be drawn aside , and forbad any person to be let out.... Sometimes shutting up the public granaries, he would oblige the people to starve for a while."
A freedman or a freedwoman was a slave who had been freed. In Latin, the normal terms for a properly freed freedman were libertus (liberta), probably used in connection with the person who manumitted them, or libertinus (libertina), as the more general form. The distinction between those libertini, who were properly and legally freed (via manumission), and other classes of ex-slaves was abolished by Justinian (A.D. 482-565), but before him, those improperly freed or disgraced did not receive all of the Roman citizenship rights. A libertinus, whose freedom was marked by the pilleus (a cap), was counted a Roman citizen. A freeborn person was not counted a libertinus, but an ingenuus. Libertinus and ingenuus were mutually exclusive classifications. Since the offspring of a free Roman -- whether born free or made free -- was also free, children of libertini were ingenui. Someone born to a slave was a slave, part of the master's property, but he could become one of the libertini if the master or the emperor manumitted him.
Practical Matters for the Freedman and His Children
Henrik Mouritsen argues that although freed, the former master was still responsible for feeding and perhaps housing his freedmen. He says the change in status meant that he was still part of the patron's extended family and had the patron's name as part of his own. The libertini may have been freed, but were not really independent. The ex-slaves themselves were looked upon as damaged.
Although formally, the distinction was between ingenui and libertini, in practice there was some residual taint. Lily Ross Taylor looks at the changes in the late years of the Republic and the early years of the Empire regarding the ability of the ingenui children of libertini to enter the Senate. She says that in A.D. 23, under the second Roman emperor, Tiberius, a law was passed mandating that the possessor of the gold ring (symbolizing the equestrian class from whose ranks young men were able to advance to the senate), must have both a father and paternal grandfather who were freeborn.
- The Freedman in the Roman World, by Henrik Mouritsen; Cambridge: Cambridge University Press, 2011.
- Review of Henrik Mouritsen's 'The Freedman in the Roman World,' by J. Albert Harrill, in PDF
- "Horace's Equestrian Career"
Lily Ross Taylor
The American Journal of Philology, Vol. 46, No. 2 (1925), pp. 161-170.
- "Legendary Genealogies in Late-Republican Rome"
T. P. Wiseman
Greece & Rome, Second Series, Vol. 21, No. 2 (Oct., 1974), pp. 153-164
- "Marriage and Manpower in the hannibalic War: "Assidui", "Proletarii" and Livy 24.18.7-8"
Historia: Zeitschrift für Alte Geschichte, Bd. 51, H. 2 (2nd Qtr., 2002), pp. 163-191
- On the social standing of freedmen as indicated in the Latin writers, by John Jackson Crumley (1906)
- Outlines Of Roman Law: Comprising Its Historical Growth And General Principles, by William Carey Morey
- Roman Political Institutions: From City to State, by Léon Pol Homo