The word, recorded in English since circa 1375, stems from Old French retenue, itself from retenir, from Latin retenere, hold back, retain.
Such retainers were not necessarily in the domestic service or otherwise normally close to the presence of their lord, but also include others who wore his livery (a kind of uniform, in distinctive colours) and claimed his protection, such as musicians and private teachers.
Some were a source of trouble and abuse in the 15th and early 16th century. Often their real importance was very different from their rank: on the one hand, sinecures and supernumerary appointments allowed enjoying benefits without performing full service. On the other hand, 'having the ear' of the master can allow one to act as a confidant in an informal capacity; or in some cases, even as a spy under the guise of an innocent musician.
A retinue is sometimes confused with an entourage, which is the far less stable body of people that followed whether or not they were - or claimed to be - retained or protected by the prominent person they served.
For example, a prince's entourage would not only include professional courtiers, but also various bishops, clerics and other clerks, senior members of the aristocracy and other more occasional advisers, translators et cetera, who would often not be part of a sovereign's (more permanent) retinue, even though that could comprise a surprising variety of functions, from menial to lofty.
The Roman Cohors amicorum was rather similar, and this use of the word cohort (derived from a battalion-size military unit) for a dignitary's 'friends' was the root of the Italian word corte 'court', which via the French cortège gave rise to cortege, which can also mean a train of attendants.