According to Wikipedia Referees in football are first described by Richard Mulcaster in 1581.
In this description of “foteball” he advocates the use of a “judge over the parties”. In the modern era, referees are first advocated in English public school football games, notably Eton football in 1845.
A match report from Rochdale in 1842 shows their use in a football game between the Bodyguards Club and the Fearnought Club. Traditionally referees wore black but since the 90s more colours have been introduced nationally and internationally.