There's not much detailed information about the early popes. For many years of the Church's history, celibacy was considered optional.
Based on the customs of the times, it is assumed by many that like Peter, most of the Apostles were married and had families. Author Richard McBrien writes in Lives of the Popes that "from Paul's reference to the fact that Peter and the other apostles took their wives along on their apostolic journeys, (1 Corin 9:5) the apostles did not 'put away' their wives."
In the Vatican document, Priestly celibacy in patristics and in the history of the Church, Roman Cholij writes, "It is clear from the New Testament (Mk 1:29-31; Mt 8:14-15; Lk 4:38-39; 1 Tim 3:2, 12; Tit 1:6) that at least the Apostle Peter had been married, and that bishops, presbyters and deacons of the Primitive Church were often family men."
"It is also clear from epigraphy, the testimony of the Fathers, synodal legislation, papal decretals and other sources that in the following centuries, a married clergy, in greater or lesser numbers was a normal feature of the life of the Church. Even married popes are known to us."
Mandatory celibacy was enforced because there was so much political and economic power attached to the papacy especially during the fifteenth and sixteenth centuries. The Church has adopted celibacy as a matter of discipline, not as a matter of doctrine.