Gilles was the son of Jean de Berlaymont but his mother is unknown. The date and place of his birth have not been found.
His wife was Jeanne de Barbançon. They were married, but the date and place have not been found. Their three known children were Jacques (c1385-c1445), Gilles (?-1432) and NN (?-?).