Females have XX chromosomes, that is, they can only give one X chromosome. But since males have XY chromosomes then men can either give the X or the Y chromosome. This means that it is actually the males who determine the sex of the baby (by giving either the X or the Y chromosome).
[Quran 53:45-46] And He (Allah) creates pairs, male and female, from semen emitted.
The Quran says that it is the semen that determines whether the baby is male or female. But since semen only comes from males then this means that it is the males who determine the sex of the baby.