SB 3.31.39

saṅgaṁ na kuryāt pramadāsu jātu
 yogasya pāraṁ param ārurukṣuḥ
mat-sevayā pratilabdhātma-lābho
 vadanti yā niraya-dvāram asya
Synonyms: 
saṅgam — association; na — not; kuryāt — one should make; pramadāsu — with women; jātu — ever; yogasya — of yoga; pāram — culmination; param — topmost; ārurukṣuḥ — one who aspires to reach; mat-sevayā — by rendering service unto Me; pratilabdha — obtained; ātma-lābhaḥ — self-realization; vadanti — they say; yāḥ — which women; niraya — to hell; dvāram — the gateway; asya — of the advancing devotee.
Translation: 
One who aspires to reach the culmination of yoga and has realized his self by rendering service unto Me should never associate with an attractive woman, for such a woman is declared in the scripture to be the gateway to hell for the advancing devotee.
Purport: 

The culmination of yoga is full Kṛṣṇa consciousness. This is affirmed in Bhagavad-gītā: a person who is always thinking of Kṛṣṇa in devotion is the topmost of all yogīs. And in the Second Chapter of the First Canto of Śrīmad-Bhāgavatam, it is also stated that when one becomes freed from material contamination by rendering devotional service unto the Supreme Personality of Godhead, he can at that time understand the science of God.

Here the word pratilabdhātma-lābhaḥ occurs. Ātmā means “self,” and lābha means “gain.” Generally, conditioned souls have lost their ātmā, or self, but those who are transcendentalists have realized the self. It is directed that such a self-realized soul who aspires to the topmost platform of yogic perfection should not associate with young women. In the modern age, however, there are so many rascals who recommend that while one has genitals he should enjoy women as much as he likes, and at the same time he can become a yogī. In no standard yoga system is the association of women accepted. It is clearly stated here that the association of women is the gateway to hellish life. The association of woman is very much restricted in the Vedic civilization. Out of the four social divisions, the brahmacārī, vānaprastha and the sannyāsī — three orders — are strictly prohibited from the association of women; only the gṛhasthas, or householders, are given license to have an intimate relationship with a woman, and that relationship is also restricted for begetting nice children. If, however, one wants to stick to continued existence in the material world, he may indulge in female association unrestrictedly.