If having sex represents your highest excitement in this moment, don't think about it - Just do it! Love it and enjoy it, there's nothing wrong.

There are many beliefs surrounding sex and different systems say different things like, do it, don't do it or do it X times per month or whatever.

The most frustrated people I've ever interacted with are celibate monks. IMHO, far more energy is lost and psychological damage done by those who self-impose abstinence, but at the same time have a deep yearning for sex, thus causing negative mental conflict due to unfulfilled desire.

The answer varies according to the "spiritual progress" definition. There are many systems of spiritual development.

• In Yoga, sexual experiences are stimulated by the same energy that can make you enlightened. There is a chakra system in every human with several energy centres placed vertically in the non-physical part of us. The lower three chakras are involved in the sexual experience, making it come true. People who do not practice Yoga (and have sex often) constantly keep the energy down. Yoga techniques are focused on bringing this energy up. This is called Kundalini. Eventually you reach the chakra on the top of your head and become enlightened. This is a very simplified description.
  So, as you see, to have sex for a Yogi is to back himself up. However, you can practice Yoga and be a householder. You are actually called a "householder" if you do, distinctively from "renunciant". You can have sex for the sake of your family/relationship but, let's say, only three times a month.
  This is Yoga. Note that it is very specific: not having sex plays a particular, reasonable reason.

• If you talk about sprituality, metaphysics at all, sex does no harm. If for you spiritual development is the development of your mind and your morality, sex cannot back it up. Concerning that you won't be eaten by the sexual passion and focus your mind entirely on sexual experiences, forgetting about the mental development.
  In fact, there is a system of spiritual development whose rule is balance. Having too much sex is as unnatural as having no sex at tall.

I have shown you this because you cannot give a straight answer about sex without defining the "spiritual progress" first.

There are likely as many ways of answering this as there are spiritual paths. I think it really has to do with how attached to sex you are that is ultimately the issue. The two conflicting facts of the matter are that A) as biological entities, humans are as hard-wired to procreate as most other things, and B) humans have the capability, though our developed intellect and will, able to consciously thwart and redirect many of our instinctual tendencies. What we are told about sex strongly colors our approach to it. For some, sex is a sin and it is almost unfortunate that as spiritual beings we have to resort to something so squishy and sensual to perpetuate our species (and the faith). Think about how Agent Smith reacts to his human body when he transcends his limitations as an abstract entity and enters into a physical body in Matrix Revolutions.

For others, as an act present in man's natural state and all his worldly brethren, sex is much more tolerated and sometimes worshipped, made sacred and not profane. Many fall in between these two extremes.

What many yogic and Buddhist schools would say is that for a layperson, sex should not be neither demonized nor taken for granted. It should not be a fixation either way, as much as nothing should really be a fixation. Attaching to and identifying with things and activities on this plane can only sap energies, increase suffering, lock you into the cycle of rebirth, however you want to say it. Just like how focusing on how you want to sleep will likely only keep you up, focusing on how much you want to avoid sex or how great it is aren't going to help much in the end.

I think it is much easier to see sex as a diversion of energy from spiritual development than it is to see it as a way of connecting to the Divine. Monks are often celibate because it is so simple to deny, forget about, move beyond something than it is to actually incorporate it in a balanced way.

So, it all comes down to you. Look inside and see what your higher self is telling you. If it says that its blocking you, its blocking you. If it says otherwise, it probably knows whats up.

It has for me, I really don't know why but it seems that some energy escapes.

For example, mediation becomes more difficult.