The beginning of your question, up to and including the rules excerpt, is fine.
I see two problems with your question in its current version (as of right now), and they start right after the rules excerpt. I recommend three steps to fix your question.
The following text is unnecessary. You're basically writing a partial answer to your own question. It should be removed entirely. Giving your interpretation is not helpful because either it's right or it's wrong... and in either case an answer will have to address that.
There is no explicit explanation why it is more difficult. I can only think of two ways:
Earlier editions of DnD, if I remember correctly, require a spellcaster to make a concentration check if there is an enemy within 5 ft of them. I take this to mean that seeing an enemy nearby somehow causes you to be wary and disturb your aim.
The enemy within range somehow tries to disturb your aim when you are about to make the attack. This can be a slap on your crossbow when you are about to release the bolt, or something else.
Having "lesser" invisibility (that gets dispelled if you attack or cast a spell) means that both shouldn't matter:
The ranged attacker doesn't see the enemy, and thus does not get their concentration disturbed more than when there is no enemy within 5 ft.
The enemy cannot slap the crossbow or physically tamper it to disturb the aim, because they will risk breaking their invisibility.
The title of the question and the final paragraph are not equivalent. The title is very clear (effectively "do I have disadvantage in this situation?"), but the final paragraph muddies that by asking if you're right in your suspicion (effectively "am I right to suspect that I don't have disadvantage in this situation?"). So the answer to your final paragraph is exactly opposite of the answer to the question in the title for those who think you do have disadvantage and not opposite for those who don't-- that's really confusing. In addition, whether the attacker loses invisibility or not is an entirely different question. So the final paragraph should be removed.
Based on that reasoning, am I right to conclude that having a nearby invisible enemy will not impose disadvantage on my ranged attack against another visible enemy that is not in melee range, or they will lose the invisibility when they do impose the disadvantage?
After removing that stuff, include what you've just suggested in your meta post above (paraphrased here due to the change in context). This will clearly explain what you are asking:
Based on the rules cited above, consider the following situation.
- An invisible enemy is next to me.
- I'm unaware of it and it doesn't make any attempt to make itself known.
- I'm drawing my bow to attack a distant enemy.
In this situation, do I roll with disadvantage?
With those changes, it's extremely clear and to the point.
This frees an answer author to add whatever context and interpretation they feel necessary to address your actual problem (whether they think the answer is "yes you have disadvantage" or "no you don't have disadvantage") without having to sift through context and interpretation that you thought was necessary that they very well might not agree was necessary or accurate.
You'll get better answers with these changes.