So what is the bit with the green arrow about? That's where I have been connecting them.
Correct - the green arrow is where you connect passages.
The numbers on the doors (and a d6) are used for determining which doors are open/closed/gates after you reveal your exploration token.
So if you have a room with 3 possible doors, and your exporation token shows you have only 1 door (or gate), roll a d6. Compare the roll to the numbers on the map card to select your open door (or gate), and put end caps on the other exits.
Just make sure you specify whether you're rolling for open or closed doors, especially if a room has two possible exits and only one is open. My husband will often roll a die and announce the result. And I'll ask "is that exit open or closed?" The response is usually "I dunno." So we try to say "This exit is closed" before/as we roll the die.