I'd say it sound like the 'jealous' person has some self esteem issues. That's usually where jealously comes from in the first place.
It also sounds a bit like the 'non jealous' person is pretty bitter about his or her fun being spoiled which is childish.
You should both be concerned with the other persons feelings and you should both do your very best to make you spouse happy.
Each person should be 100% on this 'lifestyle' you have chosen to live or you should not be together at all.