Here is what you missing though:
If UK manages somehow to keep the good part of being in the EU w/t the worst part aka the Brussels ruling and according to UK the freedom of movement, then what would keep other EU countries from doing exactly the same thing?
In other words if they allow you to do that thing, EU would most probably siege to exist. This is the reason that i believe that not only you would have to agree to the freedom of movement but they would also shaft you somehow to make an example of you as a couple of other EU countries are preparing referendums.
That is, if you ever decide to leave EU (because this is a small chance you wont)