Memory usage is rarely as much of an issue as people make it out to be.
For Action Bar /Unitframe addons it should remain fairly static.
Look at the memory usage of an addon compared to the wow client and you will see how little of an impact they actually have.
Performance is killed by hogging the CPU and rarely anything to do with memory.