If you're maintaining conditions that allow for plant growth, don't
worry about it. Your only issue would be if the plants become
unhealthy and stopped consuming the ammonia in the system. If you
remove plants, establish your biological filter bed, then add the
plants back, you'll end up with the same thing you'd have if you just
left the plants in -- assuming you have conditions conducive to the
growth of nitrifiying baceteria, your filter bed will compensate for
whatever your plants aren't consuming.

I.e. if your plants are healthy and your ammonia/nitrite levels are
fine, then don't worry about it.