Nitrate is not harmless, it is less lethal in a short space
of time as Ammonia or Nitrates, but it is harmful.

I'd say siphone the gravel, then after completion, do your water
change (to remove any debris from the water).

If the nitrates aren't going down, test your treated water before
putting it in the tank, if ok, then do more frequent changes
until you get the levels down.