dirt in pond

Question - and sorry for sounding clueless. That's because when it comes to
fish, I am.
However - I am wondering whether dirt that has escaped from my potted
aqautic plants onto the sand floor of the pond, can be harmful to the fish,
and can it raise nitrite levels? I don't have a clue how the dirt got out,
the fish must have done it somehow. Originally I potted the plants in
potting mix and then covered the top with 1/2 inch of sand before immersing.
Now there seem to be bits of dirt everywhere.

Thanks all
Nicola