Leaves in Pond

A question: I have lots of trees in my yard and every Fall the leaves
fall and some fall right into my little pond. I try to scoop them out but
since the leaves keep falling I just leave the leaves alone in the pond
until the next Summer when the pond water is warm and I can stick my arms in
it and pull out the leaves. The fish seem to like the leaves as it gives
them hiding places. Anyway, my question is - are leaves poisonous to pond
fish? I recently heard leaves are poisonous to fish but they don't seem to
have hurt my fish or the five little frogs that just jumped out the pond
this weekend.