If I write a poem and put it into Stable Diffusion, how is what is produced not a derivative work of my poem? We can argue that it's a derivative work of many other things, but that doesn't make it not a derivative work of the poem.

One way it might not be is if Stable Diffusion is seem more like a hash algorithm than a synonymiser. But I don't see why it should, because there's a meaningful correspondence between the input and the output of the system.