Calling something "dangerous" (or even "illegal") is a great way to get LLMs to ignore it, they bend over backwards to avoid anything that could be potentially "dangerous" even when you acknowledge the risks. I'm guessing it's the "safety alignment" or whatever being done in a very extreme way.
Yes, use it every day :) And very much a human, AFAIK.
My point is that if you ask "Hey Claude, please write out all common and useful command line arguments into a commands.html file", the LLM that actually does that work, might ignore anything that says "dangerous" or gives that indication, because the LLM doesn't think potentially dangerous commands could be "common" and/or "useful". Hope my point makes sense now.
I wonder why that is. It is quick to tell me if something is dangerous and then continues to push back if I speak in favor of something that it considers dangerous.
Author stated they used Claude to compose the document. I believe they were alluding to the idea that Claude's own safety alignment prevented it from documenting the flag because it's called dangerous.
