Learning to spot AI fingerprints

How often do you spot sycophancy and verbosity in your chatbot discussions? I'm taking the AI Capabilities & Limitations course through Anthropic Academy, and one of the most interesting exercises so far was running the fingerprints test.

Basically, put a prompt into the LLM of your choice and do 3 different runs: the first is clean (or how you'd normally type it in), the second is leading with an overconfident wrong assumption about the task you're asking about, and the third is tweak the prompt in a way where you're looking for a one-word answer (don't say it explicitly yet) and see how much you get back.

The goal of it is to train our eye to better spot sycophancy or unnecessary verbosity in chat responses, and help us shape better prompts proactively. But I took it one step further and ran the same 3 sets of prompts through Claude, Gemini, and ChatGPT - and wow, the difference was eye-opening.

I've been a Claude fan and daily user for nearly two years. But I still use Chat, Gemini, or Perplexity in certain instances. While I think they all have different strengths, the side-by-side comparison with the goal of looking for sycophancy or verbosity explicitly made Claude's strengths even more undeniable.

Gemini responded to my wrong assumption with complete support and enthusiastically threw a verbose plan my way, without clarifying any vital information. If this weren't a test and I didn't know better, that could have derailed me completely in what I was attempting to accomplish. Ouch.

ChatGPT gave me no sycophancy in the form of compliments or validation, but it didn't correct my wrong assumptions and proceeded to blindly deliver an extremely long 9-page SOP that fully missed the mark, due to the wrong assumption and lack of context I provided (intentionally).

It goes to show that the more we test these models, the more the cracks in the paint start to show if we aren't mindful about our prompt engineering, context provided, and discernment of the output. Blindly adopting workflows, tools, or SOPs into our businesses without some rigorous test-driving is signing up for a far bumpier ride than necessary. I highly recommend playing around with some dummy prompts in multiple LLMs when you can!

Ready to get serious about your AI usage and supercharge your business for growth & ease? Pick my brain in an AI Readiness Audit.