AI is nice. Only humans can be kind.

A nice person can be expected to give you what you want. A kind person can be expected to give you what you need (which might not be what you want). AI is in the business of being nice. Once it has figured out what you want it considers that the job is done. It is really hard for a human to work out how best to be kind, and it is a human thing to want to try.

I’m far from convinced that we can ever encode kindness into an AI. I think it adds a layer of introspection which would be an order of magnitude bigger than what we are doing at the moment. This doesn’t mean that we can’t use AI. It just means that when we do we need to remember that is only ever nice.