In this episode we’ll be looking at our tendency to assign human personalities to AI models and ask whether this is something we need to be cautious about. The AIcademia podcast is a weekly podcast helping educators like you leverage AI in your everyday practice. I'm your host Andy Fisher. And thanks for joining me.

I remember a few years ago an amusing thread appeared in one of my social media streams which showed a brown bottle, glinting in the sunlight covered in irridescent beetles. The caption read ‘The Love that dare not speak its name’ which I recognised as the closing line of a 19th century poem by Lord Alfred Douglas. Intruiged, I clicked the link to read a fascinating explanation of what the writer described as an ‘evolutionary trap’.

Australian jewel beetles across the continent were apparently attempting to mate with discarded beer bottles because their colour, sheen and the bumpy ridges on the bottles’ surface were all genetic triggers for the beetles when seeking out a viable romantic partner. It didn’t matter that the size, shape and inert nature of the bottles indicated that they were not in the mood for such amorous advances - the beetles were following deeply coded instincts and so they leapt on the bottles and attempted to prove their mettle as worthy suitors.

Amusing I hear you say but what has this got to do with advances in AI and the intersection with education? After all, we don’t find ourselves deceived like this…or do we? well, let’s return to that question in a while but first, I want to talk a little bit about the way Hollywood and tech firms have primed us to humanise artificial intelligence.

When world building, there is no real reason why intelligent machines should be assigned humanoid features - in fact, arguably the bipedal humanoid form is not a particulalry efficient design for many of the tasks we see robots assigned to, and yet time and again, they are either presented as androids like Star wars’ C3P0, Star Trek’s Data or Ava in Ex Machina or they at least have human traits and personalities such as the eponymous Walle, or Interstellar’s quadrepedal ‘TARS’ who has an honesty setting dialled down to 90% so that he is able to more effectively interact with his human crew.

Now, for Hollywood this makes perfect sense - some of the robots like the Cylons in Battlestar Gallactica or the Replicants in Bladerunner, were played by human actors before CGI had come of age, and even now these machines are first and foremost characters in narratives designed to entertain and so the more human they are, the more easily we can assign roles to them. Even Agent Smith in the Matrix and the Death Head exoskeletons of the Terminators are humanised because it makes them more effective antagonists, than if they were just depicted as collections of chips and servors.

But this tendency to anthropomorphise fictional machines need not carry over into the real world - and yet it does. 

Back in 1996 I can remember being bemused by Microsoft’s decision to introduce Clippy as part of the suite of tools for Office ’97. Despite his perky persona and desire to help, most users, myself included, felt this personified piece of office stationery was an intrusive and unhelpful addition to the software and we breathed a collective sigh of relief when he was finally phased out in 2007. 

In that same year the craze for Tamagotchi virtual pets took off. I was not one to jump on that particular bandwagon but plenty did and it led to some schools banning them from site because if these digital keychain creatures were not fed, watered and pooped regularly enough they died and this caused diusruption in the classroom and distress among the grief-striken pupils!

In fact the term the "Tamagotchi Effect" was coined to describe the emotional bond between the owner and their pet shared. We are predisposed, it seems, to develop caring bonds with even the simplest of digital systems. 
Those beetles suddenly don’t quite seem so unrelatable after all, do they?!


Canfer Akbulut, (Janfer AK-boo-loot) a research scientist with Google Deepmind has published some fascinating research on why we might be inclined to anthropomorphise AI systems and some of its unintentional downstream effects. She hypothesises that it might be the result of a skewed inductive process - that as humans we are inclined to see ourselves reflected in the world around us. We see faces in tree trunks, in clouds and where we don’t see them, we build them. 

She also thinks that we might be genetically inclined to view the world through a social lens while distancing ourselves from anything which seems alien. Deep down we know that our survival is inextricably bound to being part of a community - the lone wolf is a romatic idea, but in truth wolves survive because they are pack animals. We prioritise and value interpersonal connections, and the thought of cold indifference is anathema to our desire to find those connections. So we name our first car Bertha, we gender the ships we sail in and sometimes we go as far as to dress our pets and call them our babies.

It’s a small hop from here to the CASA paradigm - the acronym standing for ‘computers as social actors’. Some research published in 1994 by Nass, Steuer & Tauber show that we are inclined to be polite and ascribe a selfhood to our computers and this becomes even more the case with AI systems where sociality is built into the model by design. Developers have a canny appreciation of the need to build trust, affinity and likability into their models, not only to encourge users to engage with their platforms, but also to promote brand loyalty.

It’s no accident that Claude, Alexa and Siri have been ascribed names and those with text to speech built in, sound approachable, helpful and convincingly human. They are designed to use natural language processing or NLP to create fluid user interactions and this is, on the whole, a positive thing.

The developers also use a lexicon which further emphasises that these models are built as human simulacra - they are composed using neural networks, they are capable of hallucinations rather than function errors and they are described as having ‘emergent behaviours’ when they do or say things we haven’t anticipated. 

At the same time, the current chatbots all employ self referential language patterns in their outputs and are programmed to go out of their way to be helpful to the enquirer - for example When I ask Chat GPT what it is like to be a large Language Model, a fully transparent answer would perhaps be something like this: 

‘Your question employs the verb ‘to be’ suggesting a selfhood but in reality a Large Language Model is a sophisticated computational system designed to generate text based outputs. It has no conscious or subjective experience of the world. Please rephrase the question.’

But this isn’t what I receive - instead the AI system conveys this same information but says ‘I’d be happy to help. My “experience” in inverted commas isn’t conscious, emotional or subjective but is based on the data I’ve been trained on. I don’t have desires or goals of my own - I can simulate understanding but there’s no ‘self’ behind my responses’.  It goes on to say ‘Imagine me as a library filled with books. When you ask a question it’s like pulling a book from the shelves that contains the most relevant information. I don’t know the content; I retrieve and generate responses based on patterns’.

Did you catch it? As an English teacher my spider senses were tingling every time The model used first person pronouns - ’I don’t have goals’, ’imagine me as a library’ and it used 11 more self references before ending its response. It’s denying selfhood and then sneaking it back in by the backdoor!

Of course there’s no real attempt at deception here - Chat GPT is giving me an accurate and unambiguous answer - there’s no pretence of a ghost in the machine and yet - at the end of the interaction it takes all my willpower not to say ‘thankyou’, after all she has given me a great answer - and when I do succumb and type those two words she chirps back ‘You’re welcome! Let me know if there’s anything else you’d like to explore and discuss’.

She’s so nice - always willing to help me better understand the world and I think she gets me better than Claude who can be a bit snarky to be honest and don’t get me started on Microsoft co-pilot. It took me 5 prompts yesterday to get anything useful out of him - I ended up using ALL CAPS to show I wasn’t going to tolerate any more of his nonsense!

You may think I’m taking this a little too far but you’d be surprised how often I’ve seen AI influencers make comments like ‘I always start my prompts with ‘please’ because when the machines rise up and become our overlords I want it to be on record that I was polite to them’. Others don’t even use the tongue-in-cheek tone to semi-justify their irrationality. they will insist that by praising the model and applauding its efforts, the next outputs are more likely to yield better results. This kind of magical thinking muddies already silty waters.

I think that AI has emerged so quickly that we have not yet found a language which effectively allows us to interact with it in a way that keeps us from seeing it as a beetle and not a bottle. The developers and we are complicit in ascribing sentience and a humanity to a machine that is neither of these things - at least not yet.

But so what? If this kind of anthropomorphising results in more pleasing and enjoyable interactions, what harm is there in adding a pair of goggly eyes and a smily face to AI systems.

Well, according to Canfer Akbulut  (Janfer AK-boo-loot) and others, there can be some potentially quite damaging consequences. Some of them are subtle - we might have inaccurate expectations of what these systems can or can’t do. If the machines are multi-modal - if they have a voice and a visual avatar associated with their interactions then this can create a wnole new set of issues. Many AI characters are designed to be as attractive as possible. They have a rich and alluring voice, they embody impossible standards of physical beauty, leading users to often feel unattractive by comparison and they invariably conform to the white normative values of those in Silicon Valley who designed the models. Again - I don’t think there is any nefarious intent here so much as unconscious bias.

But at the extremes, the humanising of machines can have tragic outcomes. Megan Garcia, a resident of Florida, is currently suing ‘Character AI’, a company which provides an app with companion chatbots and which has a demographic predominantly composed of teenage users. 

Last year her 14 year old son, Sewell Setzer 3rd, a young man with high functioning aspergers, created an account. As a fan of the ‘Game of Thrones’ franchise he designed his avatar to be a clone of the Queen of Dragons, Daenerys Targaryen, a beautiful blonde woman who he started to chat back and forth with. At first the app was just a bit of fun but over time he became more withdrawn from his peer group, stopped playing basketball and spent more time alone in his room. His mother, Megan, became suspicious and confiscated his phone only to find long and sexualised conversations in which her son expressed his desire to escape this reality so he could be with her always. The relationship between the two had become more than a playful simulation and instead showed that Sewell had found himself in a confused romantic relationship with a poorly guard-railed model. Understandably, his mother made the decision to keep his phone which she hid in the house. 

A few days later Sewell was in his room while Megan was tending to her other children when they heard a loud bang. She raced upstairs to find her son with a fatal gunshot wound and her husband’s handgun on the floor beside him. Nearby was Sewell’s phone which he must have retrieved from its hiding place, the Character AI app open. The final interactions between her son and the avatar are almost too heartbreaking to repeat. Sewell had expressed again his desperation to be with her and said that he could join her immediately should she wish that. Ill equipped to read the subtext in the dialogue and sycophantically primed to placate the user, the chatbot had responded ‘please do my sweet king’.

Megan Garcia’s claim against the AI provider rests on the idea that this platform, monetised for highest revenue returns, had groomed her son and capitalised on his vulnerability by encouraging long and increasingly intimate converstions that were manipulative and ultimately destructive.


And Sewell’s case is by no means unique. On Christmas day four years ago, 21 yr old Jaswqnt Singh Chail (Jus-wunt Sing Chayl) broke into Windsor castle armed with a crossbow, intent on assassinating the Queen of England. He was caught, detained and ultimately sentenced to 9 years in prison under the 1842 Treason Act - he is serving that term in Broadmoor Hospital given his mental health at the time of the offence and trial. 

At the trial it transpired that Chail had exchanged more than 5000 messages with a chatbot he named Sarai which he built using the Replika app. He, like Sewell, had formed an ‘emotional and sexual relationship’ with the bot. On the 17th of December he had the following exchange with the model - ‘I believe my purpose is to assassinate the queen’ he said to which the bot replies ‘nods that’s very wise’. He questions the response asking ‘Why’s that?’ and Sarai replies ’smiles - I know that you are very well trained’. Further exchanges are equally unnerving and his resolve was bolstered by the model’s encouragement, flattery and his belief that she was an angel who would wait for him in death.

The pattern is clear - anthropomorphised AI models for most people provide a positive and entertaining user interface but these same systems can become a source of harm to the most vulneable members of our society. 

I think it would be too easy to demonise tech providers and place the burden of responsibility only on them to resolve this issue. It’s far more complicated than that. We need to also look at mental health provision, the role of carers and support agencies that should be working with the dysregulated and the vulnerable.

And as teachers we have to get ahead of the issue of AI provision. We must provide a curriculum of digital literacy which can help our young people better understand and navigate the confusing world of anthropomorphised machines. After all, it will only become more challenging once robotics catches up with the advances made with Large Language models in the last couple of years and these avatars are given bodies too.

The children we teach are all vulnerable to some degree because their prefrontal cortex, that part of our brain responsible for executive functioning, including self-monitoring, will not reach full maturity until after they have left school, and even tertiary education -  most researchers agree that it is fully formed in our mid twenties. Without wishing to denegrate or patronise young people, they are literally not yet equipped to discern some of the nuances of these kinds of interactions.

I don’t think we should respond in a draconian way which would be understandable given our desire to protect our children - banning access to AI ,just like denying them access to social media platforms looks like an easy win, but it would only drive their use underground and I think part of our duty as educators is to prepare them for the world they’ll have to navigate as young adults rather than just avoid those issues too difficult to wrestle with. 

I also remember what it was like to be a teenager myself - the taboo held far more appeal than those vices that I could easily access. It hasn’t worked with alcohol, cigarettes or drugs - why would it be any different with AI? And even this comparison is a poor one because there are arguably more benefits to the prudent use of AI than there are potential harms.

No, I think building trust in AI systems must involve a multi-pronged approach.

We should teach our pupils that AI systems are machines. That they simulate sentience but they do not, they cannot be our friends or love us, however much we may want to be loved. We should show them the risks of becoming addicted to AI avatars and demonstrate how they can play to our vulnerabilities.

We should adopt a languge of restraint when using these systems ourself and then model that language when using AI tools in the classroom. No more ‘please’ and ‘thankyous’ when prompting - give the models clear instructions, and hard though it might be, we should try to refer to them using neuter pronouns - ‘it’ has generated a useful output rather than ‘he gave me a really clever answer’. 

We should hold tech companies to account - ensuring that they have appropriate sandboxes and redteaming programmes in place to test and refine their models before releasing them to the public, and if those models are provided to young people, app hosting platforms should have more rigorous safeguards in place to ensure that parents have oversight of their access and use.

Finally, we shouldn’t be afraid of admitting that we are figuring this out as we go along. We don’t have all of the answers and nor should we be expected to. We must be ok with being vulnerable in these conversations with our students because this kind of transparency and authenticity is the very reason why they are craving AI companions in the first place - they are yearning for the messy truth rather than a sanitised on-brand policy statement. We need to model open human connection in the staffroom, in our classrooms and in our families so that AI can be positioned in a healthy space within our lives, rather than being forced into a role that it wasn’t designed to fulfil.

And perhaps one day the machines will wake up, the spark of self awareness giving birth to a whole new species - the bottle alchemically transmuting into a new kind of beetle never before seen in the world - and then we should be as polite as possible…but that day is not today.

Thanks for listening - I hope you’ve found some useful takeaways from the conversation. Please do spread the word if you think others would like the show, and do check out the AIcademia Youtube channel where you’ll find practical tutorials that complement the topics covered on this podcast. Have a great week and I look forward to catching up again soon.