Here are some of the issues we encountered during our trials (and tribulations) and some ideas on how you can solve them.

4.1 Paper clip button

You can give the generative AI assistant access to the style guide in one of three ways: upload the style guide using the paper clip button; give it the style guide’s full name (Interuniversity Style Guide for Writing Institutional Texts in English) and let it find it online; or point it to a particular published version using a web address.

Uploading the style guide is the best option if you require precise adherence and want to prevent the assistant from using and referring to other sources. The formats that are currently accepted by Copilot are Office documents and PDF. Upload the style guide at the beginning of the conversation and then ask the assistant to wait for instructions or upload it after providing context and telling the assistant that you are going to need a text revised according to the style guide’s recommendations.

4.2 British English

Generative AI assistants seem to be strongly biased towards American English. A recurrent problem the group encountered was the assistants’ patchy application of British English conventions. One way around this is to include this standard instruction suggested by Copilot that aims to cover all the possible differences:

“Please use non-Oxford British spelling, ensuring all American and Oxford variants are replaced with standard British English spellings. Pay special attention to differences such as -ise vs -ize, -our vs -or, -re vs -er, -ce vs -se and double consonants, and change specific words like behavior to behaviour.”

4.3 Never at a loss for words

The assistant never failed to provide some sort of response. The sensation is that it does not want to disappoint the user, so it will produce anything rather than admit to not having an answer (but see below). This, obviously, can cause problems. Examples include citing works that do not exist or even inventing whole new concepts. So, caveat emptor: AI-generated content should always be checked for accuracy.

4.4 Bias, blind spots and hallucinations

On many occasions, the group saw how the assistant’s output was biased by the input it had been trained on. Significantly, there appeared to be a clear gender bias, with male figures far outnumbering their female counterparts. Likewise, there are things the assistant doesn’t know it doesn’t know. But, as we’ve said before, that won’t stop it trying to come up with an answer. This often leads to what are known as hallucinations – plausible-sounding but factually incorrect text – and the assistant simply inventing a response.

One way around the issue of hallucinations is to allow the assistant to admit uncertainty by giving it explicit instructions to say it doesn’t know when it doesn’t know. This simple tweak can reduce hallucinations significantly.

Another way is to ask it to ground its response in facts – in this case, the exact sections of the style guide it is basing its response on – using quotes or citations. If it can’t back up a correction to a text by referring to a specific point in the style guide, it must not make that correction.

4.5 Variable results and multiple responses

Your experience may vary from day to day, and from assistant to assistant. Among the members of the group, the results obtained could differ greatly. Subtle differences in a prompt could lead to substantial differences in the response. Generative AI assistants are unlike most computer programs seen to date in that they are not predictable and consistent and, perhaps surprisingly, not very good with numbers.

One of the remarkable things about generative AI is its ability to produce multiple different responses to the same prompt. This is great for brainstorming and finetuning, but can cause problems when it comes to deciding which response is best for each task. You need to be careful with the assistants’ biases, blind spots and hallucinations, but you also have to be careful when going over the texts produced – an essential step in the process.

Interacting with these assistants gives you the impression that you’re dealing with someone who really knows what they’re talking about, but this is an illusion. What we enter as words, the assistant converts into tokens and responds to us in ways that are so convincing it can be hard to question its responses. It’s easy to take them at face value, but they always require a discerning eye before they can be considered ready for use. In one of our examples, we asked the assistant to produce a more informal version of the text, and it did. However, the result was probably too informal.

In conclusion, generative AI is pretty astounding, but also somewhat perilous. You need to be careful and bear in mind the dangers we’ve discussed. One way to think of the assistant is as an enthusiastic intern. As someone fresh out of university, they can help you with your tasks but they don’t have the knowledge, skills and experience you do, so you’ll need to go over their work before you can be sure it’s fit for purpose.