Note: As I start writing this in the EditCred browser extension, I realise the only way to start a new document is by copying some text from a chat with AI.
While this was a design oversight, it is also a reflection of the behaviour observed at large leading to the creation of EditCred
- Most thinking now begins in the AI prompt box
- Selected text snippets are copied out into an editing tool of choice for final output
- A whole lot of back and forth in between!
In case of this post though, I knew what I was going to begin with and maybe use ChatGPT or Gemini to add facts and references as I went along.
This is now a feature request in our roadmap.
Swiftly brings me to the title of this post.
The flow described above is the ideal state - different for different individuals and their occasions or requirements.
Often though, I personally start by chatting with a LLM - presuming efficiency, quality, coverage or all of these.
And I’m probably not wrong.
A 2024 Harvard study found that for mid-level professional writing tasks, ChatGPT reduced time taken to complete tasks by 40% and improved quality by 18%.
You probably do the same when working on an assignment, note or presentation.
- [Start with a rough outline in your mind or in a doc]
- Prompt AI with your overall/specific goal
- Receive a detailed response
- Pick out parts of the response and create/modify your own document
- Go back to the conversation for refinement/expansion
- Pick out relevant parts again
- Go back to your document and modify based on inputs - including complete rewrites and restructuring
- Repeat the preceding 3 steps
- Perform a final check of the document before it’s ready for the next bit of your workflow - be it sharing as is, posting online, or converting into a presentation
If true, it seems most people are not using LLM responses directly. A few questions then come to mind related to this workflow.
- Is this process actually effective? Or, ceteris paribus, are there real gains?
As it seems, yes.
As per the Harvard study, the greatest “real” gains are seen in inexperienced/lower-performing individuals.
Another study on Human vs. LLM creativity states that LLMs provide a “scaffolding” that allows those with weaker writing or organizational skills to produce work at a standard professional level - a phenomenon researchers call “Raising the Floor.”
LLMs excel at execution quality - producing grammatically flawless, well-structured, and logically coherent text (almost always). This reduces the “cognitive load” of drafting and initial ideation.
- Are these gains real or just perceived to be real?
It seems there is often a gap between how productive a user feels and how much they actually achieve.
Gemini says (without a linked reference study) that, in coding and complex logical tasks, users often perceive a 24% productivity gain, but in reality, they can be up to 19% slower due to the time spent “babysitting” or fixing subtle LLM errors. I’m inclined to believe this.
In my experience, LLM output seems fairly robust at first glance - a phenomenon titled “Simulated Authority” in a study on long term adoption of AI in academic writing. Often though, a careful second reading reveals inconsistencies. This leads to either restarting with a revised prompt or providing context for realignment in the same conversation - which often results in even more convoluted/inconsistent responses.
Hence, while the gains are real in abstract, in the real world their effect may get diluted with more effort required for fact-checking, alignment and prompting.
- What is the impact of these gains? What does one human user contribute over and above the condensed collective wisdom of all of humanity? What can we learn from the professionals?
It seems then - and you’ve probably heard this enough times already - that with good quality prompting, which is the real needed qualification, it is now possible to reach near or at professional levels of output in virtually no time.
What happens though when everyone, almost suddenly, is a professional! Or has the potential to be.
Researchers highlight the mechanism of ‘Semantic Space Collapse’ in generative AI and establish a homogenisation effect that ‘lowers the ceiling’ and does not improve ‘collective diversity’.
The answer seems near obvious then - creative originality is one, and maybe the only, way to contribute over and above LLM assisted writing.
But wait. With excellent, say top percentile, prompting skills, and ever improving LLMs, it seems easy enough to produce an output that ‘feels’ original.
For many situations, that is more than sufficient. New breakthrough technology is supposed to do that - make our lives easier.
In other situations, maybe not. Say in situations that need value creation. Or outsized value creation, since baseline value creation is obviously made more accessible with new technology.
For those creating, or attempting to create, or measuring outsized value, the problem then is of attribution.
Much like commodity brands, for whom attribution is one of the main ways to escape pure price competition.
In writing, would one then ask - will the real professional please stand up?
Slim shady references aside, how does one signal or measure the creative originality?
Most, if not all, of us humans are anthropocentric - and hence will tend to collectively value human output more than any other.
Which is maybe why we have collectively decided to call AI generated content ‘slop’! Don’t get me wrong - not that there isn’t any slop, on the contrary. But definitely not all of it is slop, and not for everyone.
Nevertheless, it seems then that the only measure of creative originality is discounting for AI generated content.
We already have a flood of tools attempting to do just that - detect AI generated content, post facto.
That isn’t deterministic enough. And even if it got to be, why is it human vs. AI. And why not human vs. the internet, or a book!
“There is no such thing as a new idea. It is impossible.” - Mark Twain
Going back to the study on Human vs. LLM creativity, researchers find that humans excel in creative writing tasks over LLMs because they are able to pre-emptively anticipate reader response, create character arcs with internal conflicts and provide deep, genuine novelty.
Outside of creative writing, the value add by a human writer varies by domain and is often measured by domain expertise itself.
What we are left with then is measuring the value of human contribution - or Agency.
The fact that a human provided intentionality and a theory of mind to an output, irrespective of the assistance they sought, is what needs to be measured, signalled, and be verifiable. Much like their domain expertise is.
Mark Twain continues: “We simply take a lot of old ideas and put them into a sort of mental kaleidoscope. We give them a turn and they make new and curious combinations. We keep on turning and making new combinations indefinitely; but they are the same old pieces of colored glass that have been in use through all the ages.”
And now that AI is more than capable of helping us with these combinations, in an instant, maybe the following quote is more relevant.
“There are no new ideas. There are only new ways of making them felt.” - Audre Lorde
I don’t have a recommendation for professional thinkers/writers/creators. What is definitely now possible is for anyone to take the most raw intentions, and turn them into professional grade valuable output by giving enough thought and attention, as and when required. Much like pruning a bonsai tree.
And while that happens, EditCred’s mission is to signal Human Agency aligned with AI.
C,
Team EditCred.