Evaluating AI and Human Performance

September 17, 2023

So how much does AI improve human performance?

The impact of AI on human performance varies depending on the task and the specific AI system being used. In general, AI can help humans to be more productive, efficient, and accurate. However, AI is not yet able to outperform humans on all tasks, and there are some tasks where AI can actually hinder human performance.

For example, a study by McKinsey found that AI could boost global productivity by 0.8% to 1.4% per year between 2018 and 2025. However, the study also found that AI could displace up to 800 million jobs by 2030.

What is the best method for obtaining such improvements?

The rise of large language models (LLMs) like ChatGPT has sparked much interest and debate around how artificial intelligence will impact human work. Two recent studies provide some intriguing early insights into LLMs’ effects on creativity and knowledge worker productivity.

In one study, researchers compared ChatGPT to human students on a creative divergent thinking task. On average, the AI chatbot outperformed humans in generating more creative and higher quality ideas. However, the very best human ideas still matched or exceeded ChatGPT’s. The study suggests AI can enhance average creativity, but uniquely human traits likely underpin the most highly creative contributions.

Another study looked at how consultants working for a management firm used ChatGPT on realistic business tasks. For analytic activities clearly within ChatGPT’s capabilities, the AI significantly boosted consultants’ productivity and work quality. Consultants completed over 12% more subtasks and 25% faster with AI aid. Response quality rose over 40%. However, for a task designed just beyond ChatGPT’s frontier, its use actually lowered performance and accuracy.

These initial results highlight AI’s potential to augment human skills, but also the need to thoughtfully navigate its limitations. As LLMs continue rapidly evolving, understanding where to best apply them will be crucial. The frontier between human and machine strengths is likely to keep shifting. Mastering human-AI collaboration and knowing when AI reaches its breaking point appear critical to unleashing benefits. More research is still required, but these studies provide early insights into AI’s emerging impacts on knowledge work.

There are a number of different ways to use AI to improve human performance. One common approach is to use AI to automate tasks that are currently performed by humans. This can free up humans to focus on more complex and creative tasks.

Another approach is to use AI to augment human capabilities. For example, AI can be used to provide humans with real-time feedback and assistance. This can help humans to improve their performance on a variety of tasks.

Are higher or lower performers helped more?

The impact of AI on human performance can vary depending on the individual’s skill level. In general, AI can help both high and low performers to improve their performance. However, the magnitude of the improvement may be greater for low performers.

Recent studies show some interesting findings regarding how AI impacts workers of different skill levels:

For example, a study by the University of California, Berkeley found that AI-powered tutoring could help students to improve their math scores by an average of 0.4 grade levels. However, the improvement was even greater for students who were initially struggling in math, with these students seeing an average improvement of 1.1 grade levels.
In a creativity study, AI seemed to boost the average performance more so than the peak creativity of top performers. This suggests AI may better augment ordinary creativity rather than enhance extreme genius.
However, the workflow study found that lower-performing consultants benefited more from AI assistance than higher performers. Those in the bottom half of skills had their work quality boosted over 40% with AI versus 17% for top performers. But AI still significantly improved outcomes for all.

So there are somewhat contradictory results on whether top versus average performers gain more from AI collaboration. The workflow study indicates AI particularly helps raise lower performers closer to top levels. But extreme creativity may be harder to augment. More research is needed, but it appears AI can enhance skills across ability levels in many real-world tasks, while truly exceptional creativity remains more uniquely human. Understanding how to best combine strengths of both humans and AI across the skills spectrum is an important area for ongoing study.

What is the best strategy?

The best strategy for using AI to improve human performance will vary depending on the specific task and the individual’s skill level. There are a few key strategies emerging for how humans can best collaborate with AI tools like large language models:

Centaur Approach - This involves strategically dividing tasks between the human and AI based on each entity’s strengths. Humans focus on high-level thinking and judgment, while delegating rote tasks to AI. The goal is to create an integrated human-AI team, like the mythical centaur creature.
Cyborg Approach - With this strategy, the human intricately meshes with the AI, collaborating together at a granular level. The human guides the AI, providing examples and corrections, while also benefiting from the AI’s capabilities. The cyborg integration allows for a tight human-AI synthesis.
Complimentary Collaboration - Here the human and AI each focus on what they excel at to complement one another. For creative tasks, the human generates novel ideas while the AI refines and expands on them. In analysis, the human provides oversight and validation while the AI crunches data.
Iterative Workflow - Humans can use AI to accelerate portions of workflows, enabling more time for uniquely human contributions. Quick AI-generated drafts, data processing, etc. gives humans more bandwidth to devote creativity and judgement.

The optimal strategy likely utilizes elements of all approaches. Key factors include assessing current AI capabilities, mapping workflows, and matching AI tools to suitable tasks. Continuously training both humans and AI will further enhance collaboration. But fundamentally, combining human ingenuity and judgement with AI’s scale and speed offers the greatest potential.

References