Consultants using GenAI for data science work perform up to 49% better in certain tasks beyond their existing capabilities, according to new research.
The research, by the BCG Henderson Institute (Boston Consulting Group’s think tank) in collaboration with BCG X and Boston University, explored what happens when people use AI to complete tasks beyond their existing capabilities, rather than just using it to improve their performance within the context of their current skillset.
In the experiment, 480 BCG consultants completed short tasks that mimic the daily activities of a data scientist: writing code to merge and clean datasets, building a predictive model for sports investing using analytics best practices (e.g. machine learning), and validating ChatGPT-generated statistical analyses. These tasks were significantly challenging and could not be fully automated by the GenAI tool (Enterprise ChatGPT with GPT-4 and its Advanced Data Analysis Feature). To help evaluate participants’ performance, their results were compared with those of 44 BCG data scientists who worked without the assistance of GenAI.
The study aimed to answer two questions:
- Should professionals use Generative AI for tasks beyond their area of expertise, particularly those at the edge of Generative AI's capabilities?
- How can companies best enable their workforce to take on new tasks in collaboration with Generative AI?
When using GenAI, the consultants in the study were able to expand their aptitude for new tasks. Even when they had no experience in coding or statistics, consultants with access to GenAI tools were able to write code, appropriately apply machine learning models and correct erroneous statistical processes, with biggest skill expansion observed in coding.
Participants in the study who used GenAI achieved an average score in the coding test equivalent to 86% of the benchmark set by data scientists, a 49% improvement over participants not using GenAI. The GenAI-augmented group also finished the task roughly 10% faster than the data scientists.
In addition, GenAI-augmented consultants with moderate coding experience performed 10 to 20 percentage points better on all three tasks than their peers who self-identified as novices, even when coding was not involved. In fact, those with moderate coding experience were fully on par with data scientists for two of the three tasks - one of which had zero coding involved.
In contrast, predictive analytics was the task which the GenAI-augmented consultant was least likely to perform on par with a data scientist, regardless of previous experience in coding or statistics, as neither the consultants nor the GenAI tool had a high level of proficiency in this area. As a result, participants with access to GenAI were more likely to be led astray by the tools than their non-augmented counterparts.
“Our findings suggest that GenAI-augmented workers can adeptly handle new tasks beyond their existing skills in fields that are in the tool’s capabilities,” said Dan Sack, a BCG managing director and partner, and coauthor of the study. “Executives need to be ready for this future, redefining expertise and identifying the skills to grow and retain their talent for the long term.”
As the research acknowledges, however, while GenAI-augmented workers were “reskilled” in the sense that that they gained new capabilities that were beyond what either the human or GenAI could do on their own, they were not intrinsically reskilled, because “doing” with GenAI does not inherently mean “learning to do” – although with more repetition and a learning-focused intention, learning would happen.