There’s plenty of buzz about how AI is reshaping enterprise software at a fundamental level. There’s enhanced decision-making capabilities, automated processes, streamlined operations—all so exciting, yet there often is a key ingredient missing.
Without a human touch, AI can go wrong and be annoying or even harmful.
- McDonald’s recently pulled a drive-thru order automation system after people posted videos hilariously shrieking as the assistant won’t stop adding hundreds of McNuggets or bacon to McFlurries.
- Google Gemini suggested a potentially poisonous recipe that a Reddit user thinks could have killed him with botulism.
- Waymo’s autonomous vehicles were recently honking at each other throughout the night, keeping San Francisco residents up (and very cranky).
Ensuring Success with the Human-Centered Approach
To avoid gaffes and create a service that is not only business viable and technically feasible but will also be well-adopted and received, we need deep empathy for the people who will use the service. Only with a holistic understanding of the user’s true needs, attitudes, and motivations can we tackle the best experience and ethical implications.
To do this, we employ our Human-Centered Approach that considers the three key areas:
- Desirability: Who are the people using the product? What are their day-to-day needs, attitudes, and motivations?
- Feasibility: What are the underlying technologies? How will the solution be supported? What is the ideal enterprise architecture?
- Viability: How will the solution provide business value? What about in the long run?
When thoroughly addressing all three, we arrive at the sweet spot for innovation. This is a front-loaded investment that ensures much better long-term success.
Case Study: ML-Driven Student Advising
In a university’s MBA program, advisors need to provide highly personalized guidance to help students align their academic choices with their career goals and personal aspirations. About 50% of the students struggled to select courses that fit their needs. The university consulted the SAP AppHaus to find a solution that gave advisors ML-based insights to boost advice confidence and reveal novel career pathways.
Through user research, our team of SAP AppHaus designers found that the students trusted “evidence-based” guidance from advisors and other students, and the advisors wanted to give these data-driven insights but needed a more holistic view of each student.
We sought to streamline the connection between student goal setting and coursework with a tool that tracked students’ activities and aspirations, then allowed advisors to see their profiles as well as relevant course recommendations. For this information to be reliable, students and advisors needed to know where it was coming from and why, and the information had to be unbiased.
On the business side, the management wanted the students to succeed and for the university to have a positive, innovative public image. The risk of bias came into play again. Similar AI projects at other universities have faced public scrutiny, such as the University of Texas at Austin’s ML system for admissions decisions in a Ph.D. program. Earlier this year, a study by AERA Open uncovered that AI models disproportionately predict failure for students from underrepresented backgrounds, negatively impacting fair college admissions.
Our solution would also need to account for the university’s technical landscape. We interviewed the technical users, and our architect and developers analyzed the appropriate data, technologies, and systems needed so that the solution was well-engineered, stable, and easy to maintain.
We co-designed the solution and validated the architecture with iterations to ensure we met the requirements of desirability, feasibility, and viability. The result was a successful student goals and course recommendation tool that:
- Gave advisors visibility on the holistic student’s goals, interests, and aspirations
- Brought about a richer advising experience enhanced by ML recommendations and insights
- Provided transparency and rationale behind why each course was recommended (e.g., this obscure class suits your interests and prepares you well for these careers you’re interested in)
Conclusion
The case study illustrates how a human-centered approach transforms AI initiatives into powerful tools that benefit users. As an SAP AppHaus Partner and SAP Gold Partner, Mindset maintains the principles of understanding user needs, technical capabilities, and business goals to create enterprise solutions that resonate with the users they serve.
However, our journey doesn’t end here. As AI capabilities continue to evolve, so too must our approach. It is essential to remain vigilant against new biases, actively seek user feedback, and adapt to new risks with mitigations that put people first. End users will remain critical in navigating the ethical landscape of AI.
As we look to the future, Mindset champions a design philosophy that ensures the human touch is not an afterthought but a guiding principle. Only then can we unlock AI’s true potential, steering clear of mishaps like accidentally ordering hundreds of nuggets, noisy honking into the night, and inadvertent bias, while instead crafting products that sit perfectly within the sweet spot of innovation.
