At Mindset, we’ve been working hard on understanding how generative AI systems can provide value for SAP customers. The most common question we see, and the biggest challenge we face, is how those systems can use the rich data and process model that SAP provides in order to serve and delight users while offering productivity benefits to businesses. I’ll outline three ways that these systems can interact with SAP, and why you might consider one over the other or a combination of approaches.
Generate Embeddings and Use Search or Retrieval Augmented Generation (RAG)
The go-to approach for integrating business data into generative AI systems is retrieval augmented generation (RAG). This approach is incredibly powerful for making available large amounts of text data to AI systems. It involves breaking the data up into chunks, creating embeddings for those chunks, and then enabling search across those embeddings, matching a question or description to the most closely related chunks by comparing embeddings. In other words, search.
Of course, the process of generating embeddings is based on text, not structured data or database records. So step 1 of generating embeddings based on SAP data is to convert that data into a pre-defined text format.
The drawback of this approach is that the text representation won’t capture the richness of SAP data, because of choices that need to be made upfront such as:
- What objects and fields to include – e.g. which fields in the sales order header to include in the representation?
- What relationships to include – e.g. Do we include sales order items along with the headers? Deliveries? Scheduling agreements? Material Movements? Deliveries?
- The exact structure of the text representation, including field names
As such, this approach is very powerful for answering questions of types and in areas that are anticipated by the design of the tool but can struggle with answering questions that fall outside of the design space. RAG and search are also not as useful for summarizing information and are not appropriate for use cases that require calculation, aggregation of data, or complex queries that span multiple types of business objects.
Use the Generative AI System to Query SAP APIs
Another approach that we have found very effective is to use generative AI systems to drive API calls to an SAP system. We use the generative AI model to interpret the context and based on the topic or question to determine what API should be called. Once the API is chosen, we can provide the API’s metadata document to the generative AI system and have it generate the actual query for us. When built properly, systems like this can also handle and correct errors. This type of system underpins our context-aware AI assistant demo application.
The advantage of systems like these is that they can intelligently leverage the full scope of functionality of the SAP system via its APIs. At the moment we recommend keeping these systems read-only, but there is nothing stopping us from enabling the ability to transact in the system as well as just reading it. And there are more advantages:
- Because OData services enable associative queries (especially OData version 4), some very expressive queries can be generated.
- The data is real-time, so we don’t have to worry about keeping a search index or database up to date.
- We can execute these queries as the logged-in user, so there is no concern about authorization issues.
This approach isn’t perfect though. It will struggle to generate very complex queries, and it is inherently limited by the APIs that are available and the fields that are available in those APIs. Additionally, if calculations or aggregations are required but are not directly supported by the available APIs, this system will still fail to accurately execute those functions. Generative AI systems just are not very good at counting things!
Create a Database with SAP Data and Allow Generative AI System to Query with SQL
So how do we deal with that calculation and aggregation problem? Well, SQL is very good for most types of queries, aggregations, and calculations, so the natural solution would be to allow a generative AI system to build SQL queries to run against a schema that includes all the relevant SAP data. This is a common generative AI implementation approach, and is the third approach we think should be deployed to leverage SAP data from generative AI systems.
Most generative AI models are actually quite good at generating SQL to answer questions or gather information, so deploying generative AI models into such a system is relatively straightforward. That said, SAP data models tend to be very complex, whether they are actually in your SAP system or have been extracted into a data warehouse. Therefore, it may be required to do some fine-tuning or provide examples in context (few-shot learning) when requesting that the system generate queries against SAP data. Fortunately, both of these approaches have proven very effective.
On the other hand, it is highly recommended to run these systems against a database that is not used for other business-critical operations, since queries will error out and the model will write queries with poor performance. Setting up such a database, populating it, and keeping it up to date is a challenge. Further, there are security considerations, since SAP authorizations are not enforced at the database level, so a parallel authorization framework would need to be imposed if there are any security concerns.
This review explains the practical ways Mindset has explored and used to leverage your valuable SAP systems from generative AI tools and applications. Mindset runs generative AI discovery and solutioning workshops regularly, so if you’re interested in running one of these workshops at your company, just reach out.
