New highlights added 2024-01-18
question answering using GPT-4, with zero-shot prompts directly on SQL databases, achieves an accuracy of 16%. Notably, this accuracy increases to 54% when questions are posed over a Knowledge Graph representation of the enterprise SQL database (View Highlight)
While question answering systems have shown remarkable performance in several Text-to-SQL benchmarks [4, 2], such as Spider [18], WikiSQL[19], KaggleDBQA[11] their implications relating to enterprise SQL databases remain relatively obscure. 1 We argue that existing Question Answering and Text-to-SQL benchmarks, although valuable, are often misaligned with real-world enterprise settings:
- these benchmarks typically overlook complex database schemas representing enterprise domains, which likely comprise hundreds of tables, (View Highlight)
Low Question/Low Schema, knowledge graph accuracy was 71.1% while the SQL accuracy was 25.5% • HighQuestion/Low Schema, knowledge graph accuracy was 66.9% while the SQL accuracy was 37.4% • Low Question/High Schema, knowledge graph accuracy was 35.7% while the SQL accuracy was 0% • High Question/High Schema, knowledge graph accuracy was 38.7% while the SQL accuracy was 0% (View Highlight)