Mnemosyne
Open Access DepositedUsing Large Language Models to Convert Documents to Knowledge Graphs to Check for Completeness and Consistency
Downloadable Content
It is difficult for people to maintain large complex documents like town-ship ordinances or technical manuals. The review process is slow and costly and is prone to human error. This lets internal inconsistencies and incom- pleteness make their way into documents. That can trigger legal disputes and ultimately financial losses. Large Language Models (LLMs) would seem to be an obvious solution to this. However, all LLMs have a finite context window and therefore cannot process an entire large document. This Praxis details the design, creation, use, and evaluation of Mnemosyne. It is a system built to overcome the LLM context window lim- itation. Mnemosyne makes use of LLMs to convert large documents into attributed knowledge graphs (KGs). It does this by breaking the document into small chunks. It has a multi stage pipeline that uses a two tiered mem- ory architecture. Through this design the system can incrementally build and iteratively refine the knowledge graph. In doing so, it creates a KG that faithfully represents the full semantics of the document. Mnemosyne’s was tested using the ordinances of Pennsylvania townships. These test included controlled injection of related material and unrelated material. Another test involved the removal of a section. The results demonstrate that Mnemosyne can construct useful knowledgegraphs from township ordinances. These knowledge graphs retain the ability to trace back to the source documents. Experiments showed that the system can be used to identify inconsistencies. These appear as irregularities in the knowledge graph’s structure. A key contribution of this research is a two part definition of documentcompleteness. The experiments show that a knowledge graph is effective at detecting internal completeness issues. For example, undefined terms and broken cross references. This is done through simple structural analysis. However, the research also establishes that external completeness cannot be detected by internal analysis alone, it requires external domain knowl- edge. This Praxis validates that an LLM approach to building and refining an attributed knowledge graph provides a practical and robust methodology for verifying the internal consistency and structural completeness of large documents.
- All rights reserved
Notice to Authors
If you are the author of this work and you have any questions about the information on this page, please use the Contact form to get in touch with us.