Reference Checking Techniques

RAG stands for Retrieval-Augmented Generation. It’s a technique that combines the power of LLMs with real-time access to external information sources. Instead of relying solely on what an AI model learned during training (which can quickly become outdated), RAG enables the model to retrieve relevant data from external databases, documents, or APIs—and then use that information to generate more accurate, context-aware responses. How does RAG work? 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗲: The system searches for the most relevant documents or data based on your query, using advanced search methods like semantic or vector search. 𝗔𝘂𝗴𝗺𝗲𝗻𝘁𝗮𝘁𝗶𝗼𝗻: Instead of just using the original question, RAG 𝗮𝘂𝗴𝗺𝗲𝗻𝘁𝘀 (enriches) the prompt by adding the retrieved information directly into the input for the AI model. This means the model doesn’t just rely on what it “remembers” from training—it now sees your question 𝘱𝘭𝘶𝘴 the latest, domain-specific context 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗲: The LLM takes the retrieved information and crafts a well-informed, natural language response. 𝗪𝗵𝘆 𝗱𝗼𝗲𝘀 𝗥𝗔𝗚 𝗺𝗮𝘁𝘁𝗲𝗿? Improves accuracy: By referencing up-to-date or proprietary data, RAG reduces outdated or incorrect answers. Context-aware: Responses are tailored using the latest information, not just what the model “remembers.” Reduces hallucinations: RAG helps prevent AI from making up facts by grounding answers in real sources. Example: Imagine asking an AI assistant, “What are the latest trends in renewable energy?” A traditional LLM might give you a general answer based on old data. With RAG, the model first searches for the most recent articles and reports, then synthesizes a response grounded in that up-to-date information. Illustration by Deepak Bhardwaj

ChunkRAG, a framework that enhances #RAG systems by evaluating and filtering retrieved information at the chunk level. Traditionally, RAG systems retrieve large amounts of the text of entire documents or lengthy portions thereof, assuming that it is likely that these lengthy fragments will contain the relevant information. Such systems very rarely examine the sections or paragraphs of the retrieved documents individually and, therefore, there is a strong likelihood that irrelevant or only partially related information will flow into the generation stage. The approach employs semantic chunking to divide documents into coherent sections and utilizes LLM-based relevance scoring to assess each chunk’s alignment with the user’s query. By filtering out less pertinent chunks before the generation phase, we significantly reduce hallucinations and improve factual accuracy. Experiments show that this method outperforms existing RAG models, achieving higher accuracy on tasks requiring precise information retrieval. This advancement enhances the reliability of RAG systems, making them particularly beneficial for applications like fact-checking and multi-hop reasoning. The figure shows that without chunk filtering (top), irrelevant information like other French cities is included in the response. The LLM-driven chunk filtering (bottom), however, removes unnecessary content, delivering the precise answer, "The capital of France is Paris." Access the complete research paper on ChunkRAG: https://lnkd.in/gmsnjTqm Here is my article that proves how semantic chunking improves your RAG systems: https://lnkd.in/gTXsjBQA

It is easy to criticize LLM hallucinations but Google researchers just made a major leap toward solving them for statistical data. In the DataGemma paper (Sep ’24), they teach LLMs when to ask an external source instead of guessing. They propose two approaches: Retrieval interleaved generation (RIG) - the model injects natural language queries into its output, triggering fact retrieval from Data Commons. Retrieval augmented generation (RAG) - the model pulls full data tables into its context and reasons over them with a long-context LLM. The results are impressive: (1) RIG improved statistical accuracy from 5–17% to ~58% (2) RAG hit ~99% accuracy on direct citations (with some inference errors still remaining) (3) Users strongly preferred the new responses over baseline answers. As LLMs increasingly rely on external tools, teaching them "when to ask" may become as important as "how to answer." Paper https://lnkd.in/gaKY_VNE

I just came across a groundbreaking paper titled "SAGE: A Framework of Precise Retrieval for RAG" by researchers from Tsinghua University (Jintao Zhang, Guoliang Li, and Jinyang Su). This innovative framework addresses critical limitations in current Retrieval-Augmented Generation (RAG) systems. As many of us working with RAG systems know, they often fail not because of LLM limitations, but due to inaccurate information retrieval. SAGE tackles two major problems: 1. Semantic Segmentation: Current RAG methods typically segment corpus without considering semantics, making it difficult to find relevant context. SAGE trains a lightweight model that quickly segments text into semantically complete chunks, ensuring retrieved information is coherent and relevant. 2. Dynamic Chunk Selection: Instead of retrieving a fixed number of chunks (which can include irrelevant information), SAGE employs a gradient-based selection algorithm that dynamically chooses chunks based on relevance score patterns. The system identifies significant drops in relevance scores to determine the optimal cutoff point. 3. Self-Feedback Mechanism: SAGE incorporates LLM self-assessment to evaluate whether retrieved chunks are excessive or insufficient, then adjusts accordingly. The technical implementation is fascinating - their segmentation model uses an embedding model with an MLP structure that evaluates sentence pairs using both direct embeddings and feature augmentation (calculating differences and products between embeddings). The training leverages Wikipedia's paragraph structure to learn semantic relationships between sentences. The results are impressive: SAGE outperforms baselines by 61.25% in QA quality while achieving 49.41% better cost efficiency by reducing token consumption during LLM inference. This research offers valuable insights for anyone working with RAG systems and demonstrates how precise retrieval can significantly enhance performance while reducing costs.

Uber processes millions of invoices globally – different formats, currencies, tax codes, and languages. Traditional rule-based OCR pipelines just don’t scale for that level of variability. Interesting to see how Uber solved this using a two-stage GenAI approach: 1. LLM-based field extraction: zero-shot parsing of key fields like vendor, total amount, tax ID. 2. Post-processing logic: country-specific rules (e.g. GST validation for India). The system improves itself through feedback. But this is where data labeling becomes critical. Without accurately labeled fields and validation, the model can hallucinate or misinterpret formats, especially for low-resource languages or unusual layouts. Labeling ensures: 1. Feedback loop quality 2. Accuracy tracking by field 3. Reliable onboarding of new invoice types It’s a solid example of blending GenAI with traditional ML workflows and domain logic for real-world scale. Worth a read 👇 https://lnkd.in/gFqMS9zW #GenAI #DataScience #UberAI #DocumentUnderstanding #LLM #AIInOperations #DataLabeling #InvoiceAutomation

𝗜𝗳 𝗟𝗟𝗠𝘀 𝗮𝗿𝗲 𝘀𝗼 𝗳𝗹𝘂𝗲𝗻𝘁, 𝘄𝗵𝘆 𝗱𝗼 𝘁𝗵𝗲𝘆 𝘀𝘁𝗶𝗹𝗹 𝘀𝘁𝘂𝗺𝗯𝗹𝗲 𝗼𝗻 𝗿𝘂𝗹𝗲-𝗵𝗲𝗮𝘃𝘆 𝗱𝗲𝗰𝗶𝘀𝗶𝗼𝗻𝘀 𝘄𝗵𝗲𝗿𝗲 𝗰𝗼𝗿𝗿𝗲𝗰𝘁𝗻𝗲𝘀𝘀 𝗮𝗻𝗱 𝘁𝗿𝗮𝗰𝗲𝗮𝗯𝗶𝗹𝗶𝘁𝘆 𝗺𝗮𝘁𝘁𝗲𝗿? They fail because they’re optimized for producing plausible text, not executing formal rules: they can miss hidden constraints, "average out" exceptions, struggle to consistently apply multi-step logic, and rarely produce auditable reasoning paths that prove which rule or policy drove a decision. Neurosymbolic AI addresses this by combining neural models (LLMs/NNs) for understanding messy language and data, with symbolic systems (rules, logic, knowledge graphs) for deterministic reasoning, constraints, and verifiable decision trails. https://lnkd.in/gg3knpFc Common architecture patterns for Neurosymbolic AI with LLMs. 𝟭. 𝗟𝗟𝗠 𝗮𝘀 𝗽𝗮𝗿𝘀𝗲𝗿 -> 𝘀𝘆𝗺𝗯𝗼𝗹𝗶𝗰 𝗲𝘅𝗲𝗰𝘂𝘁𝗼𝗿 : A user asks “Are these 12 vendors eligible under our procurement policy?” and the LLM extracts structured facts (vendor type, spend, region, exceptions) while a rules/logic engine deterministically computes eligibility and returns the decision + which rules fired. 𝟮. 𝗟𝗟𝗠 𝗮𝘀 𝗽𝗹𝗮𝗻𝗻𝗲𝗿 -> 𝗰𝗼𝗻𝘀𝘁𝗿𝗮𝗶𝗻𝗲𝗱 𝘁𝗼𝗼𝗹 𝗲𝘅𝗲𝗰𝘂𝘁𝗶𝗼𝗻 : A change-management agent proposes a rollout plan, but every step is validated against hard constraints (maintenance windows, approvals, dependency ordering) and blocked/rewritten if any constraint fails before any tool call executes. 𝟯. 𝗟𝗟𝗠 + 𝗸𝗻𝗼𝘄𝗹𝗲𝗱𝗴𝗲 𝗴𝗿𝗮𝗽𝗵 𝗿𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 : A support agent answers "Why did customer X’s software fail after release Y?" by traversing a knowledge graph (customer -> services -> incidents -> deployments -> config changes), then uses symbolic path evidence to justify a multi-hop explanation. 𝟰. 𝗣𝗿𝗼𝗴𝗿𝗮𝗺-𝗼𝗳-𝘁𝗵𝗼𝘂𝗴𝗵𝘁 -> 𝗲𝘅𝗲𝗰𝘂𝘁𝗲 𝗱𝗲𝘁𝗲𝗿𝗺𝗶𝗻𝗶𝘀𝘁𝗶𝗰𝗮𝗹𝗹𝘆 : A finance ops assistant converts "reconcile these statements and compute variance drivers" into executable code/queries (SQL/Python), runs them in a sandbox, and returns computed results rather than "reasoning in text."

In our daily discussions about generative AI, the fear of AI 'hallucinating'—or fabricating information—often surfaces. This conversation, however, opens the door to an exciting question: Could AI surpass human accuracy in identifying truths? Enter a groundbreaking study by #Google #DeepMind and #Stanford researchers, which introduces a novel framework called SAFE. Tested across approximately 16,000 facts, SAFE demonstrated superhuman performance, aligning with human evaluators 72% of the time and besting them in 76% of contested cases, all while being 20 times more cost-effective than traditional methods. The essence of this methodology lies in two pivotal steps. Initially, the LongFact prompt set, crafted using GPT-4, targets the comprehensive assessment of long-form content's factuality over 38 varied topics. Then, the SAFE framework takes this base further by meticulously breaking down responses into individual facts and validating each through targeted Google Search queries. The process unfolds across four critical stages: 1. 𝗣𝗿𝗼𝗺𝗽𝘁 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻 𝘄𝗶𝘁𝗵 𝗟𝗼𝗻𝗴𝗙𝗮𝗰𝘁: Crafting varied, fact-seeking prompts to elicit detailed LLM responses. 2. 𝗗𝗲𝗰𝗼𝗺𝗽𝗼𝘀𝗶𝘁𝗶𝗼𝗻 𝗶𝗻𝘁𝗼 𝗜𝗻𝗱𝗶𝘃𝗶𝗱𝘂𝗮𝗹 𝗙𝗮𝗰𝘁𝘀: Segmenting these responses into distinct facts for precise evaluation. 3. 𝗙𝗮𝗰𝘁 𝗩𝗲𝗿𝗶𝗳𝗶𝗰𝗮𝘁𝗶𝗼𝗻 𝘃𝗶𝗮 𝗚𝗼𝗼𝗴𝗹𝗲 𝗦𝗲𝗮𝗿𝗰𝗵: Using LLMs to formulate and dispatch queries, checking each fact's accuracy against search results. 4. 𝗜𝘁𝗲𝗿𝗮𝘁𝗶𝘃𝗲 𝗥𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴 𝗮𝗻𝗱 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻: Applying a multi-step reasoning process to assess the support level for each fact. This innovative approach doesn't just mark a leap in evaluating LLM-generated content's factuality; it also paves the way for more trustworthy AI applications in countless fields. For a deep dive into this fascinating study, including access to the LongFact prompts and the SAFE framework, visit: https://lnkd.in/eVr4rz-u Find the full paper here: https://lnkd.in/eSjZ5Tn9 #GenAI #LLM #Hallucination #FactChecking #DeepMind #Stanford #Google #SAFE #LongFact

𝐕𝐞𝐜𝐭𝐨𝐫 𝐬𝐞𝐚𝐫𝐜𝐡 𝐟𝐢𝐧𝐝𝐬 𝐬𝐢𝐦𝐢𝐥𝐚𝐫𝐢𝐭𝐲. 𝐆𝐫𝐚𝐩𝐡𝐑𝐀𝐆 𝐟𝐢𝐧𝐝𝐬 𝐦𝐞𝐚𝐧𝐢𝐧𝐠. Traditional RAG retrieves relevant chunks, but it can miss relationships connecting people, concepts, events and decisions. GraphRAG adds that missing layer by combining LLMs with knowledge graphs. It starts with entity extraction, turning unstructured text into people, places, concepts, and structured elements. Ontology mapping aligns those entities to a domain model, so information from different sources follows consistent definitions. The system builds a knowledge graph by connecting entities through relationships. The real value appears during retrieval: ↳ Hybrid retrieval combines vector similarity with graph traversal. ↳ Graph traversal uncovers multi-hop relationships across documents. ↳ Context enrichment adds connected facts around the original query. ↳ Evidence ranking prioritizes information by relevance, confidence, and relationship strength. ↳ Grounded generation helps the LLM produce answers supported by traceable context. This makes GraphRAG useful when questions require more than keyword matching. Think enterprise search, fraud detection, compliance, research, supply chains, and decision-support systems. The goal is not to retrieve more information. It is to retrieve the right information, understand how it connects, and provide enough evidence to explain the answer. GraphRAG = Knowledge + Relationships + Context → Better Answers 𝐖𝐡𝐞𝐫𝐞 𝐜𝐨𝐮𝐥𝐝 𝐆𝐫𝐚𝐩𝐡𝐑𝐀𝐆 𝐜𝐫𝐞𝐚𝐭𝐞 𝐭𝐡𝐞 𝐦𝐨𝐬𝐭 𝐯𝐚𝐥𝐮𝐞 𝐢𝐧 𝐲𝐨𝐮𝐫 𝐨𝐫𝐠𝐚𝐧𝐢𝐳𝐚𝐭𝐢𝐨𝐧?

Day 16/30 of LLMs/SLMs - Retrieval-Augmented Generation (RAG) Large Language Models are powerful, but they have a fixed memory. They cannot know anything that happened after their training cut-off, and they struggle with facts that were never part of their dataset. When they lack the right information, they guess. The result is fluent but unreliable text — the hallmark of hallucination. Retrieval-Augmented Generation (RAG) fixes that by giving models a way to look up information before they answer. RAG is best understood as a three-stage pipeline, and LangChain has become the de facto standard framework for building each stage efficiently. 𝐈𝐧𝐠𝐞𝐬𝐭𝐢𝐨𝐧 𝐚𝐧𝐝 𝐈𝐧𝐝𝐞𝐱𝐢𝐧𝐠 You start by collecting and preparing your documents. LangChain’s loaders handle PDFs, web pages, CSVs, and APIs. These documents are then split into smaller, semantically meaningful chunks and converted into embeddings using models like OpenAI’s text-embedding-3-small, SentenceTransformers, or InstructorXL. Those embeddings are stored in a vector database such as FAISS, Pinecone, Weaviate, or Chroma, which lets you perform similarity search later. 𝐑𝐞𝐭𝐫𝐢𝐞𝐯𝐚𝐥 When a query arrives, LangChain converts it into an embedding and searches the vector store for the most relevant documents. Retrieval strategies vary — basic similarity search, maximal marginal relevance (MMR) to diversify context, or hybrid retrieval that mixes semantic and keyword search. The retrieved text chunks are then added to the prompt as contextual grounding. 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐨𝐧 The LLM receives the augmented prompt containing both the user query and retrieved passages. It synthesizes an answer based on that external knowledge. LangChain manages prompt templates, context formatting, and memory across queries, making the process modular and repeatable. 𝐖𝐡𝐲 𝐑𝐀𝐆 𝐌𝐚𝐭𝐭𝐞𝐫𝐬 RAG fundamentally improves factual accuracy and trust. On benchmarks such as Natural Questions and TriviaQA, a base model like LLaMA 2-13B might achieve 45 F1, while RAG-augmented versions reach 65–70 F1 - matching much larger and costlier models. 𝐆𝐞𝐭𝐭𝐢𝐧𝐠 𝐒𝐭𝐚𝐫𝐭𝐞𝐝 𝐰𝐢𝐭𝐡 𝐋𝐚𝐧𝐠𝐂𝐡𝐚𝐢𝐧 𝐑𝐀𝐆 If you want to experiment, LangChain makes it approachable. A minimal prototype takes fewer than 20 lines of code. Here’s a good progression 👉 Start with the LangChain tutorial: https://lnkd.in/gUpHpkKT 👉 Add a vector store: Try Chroma for local experiments or Pinecone for scalable hosting. 👉 Experiment with retrieval methods: compare similarity search vs. MMR. 👉 Integrate your own data: ingest PDFs, database exports, or web content. 👉 Deploy a chain: connect your retriever, model, and prompt template into a single workflow. Tune in tomorrow for more SLM/LLMs deep dives. -- 🚶➡️ To learn more about LLMs/SLMs, follow me - Karun! ♻️ Share so others can learn, and you can build your LinkedIn presence!

Microsoft AI Introduces Claimify: A Novel LLM-based Claim-Extraction Method that Outperforms Prior Solutions to Produce More Accurate, Comprehensive, and Substantiated Claims from LLM Outputs Microsoft AI Research has recently developed Claimify, an advanced claim-extraction method based on LLMs, specifically designed to enhance accuracy, comprehensiveness, and context-awareness in extracting claims from LLM outputs. Claimify addresses the limitations of existing methods by explicitly dealing with ambiguity. Unlike other approaches, it identifies sentences with multiple possible interpretations and only proceeds with claim extraction when the intended meaning is clearly determined within the given context. This careful approach ensures higher accuracy and reliability, particularly benefiting subsequent fact-checking efforts. From a technical standpoint, Claimify employs a structured pipeline comprising three key stages: Selection, Disambiguation, and Decomposition. During the Selection stage, Claimify leverages LLMs to identify sentences that contain verifiable information, filtering out those without factual content. In the Disambiguation stage, it uniquely focuses on detecting and resolving ambiguities, such as unclear references or multiple plausible interpretations. Claims are extracted only if ambiguities can be confidently resolved. The final stage, Decomposition, involves converting each clarified sentence into precise, context-independent claims. This structured process enhances both the accuracy and completeness of the resulting claims....... Read full article: https://lnkd.in/guZwNVg9 Paper: https://lnkd.in/gjrUpFYQ Technical details: https://lnkd.in/g439K-Bq Microsoft Microsoft Research Dasha Metropolitansky Jonathan Larson Download our OPEN SOURCE AI MAGAZINE REPORT 2025: https://pxl.to/yv08dj