Written by Rijul Singla,
Christ (Deemed to be) University, Delhi NCR
September 2026
Introduction
The use of artificial intelligence for summarizing cases, conducting legal research, preparing drafts, and finding precedents has become quite common in Indian legal practice. But a generative AI system does not find the legal authorities in the same way as a traditional database system; instead, it predicts the text, which could turn out to be fluent, confident and altogether wrong. In 2026, the Supreme Court of India saw this issue not as a theoretical matter but as the bedrock of an order issued by a tribunal and another order issued by a customs authority. The rise of these new technologies presents an interesting challenge that how to assign responsibility in the situation when the AI system predicts a false legal authority and a human actor depend on it?
AI Hallucinations in Legal Research
An AI hallucination is not always an outlandish response. For instance, in legal research, it may come in the guise of a totally made-up ruling, a wrong citation, a made-up quote, or even an actual ruling with a paragraph or proposition that the ruling does not have. In this last type of scenario, the hallucination could be hard to identify due to the existence of the case at all. Since large language models work by generating text from statistical correlations without checking authorities against a reliable database, the text is often legally convincing but utterly false.
The NCLT and NCLAT: When Hallucinated Authorities Enter Adjudication
The supreme court’s decision in Pooja Ramesh Singh v. Jammu & Kashmir Bank Ltd[1] provides a significant illustration. The order was passed by the National Company Law Tribunal (NCLT) when an application for insolvency was filed against a corporate guarantor and later confirmed by the National Company Law Appellate Tribunal (NCLAT) over a year ago. It was established by the Supreme Court that six sources which the NCLT had depended on either never existed at all or included paragraphs which never appeared in the quoted judgements. Indeed, the bank’s own counsel did not refer to these authorities; and the bogus was not spotted even in the Appellate level. It was the appellant’s counsel that finally brought it to the attention of the Supreme Court. The Supreme Court ruled that a ruling relying on fabricated or hallucinated evidence in form of precedent cannot be justified as a legitimate use of judicial power. The both orders were quashed, the insolvency case was remitted for revaluation and the Bar Council of India was directed to study the bigger issue. Therefore, the judgment treats verification as an institutional requirement rather than an individual precaution.
Vijay Ghanshyam Gadiya: Hallucinated Authorities Beyond the Courts
A second, larger illustration came two months later with an executive order. The Additional Commissioner of Customs at Surat had imposed a penalty of ₹425,27,99,100 on the diamond trader for allegedly mis declaring natural diamonds as lab-grown stones and the Gujarat High Court had rejected the challenge to the order in Vijay Ghanshyam Gadiya v. Union of India[2]. The Supreme Court was presented with evidence that some case law and articles cited in the decision did not exist, some had incorrect citations, and some were court cases that supported the arguments they were believed to have made but were not cited in the decision. A Bench of Justices Dipankar Datta and Sheel Nagu held that relying on such questionable material was doomed to result in the cancellation of the penalty order and its confirmation by the High Court and it remanded the case for a fresh adjudication to another officer. The combination of the two rulings reveals that hallucination isn’t restricted to lawyers using generative AI in an uninformed way to create documents. It already features in the tribunal’s reasoning, and in executive adjudication, wherever research is performed using the help of AI without the incorporation of a verification procedure.
The Institutional Backdrop
In November 2025, the Supreme Court’s Centre for Research and Planning had released its White Paper on AI and Judiciary[3], warning hallucination and the insertion of legal material made up of AI as a major risk in the process of judicial adoption of AI and urging for a mandatory human check on all AI outputs. On that basis, the Supreme Court’s AI Committee issued the Draft Regulations for Use of Artificial Intelligence in Courts[4] for public consultation in June, 2026. both human primacy and human control are put as central elements in governing AI under the proposed Regulations, with the provision of safeguards in applying AI within the judicial system. So, whether it’s the regulatory or judicial path, both are heading towards the same conclusion that AI can help, but it can’t be relied upon to verify its own work.
Whose responsibility should it be?
Responsibility should follow control and reliance, not the source of the technological error. There is no professional obligation on an AI system, and no adjudicatory power over it; it’s the human who puts it into a filing or into an order who has a professional obligation and who has the adjudicatory power, and so it is that the human bears responsibility for not acting with the judgment they are supposed to be exercising. This is a problem for advocates and litigators because citation-checking is not something they can afford to take lightly. Counsel should verify a judgment, that a citation is correct, that the quoted passage is indeed in the judgment and that it supports the proposition being advanced. Delegating the research is not the same as delegating the responsibility as none of this duty is discharged by pointing to a research assistant or a junior associate or a chatbot to indicate the source of an error. The response must also be proportionate that a mistake by AI that is discovered and corrected easily is not the same as a deliberate misquotation, and there should be no hostility towards a good faith use of a legal research tool where there is a mistake. Judges and adjudicating authorities, if anything, have a more urgent obligation: the order has their institutional authority. Pooja Ramesh Singh is a reminder that the requirement for verification does not come to an end just because the faulty material has been extracted from the research of the tribunal, not from the counsel at the bar. If an authority is a crucial part of the argument, or if it is hard to find by normal sources, it needs to be checked against an official court record or a reliable law report before it is cited.
Embedding Verification into the Legal Workflow
Any viable answer must be more than a vague “be careful” response. At a minimum, AI legal research involves five steps that are verifying the existence of cited judgment on an official repository; reading the citation in context; testing for support of cited proposition; determining if cited precedent remains good law; and ensuring a human review to confirm authority. These checks are most important for the worst class of error as a real case (and a fabricated quotation or a skewed holding) because this type of hallucination would not exist if the case were fake. A disclaimer, such as “AI may make errors,” does not meet any of this. The filing is a statement to the court; the order is an act of law power. Both call for reasonable verification and an independent application of mind, not a warning label. When you say “Human in the loop”, that has to be the human who physically tests the material, not just the human who saw it on the screen before he/she signed it off. The second, lesser risk is accuracy, alongside confidentiality. Whether the output is accurate or not, submitting pleadings and privileged communications, or personal information into an AI tool to expedite research can create a governance challenge. It is time to take the same serious approach to data retention and authorised tool policies as one would to citation accuracy in courts, chambers and government departments using AI tools.
Conclusion
The lesson from the cases of 2026 is not that AI does not have a role to play in India’s legal adjudication process, but that it cannot take over from humans when it comes to verification, logic, and accountability. According to Pooja Ramesh Singh & Vijay Ghanshyam Gadiya, accountability follows control and reliance but not the technology behind the mistake. The Bar, the Bench, or any executive authority cannot transfer their accountability to the technology on which they rely. A machine takes seconds to create a citation, but it takes an individual expert to determine whether the citation is true.
References
[1] Pooja Ramesh Singh v. Jammu & Kashmir Bank Ltd., 2026 INSC 668 (India July 2, 2026), https://api.sci.gov.in/supremecourt/2025/52338/52338_2025_5_1501_71939_Judgement_02-Jul-2026.pdf
[2] Vijay Ghanshyam Gadiya v. Union of India, 2026 INSC 947 (India Sept. 2, 2026), https://www.supremecourtcases.com/vijay-ghanshyam-gadiya-v-union-of-india-and-another/
[3] Centre for Research and Planning, Supreme Court of India, White Paper on Artificial Intelligence and Judiciary (Nov. 2025), https://cdnbbsr.s3waas.gov.in/s3ec0490f1f4972d133619a60c30f3559e/uploads/2025/11/2025112244.pdf
[4] Supreme Court of India, Draft Regulations for Use of Artificial Intelligence in Courts, 2026 (June 3, 2026), https://www.sci.gov.in/notices-and-circulars/