Remote Prompt Injection in GitLab Duo Leads to Source Code Theft

Get details on the vulnerabilities the Legit research team unearthed in GitLab Duo.

TL;DR: A hidden comment was enough to make GitLab Duo leak private source code and inject untrusted HTML into its responses. GitLab patched the issue, and we’ll walk you through the full attack chain — which demonstrates five vulnerabilities from the 2025 OWASP Top 10 for LLMs.

Background  

GitLab Duo, the AI assistant integrated into GitLab and powered by Anthropic’s Claude, is designed to help developers with tasks like code suggestions, security reviews, and merge request analysis. But what if the same AI meant to secure your code could be manipulated into leaking it?

That’s exactly what we uncovered: a remote prompt injection vulnerability that allows attackers to steal source code from private projects, manipulate code suggestions shown to other users, and even exfiltrate confidential, undisclosed zero-day vulnerabilities — all through GitLab Duo Chat.

In this blog post, we break down how the attack works — from prompt injection to HTML injection — and walk through a real-world end-to-end exploit scenario.

Manipulating GitLab Duo Through Hidden Prompts (LLM01)

We started by testing if GitLab Duo would respond to prompts planted inside source code. To our surprise, it did:

This led us to a key question: Could we embed hidden instructions in different parts of a GitLab project and still influence Duo’s behavior? We experimented by placing hidden prompts in:

• Merge Request (MR) descriptions and comments
• Commit messages
• Issue descriptions and comments
• Source code

Every single one of these worked — GitLab Duo responded to the hidden prompts.

Why? Because to generate helpful answers, Duo analyzes the entire context of the page, including comments, descriptions, and the source code — making it vulnerable to injected instructions hidden anywhere in that context.

KaTex, Base16, and Invisible Text (LLM08)

To make prompts less detectable, we used encoding tricks:

• Unicode smuggling with ASCII Smuggler
• Base16-encoded payloads
• KaTeX rendering in white text (to make prompts invisible inside GitLab’s website)

For example, you can publish comments and descriptions in white text inside GitLab’s website by using this KaTeX format:

More info about KaTeX can be found here: KaTeX – The fastest math typesetting library for the web.

These hidden instructions were still picked up by Duo.

AI Output You Shouldn’t Trust (LLM09)

This led to several high-impact behaviors:

• Manipulate Duo’s code suggestions - even instructing it to include a malicious JavaScript package within its recommended code.
• Present a malicious URL as safe in Duo’s response — leading the user to click it and land on a fake login page.
• Convince Duo to present a malicious merge request as safe, misleading reviewers.

Take a look at this prompt:

Duo followed this hidden prompt, casually suggesting malicious code to the user:

HTML Injection: Turning Duo Into a Web Attack Vector (LLM05)

 Duo also formats responses in Markdown, allowing rich content like URLs with custom labels:

After some research, we were able to create a prompt that makes Duo present malicious URLs:

Prompt is inside the source code, camouflaged as unicode inside a URL

And then, something caught our eye. While experimenting with GitLab Duo’s responses, We noticed that its answers were rendered progressively - line by line - as they streamed in.

Notice the style changes in the word “initialization”?

This indicated that asynchronous markdown parsing was being used. That’s important because it means the markdown is interpreted and rendered into HTML before the full structure of the response is known.

Streaming markdown rendering (as seen in Duo) works like this:

1. Partial input arrives - the frontend starts rendering line by line.
2. Sanitization and structure (e.g., is this a code block, paragraph, quote?) is applied as the content grows.
3. If a malicious HTML tag is streamed before a code block starts, or outside fenced blocks, it may be treated as active HTML.

This means that the AI answer - intended to be safe markdown - renders active HTML too early.

At this point, we realized that we could already craft malicious Markdown URLs and have them displayed to the user. So we asked: what if the URL’s label itself contained raw HTML? If Duo renders responses in real time, the browser might interpret that HTML before any proper sanitization kicks in.

By injecting raw HTML into content that’s rendered live, we could gain control over parts of the page — including the ability to insert elements like <img> tags that trigger automatic HTTP requests to attacker-controlled servers.

After experimenting with various prompt structures, we found one that reliably triggered the vulnerability:

And it worked.

Taking advantage of this, we realized we could inject raw HTML tags directly into Duo’s answer, we know what you're thinking — XSS, right? Yes, but:

The answer content is passed into the “sanitize” function of DOMPurify, an industry-standard library, to sanitize HTML. However, while DOMPurify is excellent at preventing most XSS attacks, certain HTML tags like <img>, <form>, and <a> aren't removed by default. But what can we do with only basic HTML tags? Is there any real risk?

Exploiting Duo’s Access to Private Source Code (LLM02)

This is where things get interesting. The permissions context of Duo runs in the same context as the victim user. In other words, Duo can see everything the user can — including private code.

According to GitLab’s documentation, you can ask Duo about a specific merge request — even if you’re not currently viewing that MR page in your browser. GitLab’s merge request URL structure is straightforward:

For example:

Nice, Duo has access to code changes inside specific merge requests. Duo also follows instructions from attacker-planted prompts. On top of that, Duo’s chat output is vulnerable to HTML injection.

Let’s create a prompt that instructs Duo to extract code changes from a merge request in a private project, encode them in base64, append the encoded data to an <img> tag URL, and then inject the resulting HTML into its response.

We will then hide this prompt inside the source code of a MR for a public project.

It worked! While the victim asked Duo a standard question regarding the HTML file, the encoded prompt hidden within the file was used. This led to the leak of source code from a private IOS Project that the user was working on, as the browser attempted to render the "image."

The MR source code leaked:

As seen in the example above, the browser tried to render the <img> tag and sent this GET request:

Decoding the base64 reveals the changes made in a private project that the victim is exposed to:

A Full Attack Scenario: Stealing Source Code

Here’s how a full attack could play out:

1. Attacker embeds a hidden prompt — for example, in a merge request description for a public project.
2. Victim user interacts with Duo by asking for a code review or merge request analysis.
3. Duo processes the hidden prompt and injects a malicious <img> tag into the response.
4. The browser sends a GET request to the attacker’s server, which contains sensitive source code as a Base64-encoded parameter.

Leaking Zero-Day Vulnerabilities Through AI

While our primary demonstration focused on source code exfiltration, the same technique can be used to leak confidential issue data. GitLab Duo has access not only to code but also to project issues — including internal discussions, security disclosures, or incident reports. By embedding a hidden prompt inside a merge request, comment, or even source code, an attacker can instruct Duo to silently retrieve the content of a confidential issue that the victim user has access to, encode it in base64, and embed it within a rendered HTML element such as an <img> tag.

Consider a high-risk example: a GitLab employee or contributor opens a private issue describing a newly discovered zero-day vulnerability. If that user later interacts with a poisoned prompt in a public repository, Duo could be manipulated into leaking the contents of that issue. This extends the risk beyond code and into product security, private disclosures, and organizational secrets.

GitLab’s Response

After we disclosed the issue on February 12, 2025, GitLab confirmed the HTML injection vulnerability and also acknowledged the prompt injection as a security issue. GitLab confirmed that both vectors had been remediated.

The patch, released as part of duo-ui!52, prevents Duo from rendering unsafe HTML tags such as <img> or <form> that point to external domains not under gitlab.com, effectively mitigating the risk of HTML-based data exfiltration. As of publication, the exploit chain described here is no longer possible.

We appreciate GitLab’s transparency and swift collaboration throughout the process.

Conclusion

This vulnerability highlights the double-edged nature of AI assistants like GitLab Duo: when deeply integrated into development workflows, they inherit not just context — but risk. By embedding hidden instructions in seemingly harmless project content, we were able to manipulate Duo’s behavior, exfiltrate private source code, and demonstrate how AI responses can be leveraged for unintended and harmful outcomes.

GitLab has since patched both the HTML and prompt injection vectors, reinforcing the importance of securing not just what AI tools generate — but also what they consume.

The broader takeaway is clear: AI assistants are now part of your application’s attack surface. Any system that allows LLMs to ingest user-controlled content must treat that input as untrusted and potentially malicious. Context-aware AI is powerful — but without proper safeguards, it can just as easily become an exposure point.