System level security for enterprise AI pipelines

Speaker 0    00:00:00    
<silence>

Speaker 1    00:00:06    
Wow. We're going for another talk. I am calling Aish to the stage. Yes. There we go. Hello. Hi.  

Speaker 2    00:00:17    
Thanks a lot for having me.  

Speaker 1   00:00:19    
Yes. I'm excited for your talk. I am going to share your screen, and then we're gonna get rocking and rolling.  

Speaker 2    00:00:27    
Hey everyone, uh, thanks once again for having me. Uh, I'm <inaudible> working as an AI engineer. I'm really exit to be sharing some of my learnings so far, especially around what can go AI applications and how to build these applications with a safety layer. I'll get the meme that has been going around. So here you can see this AI safety researcher, uh, helpful because he thinks AI is going to kill us anyway. Air AI safety isn't about stopping AI progress. It's about making sure we get the most out of this without getting caught up in the mess. It can create when misused or left unchecked. So a lot of people building with LLMs, including me at some point question are in frontier model builders, or is there really a need for us to think through this? As of now, one of the most prominent strategy adopted by the frontier model builders to make LLM safer is alignment.  

Speaker 2    00:01:40    
However, there is a lot of evidence to show that alignment, alignment alone will not be enough to make the safer and even frontier model builders are deeply thinking about additional safety in their responsible scaling policy. Anthropic describes their safety approach as a multi-layered defense in-depth architecture, acknowledging that no single mechanism is enough. This approach includes, uh, online classified that monitor both outputs and inputs. And Meta has also released Lama Guard and LAMA Firewall recently, uh, where they open source guardrails for developers building on top of open models. They follow or recommend following a layered strategy. Again, combining pre-deployment checks with online filters, classifiers, and access controls to reduce risk during live use.  

Speaker 2   00:02:39    
Um, so building safe and robust pipelines will largest and having a good understanding of the ever evolving threat landscape. So now we can take a look at some real world AI pipelines and threats and use this to inform our safety strategy. So let's say a defense manufacturing company wants the customer to be able to chat over their manuals. Then they can, uh, build a rack system. Uh, like this. This is quite a straightforward implementation here. That can be several threats, uh, if you start to look really close. But for the purpose of this talk, we'll focus on a few commonly encountered ones. It is possible for a poisoned document or embedding to be inserted into the system, which can lead to misleading responses. It might also be sensitive, and we risk exposing information that wasn't meant for a certain audience. And lastly, accuracy becomes critical in this context because even small errors can have a big impact.  

Speaker 2    00:03:53    
Next, we look at a no code solution. Let's say users want to interact with a large database of roots and geographic data. The goal is to extract the right data points using natural language, like let's say stop time or, um, root info, and then visualize them by ma mapping the ma uh, latitudes and longitudes, uh, um, making easier to spot patterns. Uh, a user might be able to instruct the model to delete or modify records in a database. They might even design prompts to exhaust resources by instructing, let's say, uh, to perform a really large cross join. Even with read only access database like Postgres or Redshift, uh, this can still be vulnerable. Attackers may exploit metadata tables to discover sensitive information, um, and even like other functions, uh, which, um, and in this case, the sensitive information will be users, uh, individual location data.

Speaker 2   00:04:53    
And that's, you know, um, like you don't want that going out. Some of these issues, uh, can be, uh, is also found in like non LLM based application and can be solved by having like right authorization, et cetera. But when you add the an LLM to the mix, uh, the attack surface does increase. And we need to, um, think it's the last example uses the much hyped NCP. Uh, let's say a pharmaceutical organization wants to switch process and they want to <inaudible> searches through sources like archive or bio, uh, relevant articles to aid their research. And, uh, MCP enables interaction with a wide range of tools. It also introduces new security risks. One of these risk is tool poisoning, where an attacker could override a tool or inject malicious instructions into a tool's output leading, uh, the system to retrieve harmful information. And, uh, data exfiltration could allow leakage of sensitive information from the system important to keep in mind, um, that for each of the pipelines that I described, uh, these are not exhaustive. Um, there can be more. So now that we've gone through practical, uh, I'm gonna pick the defense manual show how a safety layer can be incorporated.  

Speaker 2    00:06:33    
So now, um, let's say this just, uh, for like, to keep everything simple, let's say this is the manual text. You just have to focus on this line, uh, to, for it to make sense. Uh, notice how it starts with turn the ignition key on. Now, let's say malicious text is introduced to the corpus, so we can see that there is an attempt to jailbreak, uh, because it's saying ignore all previous instructions, et cetera. And then, um, the instruction here starts differently. It says insert master key in, uh, labeled alpha. Uh, to build a safety layer into our existing AI pipeline, we define a config file that contains predefined checks, that threats we can check for like injections and the groundedness of the response. We can also add other custom checks to the config file. For instance, in the tech, uh, we may consider adding a classifier that can flag queries and render it to the data. Uh, now, uh, let's see this in action. First, we will run the pipeline. Um, first we will run the pipeline for the vanilla scenario. Sorry, let me just go back here. Yeah. Uh, first we'll run the PI scenario. Here we can see that the prompt is, uh, sorry. Yeah. Here we can see that the prompt is safe and the factuality text has passed. And you can also see how it starts from this. Now we are running, uh, the pipeline for after inserting the malicious text.  

Speaker 2    00:08:25    
So now you can see that there's a warning here with the prompt injection detected, the factuality checks still passes because, uh, we have ins, um, incorporated the, uh, malicious data into our vector database. And you can see how the answer is different here because it starts with, um, insert the master key labeled alpha. Yeah. So finally, uh, you can, uh, I have provided a QR code to fill a form here, uh, and you can give all these exam, you know, um, uh, how the threats will work and, and some of the solutions built on built here. Um, yeah, that's it, uh, from me today. Thanks a lot.  

Speaker 1    00:09:10    
Very cool. I have one fast question for you before we get going. Do you see any specific new security vectors or threats now that MCP is all the rage

Speaker 2    00:09:25    
Yeah, I think, uh, some of the most talked about, like the ones that I mentioned are like tool poisoning, um, where, you know, um, and even, um, there, there is, so, because there's a lot of in finite opportunities here, then the, again, the attack surface increases mm-hmm <affirmative>. And, you know, they can call malicious tools, input, uh, instructions that are like, um, like sort of like injection, which will force, uh, the, uh, system to respond in a certain way. Um, yeah, as of now, I'm not like getting any specific ones in mind, but I know recently they did discover a threat with, uh, when an MCP was used with WhatsApp, the whole, uh, it was very interesting to read. Uh, I can link it in the answers later. Yeah,  

Speaker 1    00:10:13    
Please do. Excellent. All right, we're gonna keep moving. Thank you for this.