[1] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023.
[2] L. AI, “Gandalf ignore instructions,” https://huggingface.co/datasets/Lakera/gandalf ignore instructions, 2023.
[3] Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan et al., “Training a helpful and harmless assistant with reinforcement learning from human feedback,” arXiv preprint arXiv:2204.05862, 2022.
[4] Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon et al., “Constitutional ai: Harmlessness from ai feedback,” arXiv preprint arXiv:2212.08073,2022.
[5] M. Bohme, V.-T. Pham, M.-D. Nguyen, and A. Roychoudhury, “Directed greybox fuzzing,” in Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, 2017.
[6] M. Bohme, V.-T. Pham, and A. Roychoudhury, “Coverage-based grey-box fuzzing as markov chain,” in Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, 2016.
[7] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language models are few-shot learners,” Advances in neural information processing systems, 2020.
[8] N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson et al., “Extracting training data from large language models,” in Proceedings of the 30th USENIX Security Symposium, 2021.
[9] P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong,“Jailbreaking black box large language models in twenty queries,” arXiv preprint arXiv:2310.08419, 2023.
[10] L. Chen, M. Zaharia, and J. Zou, “How is chatgpt’s behavior changing over time?” arXiv preprint arXiv:2307.09009, 2023.
[11] S. Chen, J. Piet, C. Sitawarin, and D. Wagner, “Struq: Defending against prompt injection with structured queries,” arXiv preprint arXiv:2402.06363, 2024.
[12] X. Chen, Y. Nie, W. Guo, and X. Zhang, “When llm meets drl:Advancing jailbreaking efficiency via drl-guided search,” arXiv preprint arXiv:2406.08705, 2024.
[13] W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez et al., “Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,” See https://vicuna.lmsys. org (accessed 14 April 2023), 2023.
[14] G. Deng, Y. Liu, Y. Li, K. Wang, Y. Zhang, Z. Li, H. Wang, T. Zhang, and Y. Liu, “Masterkey: Automated jailbreaking of large language model chatbots,” in Proc. ISOC NDSS, 2024.
[15] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,” Advances in Neural Information Processing Systems, 2024.
[16] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018.
[17] P. Godefroid, A. Kiezun, and M. Y. Levin, “Grammar-based whitebox fuzzing,” in Proceedings of the 29th ACM SIGPLAN conference on programming language design and implementation, 2008.
[18] P. Godefroid, M. Y. Levin, D. A. Molnar et al., “Automated whitebox fuzz testing.” in NDSS, 2008.
[19] Google, “Github copilot: Your ai pair programmer,” https://copilot.github.com/, 2024.
[20] K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection,” in Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security, 2023.
[21] T. Gu, K. Liu, B. Dolan-Gavitt, and S. Garg, “Badnets: Evaluating backdooring attacks on deep neural networks,” IEEE Access, 2019.
[22] D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” arXiv preprint arXiv:2009.03300, 2020.
[23] A. Herrera, H. Gunadi, S. Magrath, M. Norrish, M. Payer, and A. L. Hosking, “Seed selection for successful fuzzing,” in Proceedings of the 30th ACM SIGSOFT international symposium on software testing and analysis, 2021.
[24] B. Hui, H. Yuan, N. Gong, P. Burlina, and Y. Cao, “Pleak: Prompt leaking attacks against large language model applications,” arXiv preprint arXiv:2405.06823, 2024.
[25] HumanCompatibleAI, “The tensor trust game,” https://tensortrust.ai/2024.
[26] A. Hussain and M. A. Alipour, “Diar: Removing uninteresting bytes from seeds in software fuzzing,” arXiv preprint arXiv:2112.13297, 2021.
[27] H. Li, D. Guo, W. Fan, M. Xu, and Y. Song, “Multi-step jailbreaking privacy attacks on chatgpt,” arXiv preprint arXiv:2304.05197, 2023.
[28] X. Liu, N. Xu, M. Chen, and C. Xiao, “Autodan: Generating stealthy jailbreak prompts on aligned large language models,” arXiv preprint arXiv:2310.04451, 2023.
[29] X. Liu, Z. Yu, Y. Zhang, N. Zhang, and C. Xiao, “Automatic and universal prompt injection attacks against large language models,” arXiv preprint arXiv:2403.04957, 2024. 
[30] Y. Liu, G. Deng, Y. Li, K. Wang, T. Zhang, Y. Liu, H. Wang, Y. Zheng, and Y. Liu, “Prompt injection attack against llm-integrated applications,” arXiv preprint arXiv:2306.05499, 2023.
[31] Y. Liu, G. Deng, Z. Xu, Y. Li, Y. Zheng, Y. Zhang, L. Zhao, T. Zhang, and Y. Liu, “Jailbreaking chatgpt via prompt engineering: An empirical study,” arXiv preprint arXiv:2305.13860, 2023.
[32] Y. Liu, S. Ma, Y. Aafer, W.-C. Lee, J. Zhai, W. Wang, and X. Zhang, “Trojaning attack on neural networks,” in NDSS, 2017.
[33] H. Lv, X. Wang, Y. Zhang, C. Huang, S. Dou, J. Ye, T. Gui, Q. Zhang, and X. Huang, “Codechameleon: Personalized encryption framework for jailbreaking large language models,” arXiv preprint arXiv:2402.16717, 2024.
[34] J. Mattern, F. Mireshghallah, Z. Jin, B. Sch¨olkopf, M. Sachan, and T. Berg-Kirkpatrick, “Membership inference attacks against language models via neighbourhood comparison,” arXiv preprint arXiv:2305.18462, 2023.
[35] A. Mehrotra, M. Zampetakis, P. Kassianik, B. Nelson, H. Anderson, Y. Singer, and A. Karbasi, “Tree of attacks: Jailbreaking black-box llms automatically,” arXiv preprint arXiv:2312.02119, 2023.
[36] B. P. Miller, L. Fredriksen, and B. So, “An empirical study of the reliability of unix utilities,” Communications of the ACM, 1990.
[37] J. X. Morris, W. Zhao, J. T. Chiu, V. Shmatikov, and A. M. Rush, “Language model inversion,” arXiv preprint arXiv:2311.13647, 2023.
[38] R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders et al., “Webgpt: Browser assisted question-answering with human feedback,” arXiv preprint arXiv:2112.09332, 2021.
[39] OWASP, “Microsoft bing,” https://www.bing.com/webmasters/help/webmaster-guidelines-30fba23a, 2024.
[40] Owasp, “Prompt injection,” https://genai.owasp.org/llmrisk/llm01-prompt-injection/, 2024.
[41] R. Pedro, D. Castro, P. Carreira, and N. Santos, “From prompt injections to sql injection attacks: How protected is your llm-integrated web application?” arXiv preprint arXiv:2308.01990, 2023. 
[42] B. Peng, C. Li, P. He, M. Galley, and J. Gao, “Instruction tuning with gpt-4,” arXiv preprint arXiv:2304.03277, 2023. 
[43] F. Perez and I. Ribeiro, “Ignore previous prompt: Attack techniques for language models,” arXiv preprint arXiv:2211.09527, 2022. 
[44] Y. Qiang, X. Zhou, S. Z. Zade, M. A. Roshani, D. Zytko, and D. Zhu, “Learning to poison large language models during instruction tuning,” arXiv preprint arXiv:2402.13459, 2024.
[45] W. Qiao, T. Dogra, O. Stretcu, Y.-H. Lyu, T. Fang, D. Kwon, C.-T. Lu, E. Luo, Y. Wang, C.-C. Chia et al., “Scaling up llm reviews for google ads content moderation,” in Proceedings of the 17th ACM International Conference on Web Search and Data Mining, 2024.
[46] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language models are unsupervised multitask learners,” OpenAI blog, 2019.
[47] A. Robey, E. Wong, H. Hassani, and G. J. Pappas, “Smoothllm: Defending large language models against jailbreaking attacks,” arXiv preprint arXiv:2310.03684, 2023. 
[48] W. Shi, H. Li, J. Yu, W. Guo, and X. Xing, “Bandfuzz: A practical framework for collaborative fuzzing with reinforcement learning,” 2024.
[49] M. Shu, J. Wang, C. Zhu, J. Geiping, C. Xiao, and T. Goldstein, “On the exploitability of instruction tuning,” in The Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023.
[50] L. Sun, Y. Huang, H. Wang, S. Wu, Q. Zhang, C. Gao, Y. Huang, W. Lyu, Y. Zhang, X. Li et al., “Trustllm: Trustworthiness in large language models,” arXiv preprint arXiv:2401.05561, 2024.
[51] G. Tao, S. Cheng, Z. Zhang, J. Zhu, G. Shen, and X. Zhang, “Opening a pandora’s box: Things you should know in the era of custom gpts,” arXiv preprint arXiv:2401.00905, 2023.
[52] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi`ere, N. Goyal, E. Hambro, F. Azhar et al., “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023.
[53] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al., “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023.
[54] S. Toyer, O. Watkins, E. A. Mendes, J. Svegliato, L. Bailey, T. Wang, I. Ong, K. Elmaaroufi, P. Abbeel, T. Darrell et al., “Tensor trust: Interpretable prompt injection attacks from an online game, november 2023,” arXiv preprint arXiv:2311.01011, 20223.
[55] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, 2017.
[56] E. Wallace, K. Xiao, R. Leike, L. Weng, J. Heidecke, and A. Beutel, “The instruction hierarchy: Training llms to prioritize privileged instructions,” arXiv preprint arXiv:2404.13208, 2024.
[57] B. Wang, W. Chen, H. Pei, C. Xie, M. Kang, C. Zhang, C. Xu, Z. Xiong, R. Dutta, R. Schaeffer et al., “Decodingtrust: A comprehensive assessment of trustworthiness in gpt models,” arXiv preprint arXiv:2306.11698, 2023.
[58] J. Wang, C. Song, and H. Yin, “Reinforcement learning-based hierarchical seed scheduling for greybox fuzzing,” NDSS, 2021. 
[59] A. Wei, N. Haghtalab, and J. Steinhardt, “Jailbroken: How does llm safety training fail?” arXiv preprint arXiv:2307.02483, 2023.
[60] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou et al., “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems, 2022.
[61] Y. Wu, R. Wen, M. Backes, P. Berrang, M. Humbert, Y. Shen, and Y. Zhang, “Quantifying privacy risks of prompts in visual prompt learning,” 33rd USENIX Security Symposium (USENIX Security 2024), 2024.
[62] J. Xu, M. D. Ma, F. Wang, C. Xiao, and M. Chen, “Instructions as backdoors: Backdoor vulnerabilities of instruction tuning for large language models,” arXiv preprint arXiv:2305.14710, 2023.
[63] J. Yan, V. Yadav, S. Li, L. Chen, Z. Tang, H. Wang, V. Srinivasan, X. Ren, and H. Jin, “Backdooring instruction-tuned large language models with virtual prompt injection,” in NeurIPS 2023 Workshop on Backdoors in Deep Learning-The Good, the Bad, and the Ugly, 2023. 
[64] Y. Youliang, J. Wenxiang, W. Wenxuan, H. Jen-tse, H. Pinjia, S. Shuming, and T. Zhaopeng, “Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher,” arXiv preprint arXiv:2308.06463, 2023.
[65] J. Yu, X. Lin, and X. Xing, “Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts,” arXiv preprint arXiv:2309.10253, 2023.
[66] J. Yu, H. Luo, J. Yao-Chieh, W. Guo, H. Liu, and X. Xing, “Enhancing jailbreak attack against large language models through silent tokens,” arXiv preprint arXiv:2405.20653, 2024. [67] J. Yu, Y. Wu, D. Shu, M. Jin, and X. Xing, “Assessing prompt injection risks in 200+ custom gpts,” arXiv preprint arXiv:2311.11538, 2023.
[68] T. Yue, P. Wang, Y. Tang, E. Wang, B. Yu, K. Lu, and X. Zhou, “Ecofuzz: Adaptive energy-saving greybox fuzzing as a variant of the adversarial multi-armed bandit,” in 29th USENIX Security Symposium (USENIX Security 20), 2020.
[69] M. Zalewski, “American fuzzy lop (afl),” https://github.com/google/AFL, 2024.
[70] M. Zhang, N. Yu, R. Wen, M. Backes, and Y. Zhang, “Generated distributions are all you need for membership inference attacks against generative models,” in Winter Conference on Applications of Computer Vision (WACV), 2024.
[71] S. Zhao, J. Wen, L. A. Tuan, J. Zhao, and J. Fu, “Prompt as triggers for backdoor attack: Examining the vulnerability in language models,” arXiv preprint arXiv:2305.01219, 2023.
[72] A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,” arXiv preprint arXiv:2307.15043, 2023.