RQ4: Detecting quality issues
Detection depends on the artifact, available oracle, language, and deployment context. Outcomes from heterogeneous evaluations support conditional selection; they do not establish a universal method ranking.
| Family | Applicable when | Operational value | Cost and limitation |
|---|---|---|---|
| Execution and resource measurement | Code can run with tests, a reference behavior, or a representative workload | Reveals behavioral failures, runtime, or memory use | Requires dependencies, inputs, and an oracle; finite testing cannot prove equivalence |
| Static, rule, metadata, and overlap analysis | Parsers, rules, API maps, provenance, or comparison sets exist | Flags encoded patterns without running the artifact | Coverage depends on rules, language support, thresholds, and metadata |
| General LLM evaluation | A rubric and capable contextual evaluator are available | Supports qualitative judgment and triage where fixed oracles are absent | Sensitive to evaluator, prompt, order, and context; requires calibration and inference cost |
| Task-specific learned scoring | In-domain labels or a learnable proxy exist | Enables versioned batch scoring | Label construction and training cost; transfer can fail under domain shift |
| Human or hybrid review | Ambiguous findings require contextual judgment | Helps construct labels and validate automated flags | Expert effort and disagreement limit throughput; most reviewed evidence concerns research annotation |
Generated-code detection
| Method | Path in taxonomy | Papers |
|---|---|---|
| Test-Based Execution Analysis | Dynamic Analysis > Test-Based Execution Analysis | 43 |
| Resource and Performance Measurement | Dynamic Analysis > Resource and Performance Measurement | 3 |
| Rule-Based Detection | Static Analysis > Rule-Based Detection | 27 |
| Human Review and Manual Inspection | Static Analysis > Human Review and Manual Inspection | 14 |
| LLM-Based Evaluation | Model-Based Detection > LLM-Based Evaluation | 10 |
| Task-Specific Learned Detection | Model-Based Detection > Task-Specific Learned Detection | 3 |
Select a paper count to open its supporting papers. Counts are multi-label literature memberships. Download the method definitions and paper–method memberships to trace each entry.
Training-data detection
| Method | Path in taxonomy | Papers |
|---|---|---|
| Execution-Based Validation | Dynamic Analysis > Execution-Based Validation | 3 |
| Syntax, Rule, Metadata, and Overlap Checks | Static and Corpus Audit > Syntax, Rule, Metadata, and Overlap Checks | 25 |
| Human-in-the-Loop Review | Static and Corpus Audit > Human-in-the-Loop Review | 8 |
| Provenance and Version-History Audit | Static and Corpus Audit > Provenance and Version-History Audit | 1 |
| Model-Membership and Attribution Audit | Static and Corpus Audit > Model-Membership and Attribution Audit | 2 |
| LLM-Based Dataset Screening | Model-Based Detection > LLM-Based Dataset Screening | 13 |
| Task-Specific Learned Classifiers | Model-Based Detection > Task-Specific Learned Classifiers | 5 |
Select a paper count to open its supporting papers. Counts are multi-label literature memberships. Download the method definitions and paper–method memberships to trace each entry.
Automation evidence and residual gaps
The 21-row automation crosswalk identifies a direct automated detection exemplar and a direct automated mitigation exemplar for each of eight code headings and 13 data headings. They can address different subtypes. The result establishes existence at a broad-heading level; it does not establish complete subtype coverage, paired detection and mitigation, or deployment readiness.
Human work remains in constructing rules, tests, labels, prompts, and thresholds, and in adjudicating context-sensitive findings. Important gaps include output-side copyright and privacy leakage, semantic contamination, repository-scale structural quality, and end-to-end CI/CD evaluation. RQ3 relationship families do not constitute an additional issue-detection denominator.
Supporting papers
Browse 94 distinct papers by detection method, separated by the artifact being checked. Expand a method to see its taxonomy path and supporting papers, or select its paper count in the tables above. Membership is multi-label: a paper can appear under several methods. The paper–method memberships retain the exact assignments.
Generated-code detection
Test-Based Execution Analysis 43 papers
Dynamic Analysis > Test-Based Execution Analysis
Copilot Evaluation
An Empirical Evaluation of GitHub Copilot’s Code Suggestions
CodeHalu
CodeHalu: Investigating Code Hallucinations in LLMs via Execution-based Verification
Mercury
Mercury: A Code Efficiency Benchmark for Code Large Language Models
HallTrigger
Code Hallucination
Lost at C
Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants
The Counterfeit Conundrum
The Counterfeit Conundrum: Can Code Language Models Grasp the Nuances of Their Incorrect Generations?
Bugs in LLM-generated Code
Bugs in Large Language Models Generated Code: An Empirical Stud
GitHub Copilot, Amazon CodeWhisperer, ChatGPT
Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT
ChatGPT Code Quality
No Need to Lift a Finger Anymore? Assessing the Quality of Code Generation by ChatGPT
LLM-generated Code Efficiency
On Evaluating the Efficiency of Source Code Generated by LLMs
Syntactic Robustness
Syntactic Robustness for LLM-based Code Generation
Bias Unveiled
Bias Unveiled: Investigating Social Bias in LLM-Generated Code
From Effectiveness to Efficiency
From Effectiveness to Efficiency: Comparative Evaluation of Code Generated by LCGMs for Bilingual Programming Questions
ENAMEL
How Efficient is LLM-Generated Code? A Rigorous & High-Standard Benchmark
PTMs
Comparing Robustness Against Adversarial Attacks in Code Generation: LLM-Generated vs. Human-Written
Artificial-Intelligence Generated Code Considered Harmful
Artificial-Intelligence Generated Code Considered Harmful: A Road Map for Secure and High-Quality Code Generation
Python Tests Quality
Quality Assessment of Python Tests Generated by Large Language Models
REAL
Training Language Models to Generate Quality Code with Program Analysis Feedback
Quality In, Quality Out
Quality In, Quality Out: Investigating Training Data's Role in AI Code Generation
Security and Quality in LLM-Generated Code
Security and Quality in LLM-Generated Code: A Multi-Language, Multi-Model Analysis
SwallowCode
Rewriting Pre-Training Data Boosts LLM Performance in Math and Code
Refining ChatGPT-Generated Code
Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues
ReCode
ReCode: Updating Code API Knowledge with Reinforcement Learning
Data-efficient Fine-tuning
Data-efficient LLM Fine-tuning for Code Generation
CRPE
CRPE: Expanding The Reasoning Capability of Large Language Model for Code Generation
DeepSeek-Coder
DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence
Beyond Correctness
Beyond Correctness: Benchmarking Multi-dimensional Code Generation for Large Language Models
Codex
Evaluating Large Language Models Trained on Code
Path Planning Evaluation
Assessing LLM code generation quality through path planning tasks
Synthetic Data Generation
Synthetic Data Generation Using Large Language Models: Advances in Text and Code
Unseen Horizons
Unseen Horizons: Unveiling the Real Capability of LLM Code Generation Beyond the Familiar
Code Generation Survey
A Survey on Large Language Models for Code Generation
DataRecipe
DataRecipe --- How to Cook the Data for CodeLLM?
ClassEval
Evaluating Large Language Models in Class-Level Code Generation
RealSec-Bench
RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories
ShortCoder
ShortCoder: Knowledge-Augmented Syntax Optimization for Token-Efficient Code GenerationPreprint
APIKG4SYN
Framework-Aware Code Generation with API Knowledge Graph-Constructed Data: A Study on HarmonyOS
MultiCodeIF
A hierarchical and evolvable benchmark for fine-grained code instruction following with multi-turn feedback
Beyond Functional Correctness
Beyond functional correctness: Investigating coding style inconsistencies in large language models
Adadec
Adadec: Uncertainty-guided adaptive decoding for llm-based code generation
AllianceCoder
What to retrieve for effective retrieval-augmented code generation? an empirical study and beyond
RustEvo^ 2
RustEvo^ 2: An Evolving Benchmark for API Evolution in LLM-based Rust Code Generation
COFFE
COFFE: A Code Efficiency Benchmark for Code Generation
Resource and Performance Measurement 3 papers
Dynamic Analysis > Resource and Performance Measurement
LLM-generated Code Efficiency
On Evaluating the Efficiency of Source Code Generated by LLMs
ENAMEL
How Efficient is LLM-Generated Code? A Rigorous & High-Standard Benchmark
COFFE
COFFE: A Code Efficiency Benchmark for Code Generation
Rule-Based Detection 27 papers
Static Analysis > Rule-Based Detection
LLMs Meet Library Evolution
LLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-based Code Completion
Copilot Security
Is GitHub’s Copilot as Bad as Humans at Introducing Vulnerabilities in Code?
Copilot Evaluation
An Empirical Evaluation of GitHub Copilot’s Code Suggestions
SStuBs
Large Language Models and Simple, Stupid Bugs
package hallucinations
We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMs
HallTrigger
Code Hallucination
Large Language Models for Code
Large Language Models for Code: Security Hardening and Adversarial Testing
Purple Llama CYBERSECEVAL
Purple Llama CYBERSECEVAL: A Secure Coding Benchmark for Language Models
Lost at C
Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants
AI Assistants Security
Do Users Write More Insecure Code with AI Assistants?
Bugs in LLM-generated Code
Bugs in Large Language Models Generated Code: An Empirical Stud
GitHub Copilot, Amazon CodeWhisperer, ChatGPT
Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT
CloudAPIBench
On Mitigating Code LLM Hallucinations with API Documentation
FairCoder
FairCoder: Evaluating Social Bias of LLMs in Code Generation
DeVAIC
DeVAIC: A Tool for Security Assessment of AI-generated Code
Codequal Analyzer
Improving LLM-Generated Code Quality with GRPO
Artificial-Intelligence Generated Code Considered Harmful
Artificial-Intelligence Generated Code Considered Harmful: A Road Map for Secure and High-Quality Code Generation
REAL
Training Language Models to Generate Quality Code with Program Analysis Feedback
Quality In, Quality Out
Quality In, Quality Out: Investigating Training Data's Role in AI Code Generation
Security and Quality in LLM-Generated Code
Security and Quality in LLM-Generated Code: A Multi-Language, Multi-Model Analysis
Refining ChatGPT-Generated Code
Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues
Seed-Coder
Seed-Coder: Let the Code Model Curate Data for Itself
CodeSmellEval
How Propense Are Large Language Models at Producing Code Smells? A Benchmarking Study
RPG
Rethinking Repetition Problems of LLMs in Code Generation
Path Planning Evaluation
Assessing LLM code generation quality through path planning tasks
MultiCodeIF
A hierarchical and evolvable benchmark for fine-grained code instruction following with multi-turn feedback
Code Copycat Conundrum
Code Copycat Conundrum: Demystifying Repetition in LLM-based Code Generation
Human Review and Manual Inspection 14 papers
Static Analysis > Human Review and Manual Inspection
Copilot Security
Is GitHub’s Copilot as Bad as Humans at Introducing Vulnerabilities in Code?
HalluCode
Exploring and Evaluating Hallucinations in LLM-Powered Code Generation
HallTrigger
Code Hallucination
Lost at C
Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants
AI Assistants Security
Do Users Write More Insecure Code with AI Assistants?
The Counterfeit Conundrum
The Counterfeit Conundrum: Can Code Language Models Grasp the Nuances of Their Incorrect Generations?
Bugs in LLM-generated Code
Bugs in Large Language Models Generated Code: An Empirical Stud
GitHub Copilot, Amazon CodeWhisperer, ChatGPT
Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT
Syntactic Robustness
Syntactic Robustness for LLM-based Code Generation
Security and Quality in LLM-Generated Code
Security and Quality in LLM-Generated Code: A Multi-Language, Multi-Model Analysis
Synthetic Data Generation
Synthetic Data Generation Using Large Language Models: Advances in Text and Code
Code Generation Survey
A Survey on Large Language Models for Code Generation
Beyond Functional Correctness
Beyond functional correctness: Investigating coding style inconsistencies in large language models
Llm Hallucinations in Practical Code Generation
Llm hallucinations in practical code generation: Phenomena, mechanism, and mitigation
LLM-Based Evaluation 10 papers
Model-Based Detection > LLM-Based Evaluation
The Counterfeit Conundrum
The Counterfeit Conundrum: Can Code Language Models Grasp the Nuances of Their Incorrect Generations?
CodeMirage
CodeMirage: Hallucinations in Code Generated by Large Language Models
Unveiling Inefficiencies in LLM-Generated Code
Unveiling Inefficiencies in LLM-Generated Code: Toward a Comprehensive Taxonomy
CoQuIR
CoQuIR: A Comprehensive Benchmark for Code Quality-Aware Information Retrieval
Seed-Coder
Seed-Coder: Let the Code Model Curate Data for Itself
Generated Code Diversity
Is Functional Correctness Enough to Evaluate Code Language Models? Exploring Diversity of Generated Codes
CODEJUDGE
CODEJUDGE : Evaluating Code Generation with Large Language Models
RealSec-Bench
RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories
MultiCodeIF
A hierarchical and evolvable benchmark for fine-grained code instruction following with multi-turn feedback
RobGen
A Preliminary Study on the Robustness of Code Generation by Large Language Models
Task-Specific Learned Detection 3 papers
Model-Based Detection > Task-Specific Learned Detection
CodeMirage
CodeMirage: Hallucinations in Code Generated by Large Language Models
DeSec
Decoding Secret Memorization in Code LLMs Through Token-Level Characterization
CIDRe
CIDRe: A Reference-Free Multi-Aspect Criterion for Code Comment Quality Measurement
Training-data detection
Execution-Based Validation 3 papers
Dynamic Analysis > Execution-Based Validation
Qwen2
Qwen2 Technical Report
TeleChat
Technical Report of TeleChat2, TeleChat2.5 and T1
Kimi K2
Kimi K2: Open Agentic Intelligence
Syntax, Rule, Metadata, and Overlap Checks 25 papers
Static and Corpus Audit > Syntax, Rule, Metadata, and Overlap Checks
Less is More
Less is More: On the Importance of Data Quality for Unit Test Generation
Qwen
Qwen Technical Report
Qwen2
Qwen2 Technical Report
Quality In, Quality Out
Quality In, Quality Out: Investigating Training Data's Role in AI Code Generation
SwallowCode
Rewriting Pre-Training Data Boosts LLM Performance in Math and Code
TeleChat
Technical Report of TeleChat2, TeleChat2.5 and T1
Data-efficient Fine-tuning
Data-efficient LLM Fine-tuning for Code Generation
CRPE
CRPE: Expanding The Reasoning Capability of Large Language Model for Code Generation
DeepSeek-Coder
DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence
StarCoder 2 and The Stack v2
StarCoder 2 and The Stack v2: The Next Generation
Repetition In Repetition Out
Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspective
Every Sample Matters
Every Sample Matters: Leveraging Mixture-of-Experts and High-Quality Data for Efficient and Accurate Code LLM
WaveCoder
WaveCoder: Widespread And Versatile Enhancement For Code Large Language Models By Instruction Tuning
Brevity is the soul of wit
Brevity is the soul of wit: Pruning long files for code generation
DataComp-LM
DataComp-LM: In search of the next generation of training sets for language models
Codex
Evaluating Large Language Models Trained on Code
Datasets for Large Language Models
Datasets for Large Language Models: A Comprehensive Survey
Cracks in The Stack
Cracks in The Stack: Hidden Vulnerabilities and Licensing Risks in LLM Pre-Training Datasets
MG-Verilog
MG-Verilog: Multi-grained Dataset Towards Enhanced LLM-assisted Verilog Generation
DataRecipe
DataRecipe --- How to Cook the Data for CodeLLM?
Training Data Extraction
Understanding Privacy Risks of Large Language Models in Japanese Based on Training Data Extraction Attacks
aiXcoder-7B
aiXcoder-7B: A Lightweight and Effective Large Language Model for Code Processing
Inter-Dataset Code Duplication
On Inter-Dataset Code Duplication and Data Leakage in Large Language Models
LLM-ProS
LLM-ProS: Analyzing Large Language Models’ Performance in Competitive Problem Solving
AATK Benchmark
Asleep at the keyboard? assessing the security of github copilot's code contributions
Human-in-the-Loop Review 8 papers
Static and Corpus Audit > Human-in-the-Loop Review
Qwen
Qwen Technical Report
DataMan
DataMan: Data Manager for Pre-training Large Language Models
TeleChat
Technical Report of TeleChat2, TeleChat2.5 and T1
StarCoder 2 and The Stack v2
StarCoder 2 and The Stack v2: The Next Generation
Datasets for Large Language Models
Datasets for Large Language Models: A Comprehensive Survey
Synthetic Data Generation
Synthetic Data Generation Using Large Language Models: Advances in Text and Code
Imperfect Code Generation
Imperfect Code Generation: Uncovering Weaknesses in Automatic Code Generation by Large Language Models
AATK Benchmark
Asleep at the keyboard? assessing the security of github copilot's code contributions
Provenance and Version-History Audit 1 paper
Static and Corpus Audit > Provenance and Version-History Audit
Cracks in The Stack
Cracks in The Stack: Hidden Vulnerabilities and Licensing Risks in LLM Pre-Training Datasets
Model-Membership and Attribution Audit 2 papers
Static and Corpus Audit > Model-Membership and Attribution Audit
CodeMI
Does Your Neural Code Completion Model Use My Code? A Membership Inference Approach
Uncovering Pretraining Code in LLMs
Uncovering Pretraining Code in LLMs: A Syntax-Aware Attribution Approach
LLM-Based Dataset Screening 13 papers
Model-Based Detection > LLM-Based Dataset Screening
Qwen
Qwen Technical Report
Qwen2
Qwen2 Technical Report
DataMan
DataMan: Data Manager for Pre-training Large Language Models
SwallowCode
Rewriting Pre-Training Data Boosts LLM Performance in Math and Code
Qwen3
Qwen3 Technical Report
Qwen2.5
Qwen2.5 Technical Report
TeleChat
Technical Report of TeleChat2, TeleChat2.5 and T1
Kimi K2
Kimi K2: Open Agentic Intelligence
DeepSeek-Coder
DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence
WaveCoder
WaveCoder: Widespread And Versatile Enhancement For Code Large Language Models By Instruction Tuning
Benchmark Builders
Large Language Models are Qualified Benchmark Builders: Rebuilding Pre-Training Datasets for Advancing Code Intelligence Tasks
Datasets for Large Language Models
Datasets for Large Language Models: A Comprehensive Survey
DataRecipe
DataRecipe --- How to Cook the Data for CodeLLM?
Task-Specific Learned Classifiers 5 papers
Model-Based Detection > Task-Specific Learned Classifiers
Less is More
Less is More: On the Importance of Data Quality for Unit Test Generation
DataMan
DataMan: Data Manager for Pre-training Large Language Models
Benchmark Builders
Large Language Models are Qualified Benchmark Builders: Rebuilding Pre-Training Datasets for Advancing Code Intelligence Tasks
DataComp-LM
DataComp-LM: In search of the next generation of training sets for language models
aiXcoder-7B
aiXcoder-7B: A Lightweight and Effective Large Language Model for Code Processing