OpenAI researchers have introduced CriticGPT, a new artificial intelligence model designed to detect bugs in the code generated by ChatGPT.
Based on the GPT-4 family, CriticGPT analyzes the code and displays possible errors, making it easy to detect errors that are difficult to notice. By training CriticGPT on a set of code samples containing intentionally added errors, the researchers trained it to recognize and flag various coding errors.
In 63 percent of cases involving naturally occurring LLM errors, CriticGPT critiques were found to be preferred by annotators over human critiques, and human-machine teams using CriticGPT wrote more comprehensive critiques than humans alone and reduced confabulation rates compared to AI-only critiques.
The researchers also developed a new technique they call Force Sampling Beam Search (FSBS). This method helps CriticGPT write more detailed code reviews. With this, you can adjust how accurate CriticGPT is when looking for issues, as well as control how often it might produce issues that don't actually exist.
