Just a post for sharing my last experimentation with a code review from an AI.
As wrote on the first sentence. During a quick review of code (by an human, then AI), I got some strange traceback lines. One was a private joke from AI. I misunderstood it. So, my bad.
The second one is a mistake (or maybe undocumented method) highlighted during process for proving that the addon will work. This thread can also illustrate the limits or errors related to many automatic reviews from AI models.
By ignoring mistakes around naming, it still points something out. Sure, the quality of review or model (should also use “temperature”:1.0, for this one) can lead to errors or false/positiv. This mistake on both sides (user and machine) has been resolved, but it also means that with current pseudo reverse review (e.g., for 0$ or few token), we should not trust most AI (even Anthropic) on all steps for such set of actions.
The above recent model and related prompt only tried to prove any action, a step by step review. So, I was able to guess why the AI created these private functions, not present on addon’s code. To understand the mixup done by the AI means to know the training sources, which is not possible in theory, for such AI models. So, by listing file references used during review, we can guess that there was a temp database name (maybe Coryphée ?), headless process and multiple custom scripts (e.g., unit tests). A common process for this type of review.
How from an addon on a github repository, this AI will check the complete gramps’ code? This was not a local review, neither a pure standalone script nor downloaded source (there was already a limitation on token number [1000] and downloaded set of files) . This model had some gramps module references in “memory”. Which version of the code? What was the source? What dependency? etc.
Sure, this might be a ‘real time’ web query/search but it leads to others issues. What about PR not yet merged? Are they using secure sources? What about timelog of the source code?
Maybe for token or memory issues, AI seems to generate its own version of the addon [own coding like private _find_duplicates(self) or _remove_duplicate(self, handle)], removing some documentation and comment lines (also generating a private joke). I am fine with that. But user or non-experimented coder (like me) can quick waste time to try to get back these line references or wrong named methods [_find_duplicates(self) or _remove_duplicate(self, handle)].
For the gramps project, the last listing of (custom) imported modules, might be my indirect ‘real question’. As even with an IDE, I was not able to understand the relation between them and this addon? Also, I was not able to understand why there was a warning for some critical methods using a pass. It was not clear if it was something around transaction commit or if it was ‘really’ pointed out by this AI during exercice (own cooking or custom gramps version) for only proving that the tool will work.