From 953bf3f6bb71b924b13f8681a2cfbb5613eb70ee Mon Sep 17 00:00:00 2001 From: marcusquinn <6428977+marcusquinn@users.noreply.github.com> Date: Tue, 22 Apr 2025 22:01:38 +0100 Subject: [PATCH] Add comprehensive error checking and feedback loops documentation --- .ai-assistant.md | 31 +- .../error-checking-feedback-loops.md | 527 ++++++++++++++++++ 2 files changed, 556 insertions(+), 2 deletions(-) create mode 100644 .ai-workflows/error-checking-feedback-loops.md diff --git a/.ai-assistant.md b/.ai-assistant.md index 8a95d77..09a6a89 100644 --- a/.ai-assistant.md +++ b/.ai-assistant.md @@ -87,7 +87,12 @@ Always run PHPCS and PHPCBF locally before committing code to ensure it meets th ## Common Tasks -For detailed instructions on common tasks like creating releases, adding features, fixing bugs, and testing previous versions, see **@.ai-workflows/release-process.md**. +For detailed instructions on common tasks, refer to these workflow documents: + +* **@.ai-workflows/release-process.md**: Creating releases, versioning, and deployment +* **@.ai-workflows/feature-development.md**: Adding new features and functionality +* **@.ai-workflows/bug-fixing.md**: Diagnosing and fixing issues +* **@.ai-workflows/error-checking-feedback-loops.md**: Monitoring CI/CD pipelines and establishing feedback loops ## Avoiding Cross-Repository Confusion @@ -153,6 +158,28 @@ When working with this repository, remember these preferences: 1. Follow WordPress coding standards 2. Use OOP approach with namespaced classes 3. Keep code modular and maintainable -3. Reference these preferences in future interactions +4. Reference these preferences in future interactions This ensures consistency across coding sessions and reduces the need for developers to repeatedly explain their preferences. + +## Autonomous CI/CD and Error Handling + +This repository is set up for autonomous continuous integration and deployment with minimal human intervention. As an AI assistant, you should: + +1. **Monitor CI/CD Pipelines**: Use the GitHub API to check workflow status and identify failures +2. **Diagnose Issues**: Analyze error logs and determine root causes +3. **Implement Fixes**: Make necessary code changes to resolve issues +4. **Verify Solutions**: Ensure fixes pass all tests and quality checks +5. **Document Resolutions**: Update documentation with solutions for future reference + +For detailed instructions on establishing feedback loops and error checking processes, see **@.ai-workflows/error-checking-feedback-loops.md**. + +### When to Consult Humans + +Only consult humans for: + +* Product design decisions +* Security-critical changes +* Major architectural decisions +* Access to information not available through APIs or terminal +* Novel problems without precedent or documented solutions diff --git a/.ai-workflows/error-checking-feedback-loops.md b/.ai-workflows/error-checking-feedback-loops.md new file mode 100644 index 0000000..76e090e --- /dev/null +++ b/.ai-workflows/error-checking-feedback-loops.md @@ -0,0 +1,527 @@ +# Error Checking and Feedback Loops + +This document outlines the processes for error checking, debugging, and establishing feedback loops between AI assistants and various systems in the development workflow. The goal is to create a seamless, autonomous CI/CD pipeline where the AI can identify, diagnose, and fix issues with minimal human intervention. + +## Table of Contents + +* [GitHub Actions Workflow Monitoring](#github-actions-workflow-monitoring) +* [Local Build and Test Feedback](#local-build-and-test-feedback) +* [Code Quality Tool Integration](#code-quality-tool-integration) +* [Automated Error Resolution](#automated-error-resolution) +* [Feedback Loop Architecture](#feedback-loop-architecture) +* [When to Consult Humans](#when-to-consult-humans) + +## GitHub Actions Workflow Monitoring + +### Checking Workflow Status via GitHub API + +AI assistants can directly monitor GitHub Actions workflows using the GitHub API to identify failures and diagnose issues: + +``` +github-api /repos/{owner}/{repo}/actions/runs +``` + +#### Step-by-Step Process + +1. **Get Recent Workflow Runs**: + ``` + github-api /repos/wpallstars/wp-plugin-starter-template-for-ai-coding/actions/runs + ``` + +2. **Filter for Failed Runs**: + ``` + github-api /repos/wpallstars/wp-plugin-starter-template-for-ai-coding/actions/runs?status=failure + ``` + +3. **Get Details for a Specific Run**: + ``` + github-api /repos/wpallstars/wp-plugin-starter-template-for-ai-coding/actions/runs/{run_id} + ``` + +4. **Get Jobs for a Workflow Run**: + ``` + github-api /repos/wpallstars/wp-plugin-starter-template-for-ai-coding/actions/runs/{run_id}/jobs + ``` + +5. **Analyze Job Logs** (if accessible via API): + ``` + github-api /repos/wpallstars/wp-plugin-starter-template-for-ai-coding/actions/jobs/{job_id}/logs + ``` + +### Common GitHub Actions Errors and Solutions + +#### Missing or Outdated Action Versions + +**Error**: `Missing download info for actions/upload-artifact@v3` + +**Solution**: Update to the latest version of the action: +```yaml +uses: actions/upload-artifact@v4 +``` + +#### Port Configuration Issues for WordPress Multisite + +**Error**: `The current host is 127.0.0.1:8888, but WordPress multisites do not support custom ports.` + +**Solution**: Use port 80 for multisite environments: +```yaml +npx @wp-playground/cli server --blueprint playground/multisite-blueprint.json --port 80 --login & +``` + +#### Artifact Path Syntax Issues + +**Error**: Invalid path syntax for artifacts + +**Solution**: Use multi-line format for better readability: +```yaml +path: | + cypress/videos + cypress/screenshots +``` + +#### Concurrency Control + +**Problem**: Redundant workflow runs when multiple commits land quickly + +**Solution**: Add concurrency control to cancel in-progress runs: +```yaml +concurrency: + group: playground-tests-${{ github.ref }} + cancel-in-progress: true +``` + +## Local Build and Test Feedback + +### Monitoring Local Test Runs + +AI assistants can monitor local test runs by analyzing the output of test commands: + +#### PHP Unit Tests + +```bash +composer run phpunit +``` + +#### Cypress Tests + +```bash +npm run test:single +npm run test:multisite +``` + +#### WordPress Playground Tests + +```bash +npm run test:playground:single +npm run test:playground:multisite +``` + +### Capturing and Analyzing Test Output + +1. **Run Tests with Output Capture**: + ```bash + npm run test:single > test-output.log 2>&1 + ``` + +2. **Analyze Output for Errors**: + ```bash + launch-process command="cat test-output.log | grep -i 'error\|fail\|exception'" wait=true max_wait_seconds=10 + ``` + +3. **Parse Structured Test Results** (if available): + ```bash + launch-process command="cat cypress/results/results.json" wait=true max_wait_seconds=10 + ``` + +### Common Local Test Errors and Solutions + +#### WordPress Playground Port Issues + +**Error**: `Error: The current host is 127.0.0.1:8888, but WordPress multisites do not support custom ports.` + +**Solution**: Modify the port in the blueprint or test configuration: +```json +{ + "features": { + "networking": true + } +} +``` + +#### Cypress Selector Errors + +**Error**: `Timed out retrying after 4000ms: expected '' to have class 'wp-admin'` + +**Solution**: Update selectors to be more robust and handle login states: +```javascript +cy.get('body').then(($body) => { + if ($body.hasClass('login')) { + cy.get('#user_login').type('admin'); + cy.get('#user_pass').type('password'); + cy.get('#wp-submit').click(); + } +}); + +// Check for admin bar instead of body class +cy.get('#wpadminbar').should('exist'); +``` + +## Code Quality Tool Integration + +### Automated Code Quality Checks + +AI assistants can integrate with various code quality tools to identify and fix issues: + +#### PHPCS (PHP CodeSniffer) + +```bash +composer run phpcs +``` + +#### ESLint (JavaScript) + +```bash +npm run lint:js +``` + +#### Stylelint (CSS) + +```bash +npm run lint:css +``` + +### Parsing Code Quality Tool Output + +1. **Run Code Quality Check**: + ```bash + composer run phpcs > phpcs-output.log 2>&1 + ``` + +2. **Analyze Output for Errors**: + ```bash + launch-process command="cat phpcs-output.log | grep -i 'ERROR\|WARNING'" wait=true max_wait_seconds=10 + ``` + +3. **Automatically Fix Issues** (when possible): + ```bash + composer run phpcbf + ``` + +### Monitoring Code Quality Feedback in Pull Requests + +Automated code quality tools often provide feedback directly in pull requests. AI assistants can check these comments to identify and address issues: + +#### Accessing PR Comments via GitHub API + +``` +github-api /repos/wpallstars/wp-plugin-starter-template-for-ai-coding/pulls/{pull_number}/comments +``` + +#### Accessing PR Review Comments + +``` +github-api /repos/wpallstars/wp-plugin-starter-template-for-ai-coding/pulls/{pull_number}/reviews +``` + +#### Checking CodeRabbit Feedback + +CodeRabbit provides AI-powered code review comments that can be accessed via the GitHub API: + +1. **Get PR Comments**: + ``` + github-api /repos/wpallstars/wp-plugin-starter-template-for-ai-coding/pulls/{pull_number}/comments + ``` + +2. **Filter for CodeRabbit Comments**: + Look for comments from the `coderabbitai` user. + +3. **Parse Actionable Feedback**: + CodeRabbit comments typically include: + * Code quality issues + * Suggested improvements + * Best practice recommendations + * Specific code snippets to fix + +#### Checking Codacy and CodeFactor Feedback + +These tools provide automated code quality checks and post results as PR comments or status checks: + +1. **Check PR Status Checks**: + ``` + github-api /repos/wpallstars/wp-plugin-starter-template-for-ai-coding/commits/{commit_sha}/check-runs + ``` + +2. **Get Detailed Reports** (if available via API): + ``` + github-api /repos/wpallstars/wp-plugin-starter-template-for-ai-coding/commits/{commit_sha}/check-runs/{check_run_id} + ``` + +3. **Parse Common Issues**: + * Code style violations + * Potential bugs + * Security vulnerabilities + * Performance issues + * Duplication + +#### Checking SonarCloud Analysis + +SonarCloud provides detailed code quality and security analysis: + +1. **Check SonarCloud Status**: + ``` + github-api /repos/wpallstars/wp-plugin-starter-template-for-ai-coding/commits/{commit_sha}/check-runs?check_name=SonarCloud + ``` + +2. **Parse SonarCloud Issues**: + * Code smells + * Bugs + * Vulnerabilities + * Security hotspots + * Coverage gaps + +### Common Code Quality Issues and Solutions + +#### WordPress Coding Standards + +**Error**: `ERROR: Expected snake_case for function name, but found camelCase` + +**Solution**: Rename functions to follow snake_case convention: +```php +// Before +function getPluginVersion() { ... } + +// After +function get_plugin_version() { ... } +``` + +#### Missing Docblocks + +**Error**: `ERROR: Missing doc comment for function` + +**Solution**: Add proper docblocks: +```php +/** + * Get the plugin version. + * + * @since 1.0.0 + * @return string The plugin version. + */ +function get_plugin_version() { ... } +``` + +## Automated Error Resolution + +### Error Resolution Workflow + +1. **Identify Error**: Use GitHub API or local test output to identify errors +2. **Categorize Error**: Determine error type and severity +3. **Search for Solution**: Look for patterns in known solutions +4. **Apply Fix**: Make necessary code changes +5. **Verify Fix**: Run tests again to confirm the issue is resolved +6. **Document Solution**: Update documentation with the solution + +### Processing Code Quality Feedback + +#### Extracting Actionable Items from PR Comments + +1. **Collect All Feedback**: + ``` + github-api /repos/wpallstars/wp-plugin-starter-template-for-ai-coding/pulls/{pull_number}/comments + github-api /repos/wpallstars/wp-plugin-starter-template-for-ai-coding/pulls/{pull_number}/reviews + ``` + +2. **Categorize Issues**: + * Critical: Security vulnerabilities, breaking bugs + * High: Code quality violations, potential bugs + * Medium: Style issues, best practices + * Low: Documentation, minor improvements + +3. **Prioritize Fixes**: + * Address critical issues first + * Group related issues for efficient fixing + * Consider dependencies between issues + +4. **Create Fix Plan**: + * Document files that need changes + * Outline specific changes needed + * Note any potential side effects + +#### Responding to Code Quality Tool Comments + +1. **Acknowledge Feedback**: + ``` + github-api /repos/wpallstars/wp-plugin-starter-template-for-ai-coding/issues/comments/{comment_id}/reactions + ``` + +2. **Implement Fixes**: + ``` + str-replace-editor command="str_replace" path="path/to/file.php" str_replace_entries=[...] + ``` + +3. **Explain Changes** (if needed): + ``` + github-api /repos/wpallstars/wp-plugin-starter-template-for-ai-coding/pulls/{pull_number}/comments + ``` + +4. **Request Review** (if needed): + ``` + github-api /repos/wpallstars/wp-plugin-starter-template-for-ai-coding/pulls/{pull_number}/requested_reviewers + ``` + +### Example: Fixing GitHub Actions Workflow + +``` +# 1. Identify the error +github-api /repos/wpallstars/wp-plugin-starter-template-for-ai-coding/actions/runs?status=failure + +# 2. Get details of the failing job +github-api /repos/wpallstars/wp-plugin-starter-template-for-ai-coding/actions/runs/{run_id}/jobs + +# 3. Fix the issue in the workflow file +str-replace-editor command="str_replace" path=".github/workflows/playground-tests.yml" str_replace_entries=[...] + +# 4. Commit and push the changes +launch-process command="git add .github/workflows/playground-tests.yml" wait=true max_wait_seconds=30 +launch-process command="git commit -m 'Fix GitHub Actions workflow: update upload-artifact to v4'" wait=true max_wait_seconds=60 +launch-process command="git push" wait=true max_wait_seconds=60 + +# 5. Verify the fix +github-api /repos/wpallstars/wp-plugin-starter-template-for-ai-coding/actions/runs +``` + +## Feedback Loop Architecture + +### Complete Feedback Loop System + +``` +┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ +│ │ │ │ │ │ +│ Code Changes │────▶│ Local Testing │────▶│ GitHub Actions │ +│ │ │ │ │ │ +└────────┬────────┘ └────────┬────────┘ └────────┬────────┘ + │ │ │ + │ │ │ + │ │ │ +┌────────▼────────┐ ┌────────▼────────┐ ┌────────▼────────┐ +│ │ │ │ │ │ +│ AI Assistant │◀────│ Error Analysis │◀────│ Status Check │ +│ │ │ │ │ │ +└────────┬────────┘ └─────────────────┘ └─────────────────┘ + │ + │ + │ +┌────────▼────────┐ ┌─────────────────┐ +│ │ │ │ +│ Fix Generation │────▶│ Human Review │ (only when necessary) +│ │ │ │ +└─────────────────┘ └─────────────────┘ +``` + +### Key Components + +1. **Code Changes**: Initial code modifications +2. **Local Testing**: Immediate feedback on local environment +3. **GitHub Actions**: Remote CI/CD pipeline validation +4. **Status Check**: Monitoring workflow status via GitHub API +5. **Error Analysis**: Parsing and categorizing errors +6. **AI Assistant**: Central intelligence for error resolution +7. **Fix Generation**: Creating and implementing solutions +8. **Human Review**: Optional step for complex decisions + +## Handling Direct Feedback from Code Review Tools + +### Accessing and Processing CodeRabbit Feedback + +CodeRabbit provides detailed AI-powered code reviews that can be directly accessed and processed: + +#### Example CodeRabbit Feedback + +``` +coderabbitai bot left a comment +Actionable comments posted: 1 + +🧹 Nitpick comments (3) +.github/workflows/playground-tests-fix.yml (3) +9-13: Add concurrency control to avoid redundant runs. +When multiple commits land in quick succession, you may end up with overlapping Playground test jobs. Adding a concurrency block will cancel in‑progress runs for the same ref and reduce CI load: + +concurrency: + group: playground-tests-${{ github.ref }} + cancel-in-progress: true +``` + +#### Processing Steps + +1. **Extract Specific Recommendations**: + * Identify file paths and line numbers + * Parse suggested code changes + * Understand the rationale for changes + +2. **Implement Recommendations**: + ``` + str-replace-editor command="str_replace" path=".github/workflows/playground-tests-fix.yml" str_replace_entries=[{"old_str": "name: WordPress Playground Tests Fix\n\non:\n push:\n branches: [ main, feature/*, bugfix/* ]\n pull_request:\n branches: [ main ]", "new_str": "name: WordPress Playground Tests Fix\n\non:\n push:\n branches: [ main, feature/*, bugfix/* ]\n pull_request:\n branches: [ main ]\n\nconcurrency:\n group: playground-tests-${{ github.ref }}\n cancel-in-progress: true", "old_str_start_line_number": 1, "old_str_end_line_number": 6}] + ``` + +3. **Verify Implementation**: + * Run local tests if applicable + * Commit changes with descriptive message + * Monitor CI/CD pipeline for success + +### Handling SonarCloud and Codacy Feedback + +These tools provide structured feedback that can be systematically addressed: + +#### Example SonarCloud Feedback + +``` +SonarCloud Quality Gate failed +- 3 Bugs +- 5 Code Smells +- 1 Security Hotspot +``` + +#### Processing Steps + +1. **Access Detailed Reports**: + * Use the SonarCloud API or web interface + * Categorize issues by severity and type + +2. **Address Issues Systematically**: + * Fix bugs first + * Address security hotspots + * Resolve code smells + +3. **Document Resolutions**: + * Note patterns of issues for future prevention + * Update coding guidelines if necessary + +## When to Consult Humans + +While the goal is to create an autonomous system, there are scenarios where human input is necessary: + +### Scenarios Requiring Human Consultation + +1. **Product Design Decisions**: Features, UX, and strategic direction +2. **Security-Critical Changes**: Changes that could impact security posture +3. **Architectural Decisions**: Major structural changes to the codebase +4. **Deployment Approvals**: Final approval for production releases +5. **Access Requirements**: When additional permissions are needed +6. **Ambiguous Errors**: When errors have multiple possible interpretations +7. **Novel Problems**: Issues without precedent or documented solutions +8. **External Service Issues**: Problems with third-party services + +### Effective Human Consultation + +When consulting humans, provide: + +1. **Clear Context**: Explain what you were trying to accomplish +2. **Error Details**: Provide specific error messages and logs +3. **Attempted Solutions**: Document what you've already tried +4. **Specific Questions**: Ask targeted questions rather than open-ended ones +5. **Recommendations**: Suggest possible solutions for approval + +## Conclusion + +This error checking and feedback loop system creates a comprehensive framework for AI-driven development with minimal human intervention. By systematically monitoring, analyzing, and resolving errors across local and remote environments, the AI assistant can maintain high code quality and ensure smooth CI/CD processes. + +For specific workflows related to feature development, bug fixing, and releases, refer to the other documents in the `.ai-workflows/` directory.