127 lines
3.9 KiB
Markdown
127 lines
3.9 KiB
Markdown
# Feedback QA Procedure
|
|
|
|
End-to-end validation of the training feedback loop across CLI, API, and web dashboard.
|
|
|
|
## Prerequisites
|
|
|
|
1. Start infrastructure and services:
|
|
```bash
|
|
./run infra # Redis + Postgres
|
|
./run dev:api # NestJS API on :3780
|
|
./run dev:web # React dashboard on :3781
|
|
```
|
|
|
|
2. Clean state:
|
|
```bash
|
|
rm -f ~/.kthulu/feedback.jsonl
|
|
```
|
|
|
|
## 1. CLI Smoke Tests
|
|
|
|
### 1.1 Approve a session
|
|
```bash
|
|
./run train feedback approve test-session-001 --reason "good tool use"
|
|
```
|
|
**Verify**: `~/.kthulu/feedback.jsonl` exists with 1 line, `feedbackType=approve`
|
|
|
|
### 1.2 Reject a session
|
|
```bash
|
|
./run train feedback reject test-session-002 --reason "hallucinated code"
|
|
```
|
|
**Verify**: `feedback.jsonl` has 2 lines, second has `feedbackType=reject`
|
|
|
|
### 1.3 Annotate with dimensions
|
|
```bash
|
|
./run train feedback annotate test-session-003 \
|
|
--tool-efficiency 0.9 --code-correctness 0.3 \
|
|
--reasoning-quality 0.7 --task-completion 0.8
|
|
```
|
|
**Verify**: 3 lines, third has 4 dimension scores
|
|
|
|
### 1.4 Summary
|
|
```bash
|
|
./run train feedback summary
|
|
```
|
|
**Verify**: `Total=3, Approved=1, Rejected=1, Annotated=1`
|
|
|
|
### 1.5 Export
|
|
```bash
|
|
./run train feedback export -o /tmp/feedback_export.jsonl
|
|
```
|
|
**Verify**: `/tmp/feedback_export.jsonl` has 3 JSON records
|
|
|
|
## 2. API Smoke Tests
|
|
|
|
### 2.1 Submit approve via API
|
|
```bash
|
|
curl -X POST http://localhost:3780/feedback \
|
|
-H 'Content-Type: application/json' \
|
|
-d '{"sessionId":"api-session-001","feedbackType":"approve","reason":"via API"}'
|
|
```
|
|
**Verify**: 201 response, record returned with `source=api`
|
|
|
|
### 2.2 Submit annotate via API
|
|
```bash
|
|
curl -X POST http://localhost:3780/feedback \
|
|
-H 'Content-Type: application/json' \
|
|
-d '{"sessionId":"api-session-002","feedbackType":"annotate","dimensions":{"tool_efficiency":0.8}}'
|
|
```
|
|
**Verify**: 201 response, dimensions present
|
|
|
|
### 2.3 Summary endpoint
|
|
```bash
|
|
curl http://localhost:3780/feedback/summary
|
|
```
|
|
**Verify**: total includes both CLI and API records
|
|
|
|
### 2.4 Session lookup
|
|
```bash
|
|
curl http://localhost:3780/feedback/test-session-001
|
|
```
|
|
**Verify**: returns array with the CLI-submitted record
|
|
|
|
## 3. Pipeline Integration Test
|
|
|
|
### 3.1 Create sample training data
|
|
```bash
|
|
mkdir -p /tmp/kthulu-qa/training
|
|
cat > /tmp/kthulu-qa/training/sessions.jsonl << 'EOF'
|
|
{"id":"test-session-001","messages":[{"role":"user","content":"fix the bug"},{"role":"assistant","content":"done"}],"metadata":{"source":"test","projectPath":"/tmp","timestamp":"2026-01-01T00:00:00Z","quality":0.6}}
|
|
{"id":"test-session-002","messages":[{"role":"user","content":"add feature"},{"role":"assistant","content":"added"}],"metadata":{"source":"test","projectPath":"/tmp","timestamp":"2026-01-02T00:00:00Z","quality":0.7}}
|
|
{"id":"test-session-003","messages":[{"role":"user","content":"refactor"},{"role":"assistant","content":"refactored"}],"metadata":{"source":"test","projectPath":"/tmp","timestamp":"2026-01-03T00:00:00Z","quality":0.5}}
|
|
EOF
|
|
```
|
|
|
|
### 3.2 Export with feedback applied
|
|
```bash
|
|
./run train export \
|
|
--threshold 0.0 \
|
|
--output /tmp/kthulu-qa/export \
|
|
--feedback ~/.kthulu/feedback.jsonl
|
|
```
|
|
**Verify**:
|
|
- `train.jsonl` + `eval.jsonl` exclude `test-session-002` (rejected)
|
|
- `test-session-001` quality = 1.0 (approved)
|
|
- `test-session-003` quality = average of annotation scores
|
|
|
|
## 4. Web Dashboard Terminal Verification
|
|
|
|
1. Navigate to `http://localhost:3781/feedback`
|
|
2. Verify terminal renders with cyberpunk theme
|
|
3. Type `help` → verify command list appears
|
|
4. Type `approve qa-test-001 --reason "from browser"` → verify success message
|
|
5. Type `summary` → verify total count incremented
|
|
6. Screenshot final state
|
|
|
|
## 5. Cross-Channel Verification
|
|
|
|
- Record created via web terminal visible in `./run train feedback summary`
|
|
- Record created via CLI visible in `GET /feedback/summary`
|
|
- All sources (cli, api, web) appear in `summary.bySource`
|
|
|
|
## 6. Cleanup
|
|
|
|
```bash
|
|
rm -f ~/.kthulu/feedback.jsonl
|
|
rm -rf /tmp/kthulu-qa /tmp/feedback_export.jsonl
|
|
```
|