Failure analysis and improvement

Methods to understand

Classify failures as missing material, bad chunking, retrieval failure, wrong citations, or distorted generation. Change one important factor at a time, compare the same question set, and keep both improvements and regressions.

Apply the method to a scenario

Query: "Does the project permit cross-department access to material?"
Record documents, sources, and answers; abstain without evidence.
Compare retrieval and citations before and after changes and record one failure.

Hands-on workshop

Repair a specific failure using comparison evidence.

1. Locate the cause across material, chunking, retrieval, citations, and generation.
2. Change one key factor and retain the original configuration.
3. Retest the same evaluation set and check improvements and regressions.

Problem: the answer cites a passage that does not support its conclusion
Baseline: the correct document was retrieved, but chunking lost a prior condition
Change: split at section boundaries and preserve necessary context
Retest: compare retrieval, answers, and citations on the same question set
Record: what improved, and did any new error appear?

How to verify

Do not present only one repaired answer. Submit the complete comparison and remaining limitations.

Practice task

Choose a case related to your work and practice "Failure analysis and improvement". Submit inputs, steps, results, and verification records.

• Explain the task goal and inputs
• Provide actual results or runnable deliverables
• Identify errors, limits, and verification evidence