1. Multi-Server Initialization

	async with MultiServerMCPClient({
	"math": {
		"command": "python", 
		"args": ["./servers/math_server.py"], ...
	},
	"fin": {
		"command": "python", 
		"args": ["./servers/fin_server.py"], ...
	},
	"chroma": {
		"command": "python", 
		"args": ["./servers/chroma_server.py"], ...
	},
	"sqlite": {
		"command": "python", 
		"args": ["./servers/sqlite_server.py"], ...
	}
	}) as client:
	# ... main processing loop ...
								

2. Flexible Question Processing

	# Args: --first N, --last N, --all
	# if args.first:
	#   qa_dict = qa_dict[:args.first]

	for i, item in enumerate(qa_dict):
	question = item['Question']
	print(f"Processing Q{i+1}: {question}")

	level = await categorize_question_with_llm(
					question, categorization_model)
	# ... strategy selection ...
								

3. Multi-Run Strategy & Verification

	temp_list = []
	for _ in range(3): # Multi-run execution
	temp_result = await router.process_question(
		question, 
		client=client, 
		model=answering_model,
		recursion_limit=level_config['recursion_limit']
	)
	extracted_log = result_verifier.extract_info(
							question, temp_result)
	temp_list.append(extracted_log)

	# Verify and select the best answer
	results_list[i] = await result_verifier.verify_results(
									question, temp_list)
								

Impact: Robust orchestration for dynamic strategy application and reliable answer generation through multi-path reasoning.

Baseline Performance (Initial)

$ python score.py
	Overall Accuracy: 0.2600 (13/50)
	...

Our initial execution resulted an accuracy of 26%.

Initial Error Analysis

• Hallucination - LLM generating incorrect financial data
• Tool Selection - Choosing wrong tools for specific tasks
• Temporal Confusion - Mixing data from different fiscal years
• Complex Calculations - Errors in multi-step financial formulas

Improved Performance (Post-Categorization)

Overall Accuracy: 0.3400 (17/50)
Question Distribution:
Level 1: 16 questions - Accuracy: 0.6250 (10/16)
Level 2: 8 questions - Accuracy: 0.2500 (2/8)
Level 3: 2 questions - Accuracy: 1.0000 (2/2)
Level 4: 12 questions - Accuracy: 0.2500 (3/12)
Level 5: 12 questions - Accuracy: 0.0000 (0/12)
Level and Correctness Summary:
levels: 12111 11111 42421 42324 23112 
        21111 55455 55444 55555 44454 
answer: oxoox oooxo xooxx xooxo xooox 
        xxoxx xxxxx xxxox xxxxx xxxxx 
Number of Quesiton hitting recursion limit: 26

Current Performance (Final Overview)

Overall Accuracy: 0.6200 (31/50)
Question Distribution:
Level 1: 16 questions - Accuracy: 0.7500 (12/16)
Level 2: 8 questions - Accuracy: 0.7500 (6/8)
Level 3: 2 questions - Accuracy: 1.0000 (2/2)
Level 4: 11 questions - Accuracy: 0.6364 (7/11)
Level 5: 13 questions - Accuracy: 0.3077 (4/13)

Level and Correctness Summary

levels: 12111 11111 42221 44324 23111
        21121 55555 55444 55555 44454 
answer: oooox ooooo oooxx oxooo oooox
        xxooo ooxox xoooo xxxxx xoxxx 
(o = correct, x = incorrect)

Error Analysis

=============================
ERROR ANALYSIS
=============================
	Multi Doc Failure     8 (16.0%)
	Wrong Calculation     6 (12.0%)
	System Error          2 ( 4.0%)
	Incomplete Data       2 ( 4.0%)
	Format Error          1 ( 2.0%)

Error Distribution

What: Uses gpt-4o-mini to classify questions into 5 difficulty levels based on a detailed prompt.

Prompt Snippet:

"You are an expert in financial analysis...
Categorize into: LEVEL_1 (fact retrieval), LEVEL_2
 (simple calc),... LEVEL_5 (complex reasoning)
Respond with ONLY a single number (1-5)"

Impact on Agent:

• Enables selection of the most resource-efficient and effective strategy.
• Avoids over-complication for simple queries and under-powered approaches for complex ones.
• Sets the stage for tailored agent behavior in subsequent steps.

• Goal: Fast, direct retrieval.
• Prompt: "MAXIMUM 25 STEPS... Provide ONLY exact numerical answer..."
• Agent Behavior: Minimal steps, immediate answering.

• Goal: Structured calculation.
• Prompt: "OPTIMIZED ANALYTICAL PROCESS: EXTRACT, SEARCH, CALCULATE..."
• Agent Behavior: Methodical data extraction & tool use.

• Goal: Complex, multi-data reasoning.
• Prompt: "STREAMLINED APPROACH: IDENTIFY, SEARCH (ALL data upfront)... NO VERIFICATION"
• Agent Behavior: Broader data gathering before calculation.

Query Rewriting:

• Retrieval scope extended (+ 5 years).
• Lookup Table-Based Fiscal Year-End Extraction.
• LLM-based normalization of relative time (e.g., "prior year" to "202X").
• LLM-based question rewriting into a formal financial language format.
• LLM-basedGeneration of multiple search queries related to the initial query.
• Impact: Reduced temporal confusion, more accurate data retrieval for time-sensitive queries.

Document reranking:

• Reciprocal rank fusion algorithm $$\text{Score}(d) = \sum_{i=1}^{N} \frac{1}{k + \text{rank}_i(d)}$$
  • A document may appear in multiple lists, and the final fused score is the sum of the reciprocal ranks from all the lists where it appears.
  • $\text{rank}_i(d)$ is the rank of focument d in the ith ranked list;
  • k is a constant (e.g., 60);
  • N is the number of ranked lists.

• calculate_eps
• calculate_operating_profit_margin
• calculate_current_ratio
• calculate_cashflowfromoperations
• calculate_shares_outstanding
• calculate_price_to_earnings_ratio
• calculate_market_capitalization
• calculate_ratio_of_metrics

• add
• multiply
• divide
• difference
• average
• sum_values
• convert_thousands_to_number
• convert_millions_to_number
• convert_billions_to_number
• calculate_percentage
• calculate_percentage_change
• average_list

• Large language monkeys: scaling inference compute with repeated sampling (3 times).
• A verifier LLM compares these answers and generate the final results.

Verifier Prompt Snippet:

"You are an expert verifier...
Compare multiple answers... Identify inconsistencies...
Select most coherent reasoning path ...
Or synthesize the best aspects of them into a final...
Final Verified Answer: ..."

Impact:

• Crucial quality control.
• Mitigates single-run errors.
• Boosts reliability by cross-checking reasoning paths.

Result Verifier Workflow

flowchart TD
subgraph Agent Run 1
	A1["Question"] --> P1["Agent Processing 1
(Tools, Reasoning)"] --> O1["Output Log 1
(Answer, Tool Calls, Responses)"]
end
subgraph Agent Run 2
	A2["Question"] --> P2["Agent Processing 2
(Tools, Reasoning)"] --> O2["Output Log 2
(Answer, Tool Calls, Responses)"]
end
subgraph Agent Run 3
	A3["Question"] --> P3["Agent Processing 3
(Tools, Reasoning)"] --> O3["Output Log 3
(Answer, Tool Calls, Responses)"]
end

O1 --> VerifierLLM["Verifier LLM
(gpt-4.1-mini)"]
O2 --> VerifierLLM
O3 --> VerifierLLM

VerifierLLM -->|"Comparison & Synthesis"| FinalAnswer["Final Verified Answer"]

classDef agentRun fill:#e3f2fd,stroke:#1e88e5,color:#000;
classDef verifier fill:#fff9c4,stroke:#f57f17,color:#000;
classDef io fill:#fce4ec,stroke:#ad1457,color:#000;
classDef answerNode fill:#e8f5e9,stroke:#2e7d32,color:#000;

class P1,P2,P3 agentRun;
class O1,O2,O3 io;
class VerifierLLM verifier;
class FinalAnswer answerNode;
									

Command-line arguments in mcp_client.py (--first N, --last N, --all) enable targeted testing. These are leveraged by GitHub Actions for automated evaluation:

• qa_dict_benchmark-first-3.yml: (Manual Trigger) Runs on the first 3 questions for a quick smoke test of major changes.
• qa_dict_benchmark-last-15.yml: (Manual Trigger) Tests the typically more complex last 15 questions.
• qa_dict_benchmark-all.yml: (On Push) Performs a full benchmark on all 50 questions to track overall performance and catch regressions.

Impact: Streamlined the testing pipeline, enabled quicker iteration, ensured consistent evaluation, and helped maintain code quality through automated checks.