EvalForge20

Build. Connect. Evaluate. Compare.

EvalForge20 is an open-source, local-first AI model evaluation platform designed to benchmark Large Language Models (LLMs) through an interactive visual playground.

Drag models, datasets and evaluation metrics onto a node-based canvas, connect them, configure your benchmark and execute evaluations using locally available models through Ollama.

Instead of writing evaluation scripts for every experiment, EvalForge20 provides a visual environment for designing, executing, inspecting and comparing AI model evaluations.

Visual evaluation workflows. Local model execution. Measurable results.

Evaluating AI models involves more than checking whether a response looks correct.

Developers need to understand model accuracy, reasoning capabilities, execution speed, resource consumption, reliability and consistency across different tasks.

EvalForge20 aims to bring these capabilities together in one visual evaluation environment.

Users can select a model, choose benchmark datasets, connect evaluation parameters and execute experiments without manually building evaluation pipelines.

1. Select an available model from the model library.
2. Drag a benchmark dataset into the playground.
3. Drag the evaluation metrics you want to measure.
4. Connect the components using the visual canvas.
5. Configure the evaluation.
6. Execute the workflow.
7. Inspect actual model outputs and calculated evaluation results.
8. Compare different models using equivalent benchmarks.

The platform is designed around one principle:

Evaluate only what matters to you.

For example, connecting a model to an Accuracy evaluator should run an accuracy evaluation, not an unnecessary collection of unrelated benchmarks.

An interactive, node-based workspace inspired by visual workflow editors such as n8n.

• Drag-and-drop model nodes.
• Connect datasets to models.
• Connect models to evaluation metrics.
• Configure individual nodes.
• Build single-model evaluation workflows.
• Build multi-model comparison workflows.
• Validate workflows before execution.
• Monitor execution progress.
• Inspect evaluation results.

EvalForge20 uses Ollama as its local model inference provider.

The intended local-first workflow includes:

• Discovering locally installed models.
• Selecting models from the available inventory.
• Executing inference locally.
• Collecting model responses.
• Measuring inference performance.
• Running benchmark datasets.
• Comparing models under equivalent conditions.

Example model families that can be evaluated when supported by the local Ollama installation:

• Llama
• Qwen
• DeepSeek
• Gemma
• Mistral
• Phi

Model availability depends on the installed Ollama models, hardware and supported model formats.

EvalForge20 is designed around independently selectable evaluators.

Evaluator availability depends on the implementation, required input data and evaluation configuration.

Support for structured benchmark datasets enables reproducible model testing.

Planned dataset capabilities include:

• CSV uploads.
• JSON uploads.
• JSONL uploads.
• Custom test cases.
• Reference answers.
• Dataset validation.
• Dataset previews.
• Benchmark dataset management.
• Dataset version tracking.

Example dataset:

{
  "input": "What is machine learning?",
  "expected_output": "Machine learning is a field of AI that enables systems to learn from data."
}

Datasets can contain additional fields such as context, expected labels and metadata.

Compare multiple models against the same dataset and evaluation criteria.

Example:

                  Benchmark Dataset
                         |
             +-----------+-----------+
             |                       |
          Model A                 Model B
             |                       |
             +-----------+-----------+
                         |
                Shared Evaluators
                         |
                Comparison Results

Compare models using:

• Accuracy.
• Latency.
• Tokens per second.
• Reliability.
• Resource utilization.
• Task-specific benchmark performance.

Comparisons should use equivalent datasets, prompts, generation settings and evaluator configurations.

The results experience is designed to provide:

• Metric scorecards.
• Model comparison charts.
• Individual test-case results.
• Expected versus generated outputs.
• Failed test cases.
• Execution metadata.
• Benchmark history.
• Exportable evaluation reports.

All displayed benchmark values should be derived from actual execution or clearly identified as estimates.

EvalForge20 follows a modular architecture that separates model inference, evaluation, workflow execution, storage and visualization.

flowchart TB
    U["User"] --> UI["EvalForge20 Frontend"]
    UI --> CANVAS["Visual Playground"]

    CANVAS --> WF["Workflow Definition"]
    WF --> API["FastAPI Backend"]

    API --> VALIDATOR["Workflow Validator"]
    VALIDATOR --> ENGINE["Native Evaluation Engine"]

    ENGINE --> DATA["Dataset Manager"]
    ENGINE --> PROVIDER["Model Provider Layer"]
    ENGINE --> EVALUATORS["Evaluator Registry"]

    PROVIDER --> OLLAMA["Ollama Local API"]
    OLLAMA --> MODELS["Installed Local Models"]

    DATA --> ENGINE
    ENGINE --> EVALUATORS

    EVALUATORS --> RESULTS["Metric Aggregation"]
    RESULTS --> DB["SQLite / PostgreSQL"]
    RESULTS --> STREAM["Execution Progress"]

    STREAM --> UI
    DB --> UI

flowchart TD
    A["Build Evaluation Workflow"] --> B["Validate Connections"]
    B --> C{"Workflow Valid?"}

    C -->|No| D["Display Validation Errors"]
    D --> A

    C -->|Yes| E["Create Experiment"]
    E --> F["Load Dataset"]
    F --> G["Resolve Model Provider"]
    G --> H["Execute Model Inference"]

    H --> I["Collect Model Responses"]
    I --> J["Run Selected Evaluators"]

    J --> K["Calculate Metrics"]
    K --> L["Persist Results"]
    L --> M["Stream Results to Frontend"]
    M --> N["Display Charts and Test Cases"]

flowchart TB
    DATA["Shared Benchmark Dataset"]

    DATA --> A["Model A"]
    DATA --> B["Model B"]
    DATA --> C["Model C"]

    A --> EA["Selected Evaluators"]
    B --> EB["Selected Evaluators"]
    C --> EC["Selected Evaluators"]

    EA --> R["Comparison Engine"]
    EB --> R
    EC --> R

    R --> OUT["Side-by-Side Results"]
    OUT --> CHART["Comparison Charts"]

Ollama provides a convenient way to run supported open-weight models locally.

Using Ollama allows EvalForge20 to:

• Discover installed models.
• Run inference locally.
• Reduce dependence on paid inference APIs.
• Benchmark models on available hardware.
• Keep model execution within the local environment.

The core evaluation engine is independent of Ollama, allowing additional model providers to be introduced through adapters in the future.

Reliable benchmarking requires transparent and reproducible evaluation methods.

EvalForge20 is designed to distinguish between deterministic metrics, model-judged metrics and estimated measurements.

Measures the proportion of predictions matching their expected answers under a defined normalization rule.

[ Accuracy = \frac{Correct\ Predictions}{Total\ Valid\ Predictions} ]

[ Precision = \frac{TP}{TP+FP} ]

[ Recall = \frac{TP}{TP+FN} ]

[ F1 = 2 \times \frac{Precision \times Recall}{Precision + Recall} ]

[ Average\ Latency = \frac{\sum Request\ Durations}{Completed\ Requests} ]

[ Tokens/sec = \frac{Output\ Tokens}{Generation\ Duration} ]

• Use identical benchmark datasets when comparing models.
• Record model identifiers and versions.
• Record generation parameters.
• Record dataset versions.
• Distinguish cold-start and warm inference.
• Use repeated runs where appropriate.
• Document evaluator versions.
• Identify unavailable measurements.
• Never replace missing metrics with fabricated zero values.

A model's accuracy is benchmark-dependent. EvalForge20 does not treat one benchmark score as a universal measure of intelligence.

The intended modular project structure:

EvalForge20/
│
├── frontend/
│   ├── app/
│   │   ├── playground/
│   │   ├── experiments/
│   │   ├── models/
│   │   ├── datasets/
│   │   ├── benchmarks/
│   │   └── leaderboard/
│   │
│   ├── components/
│   │   ├── canvas/
│   │   │   ├── nodes/
│   │   │   ├── edges/
│   │   │   ├── sidebar/
│   │   │   ├── toolbar/
│   │   │   └── config/
│   │   ├── results/
│   │   ├── charts/
│   │   └── ui/
│   │
│   ├── features/
│   ├── hooks/
│   ├── lib/
│   └── types/
│
├── backend/
│   ├── src/
│   │   └── evalforge/
│   │       ├── api/
│   │       │   └── routes/
│   │       ├── core/
│   │       ├── providers/
│   │       ├── engine/
│   │       ├── evaluators/
│   │       ├── datasets/
│   │       ├── hardware/
│   │       ├── experiments/
│   │       ├── database/
│   │       └── schemas/
│   │
│   ├── tests/
│   └── requirements.txt
│
├── docs/
│   └── assets/
│       └── evalforge-playground.png
│
├── scripts/
├── .env.example
├── .gitignore
└── README.md

Install the following:

• Node.js.
• Python 3.11 or a compatible supported version.
• Ollama.
• Git.

A compatible Ollama model must be installed before performing local inference.

git clone YOUR_REPOSITORY_URL
cd EvalForge20

Replace YOUR_REPOSITORY_URL with the actual repository URL.

Download Ollama:

https://ollama.com/

Start the Ollama service and download a model.

Example:

ollama pull qwen2.5:1.5b

Check installed models:

ollama list

Check the local API:

curl http://localhost:11434/api/tags

cd backend

python -m venv .venv

source .venv/bin/activate

pip install -r requirements.txt

On Windows, activate the virtual environment using:

.venv\Scripts\activate

Configure the environment using the provided .env.example.

Use the repository's configured development launcher:

npm run dev

Alternatively, when using Uvicorn directly:

uvicorn evalforge.main:app --reload --host 127.0.0.1 --port 8000

The actual Python module path should match the backend entry point.

API documentation:

http://localhost:8000/docs

Open another terminal:

cd frontend

npm install

npm run dev

Open:

http://localhost:3000

Default Ollama endpoint:

http://localhost:11434

Example model:

qwen2.5:1.5b

EvalForge20 should discover models dynamically rather than depending on a fixed model list.

The application should display:

• Installed models.
• Model availability.
• Model identifier.
• Model metadata where available.
• Ollama connection status.

The platform should not assume that every model can run on every machine.

The backend is organized around REST APIs.

Endpoint availability depends on the current implementation.

EvalForge20 is being developed incrementally, with evaluation correctness taking priority over feature quantity.

• Complete visual playground.
• Drag-and-drop model nodes.
• Dataset nodes and connections.
• Independent evaluator nodes.
• Ollama model discovery.
• Native model execution engine.
• Exact-match accuracy evaluator.
• Normalized accuracy evaluator.
• Latency benchmarking.
• Multi-model comparison.
• Experiment persistence.
• Results dashboard.
• Coding benchmarks.
• Reasoning benchmarks.
• Hardware monitoring.
• Local LLM-as-a-judge.
• RAG evaluation.
• Agent evaluation.
• Model leaderboards.
• CSV and JSON exports.

Feature completion should be reflected in the repository as functionality is implemented and verified.

EvalForge20 follows these engineering principles:

Local-first

Core evaluation should run locally without requiring an external evaluation platform.

Modular

Models, datasets and evaluators should be independently extensible.

Reproducible

Every experiment should retain its configuration, dataset version and evaluation methodology.

Transparent

Every metric should have a documented calculation method.

Fair

Models should be compared under equivalent benchmark conditions.

Reliable

Failed evaluations and unavailable measurements should be reported honestly.

Developer-focused

The interface should simplify benchmarking without hiding the underlying technical details.

Contributions are welcome.

Areas where contributions can make an impact:

• New model provider adapters.
• New evaluation metrics.
• Benchmark dataset integrations.
• Performance measurement improvements.
• Hardware monitoring.
• Visual playground enhancements.
• Experiment management.
• Documentation and testing.

Suggested contribution process:

1. Fork the repository.
2. Create a feature branch.
3. Make a focused change.
4. Add or update tests.
5. Verify the implementation.
6. Submit a pull request with a clear explanation.

Please ensure new evaluators include documented inputs, scoring methodology and tests.

This project is intended to be open source.

Add the repository's selected license and corresponding LICENSE file before publishing or distributing the project.

EvalForge20 builds on the open-source ecosystem surrounding:

• Ollama — Local model inference.
• React Flow / XYFlow — Visual node-based interfaces.
• FastAPI — Python API development.
• Next.js — Frontend application development.
• Recharts — Data visualization.

EvalForge20
Visualize. Benchmark. Understand AI.

_{Built for developers who want to measure AI models, not just run them.}