TOOL PROFILE TOOLAI
Study language model training with a small model, from pretraining and SFT to tool use.
View full image
THE BIG PICTURE
MiniMind is an open-source project for learning and reproducing language model training. It publishes architecture, tokenizer, data handling, pretraining and adaptation code, making the full process easier to trace in a small model. It is suited to readers with Python and basic deep learning experience.
The current MiniMind-3 Dense model has about 64M parameters. Its MoE variant has about 198M total parameters and 64M active parameters. Historical MiniMind2-small models used 26M. The April 2026 mainline update changed architecture, tokenizer, data and inference interfaces, so older tutorials may not match newer weights.
Start with released weights to establish chat inference, then reproduce pretraining and SFT with the two mini datasets. Explore LoRA, preferences or tool-use training separately afterward. Code requires no subscription; hardware and cloud resources have their own costs. Knowledge, complex reasoning and long-context behavior remain constrained by model size and data.
A CLOSER LOOK
Explore product screens and published examples. Open any image for a closer look.
An interface example published by the author, showing multi-turn requests and results. It does not represent a test performed for this profile.
Project diagram connecting embeddings, attention and feed-forward layers.
Project diagram showing expert selection through routing.
Published Dense and MoE pretraining records, not a run performed for this profile.
Published SFT records. Training loss is not task accuracy.
Original repository logo.
WHAT YOU CAN DO
Get to know the capabilities. Find your own way to use them.
Dense and MoE implementations expose embeddings, grouped-query attention, RoPE, RMSNorm and feed-forward layers. Follow the tensor path and compare it with the architecture diagram. The current mainline aligns with the Qwen3 / Qwen3-MoE ecosystem.
The pretraining script reads JSONL text examples, computes next-token loss and saves weights. The mini dataset is useful for an initial reproduction; a larger version supports further training. Text length, cleaning and learning rate affect the result.
Full-parameter SFT continues from pretrained weights using conversations. Current mainline SFT data includes basic tool-call examples, and full_sft weights include corresponding basic behavior. Tokenizer and chat template must match training and inference.
A native LoRA implementation adjusts the model with a small set of additional trainable parameters. A merge-and-export workflow is included. Use it to study task or style changes while keeping the adapter matched to its base model.
MiniMind-3-MoE has about 198 million total parameters and 64 million active parameters per token. Compare routing and auxiliary loss with Dense. Fewer active parameters do not make the full set of weights occupy the same memory or storage as a 64M Dense model.
DPO adjusts behavior using paired preferred and rejected answers. The repository includes a script and data loading. Preference criteria shape the output; training loss alone does not establish correctness or reliability.
These methods introduce rewards and include generation, scoring and parameter updates. Study them after the basic path works. They require extra generation compute, and poorly designed rewards can teach the model to satisfy the scorer rather than solve the task.
train_agent.py supports GRPO / CISPO for multi-turn tool-use scenarios. Tool interaction trajectories become part of training, making it possible to study when calls are made, how results are used and what happens in later turns.
Current code uses the chat template, think markers and open_thinking to control reasoning content. It no longer follows older tutorials built around a separate train_reason.py. Displaying reasoning and producing correct answers need separate evaluation.
The project includes distillation, cleaning, deduplication and format preparation. Inspect how teacher outputs or training signals reach a smaller model. Teacher outputs can contain mistakes, and source data permissions need their own review.
Scripts record loss, learning rate and progress, and save resume state including model and optimizer. Use --from_resume 1 to recover an existing checkpoint. A training-resume checkpoint serves a different purpose from inference-only weights.
Use eval_llm.py for command-line chat or the Streamlit interface. WebUI scans model subdirectories under scripts, so Transformers-format models must be placed there first. It displays reasoning, tool selection and multi-turn calls.
A minimal OpenAI-compatible server supports reasoning_content, tool_calls and open_thinking. Documentation also describes Ollama, vLLM and llama.cpp paths. Check the model format and engine version before integration.
The project includes evaluation paths for C-Eval, C-MMLU and OpenBookQA and uses Apache 2.0 for code. Downloading code requires no subscription. GPU, storage or cloud services have their own costs. Use a separate fixed set of business prompts to check facts, refusals and repeatability.
MADE FOR YOUR WORKFLOW
For readers familiar with Python, tensors and basic deep learning. Trace one sample through loading, forward computation, loss and weight updates before moving to larger training systems.
Keep the model, evaluation prompts and budget fixed while changing SFT data or comparing LoRA with full fine-tuning. Record versions, settings, data volume, time and mistakes.
Inspect routing, auxiliary loss and resource use. Record total and active parameters, weight size and actual training memory separately rather than comparing architectures through a single number.
Start inference with released weights and later switch to your own model. Compare checkpoints with a fixed set of Chinese and English prompts and the same generation settings.
Use tasks with checkable results, such as arithmetic or time queries, to inspect arguments, returns and final answers. The author's tool-call demo does not replace your own tests.
The small model and concentrated code paths can support separate sessions on pretraining, SFT and evaluation. Prepare hardware, dependencies and data in advance, and retain failures as well as successful conversations.
YOUR FIRST STEPS
Clone the repository, install requirements.txt and separately install a matching PyTorch build. The README reference uses Python 3.10.16, Ubuntu 20.04 and CUDA 12.2. PyTorch is commented out in the requirements file.
For CUDA, confirm torch.cuda.is_available() returns True. Record revision, GPU model, available memory and generation settings. The author's eight-GPU workstation is a reference; a single-GPU workflow is supported.
Download the Transformers-format minimind-3 under the repository root and run python eval_llm.py --load_from ./minimind-3. This first checks the model, tokenizer and inference path together.
Place pretrain_t2t_mini.jsonl and sft_t2t_mini.jsonl under dataset. Pretraining text and SFT conversations use different fields. Inspect samples before adding your own data.
From trainer, run python train_pretrain.py. Current defaults include hidden size 768 and two epochs, with weights saved under the root out directory. Inspect loss, learning rate, time and memory before increasing data.
Still in trainer, run python train_full_sft.py. It defaults to loading pretrain weights and requires matching model settings. Add --from_resume 1 to the original command to recover a saved training state.
From the root, run python eval_llm.py --load_from ./model --weight full_sft. Fix prompts and sampling settings and inspect answers, errors and speed. Training questions should not be your only evaluation set.
For WebUI, place the Transformers model directory under scripts and run streamlit run web_demo.py there. Add LoRA, DPO or Agentic RL separately and keep data, logs and weights for each experiment.
It primarily publishes training code, models, data and tutorials. Command-line chat, WebUI and an online demo are also available. A hosted demo and your own training are different workflows; a cloned repository still needs an environment, weights or training data.
The current MiniMind-3 Dense mainline is about 64M. The 26M figure belongs to older models such as MiniMind2-small. Older tutorials and settings may not match new weights, so record a consistent code and model generation.
The MoE model has about 198M total parameters and 64M active parameters per token. All expert weights still need storage. Active parameters are not the weight-file size and do not directly determine training memory.
Pretraining can begin with newly initialized weights and learn text patterns. SFT usually continues from pretrained weights using conversations. LoRA changes how parameters are adapted. Running downloaded MiniMind-3 weights is inference, not your own training run.
No. The README describes one SFT epoch on a single RTX 3090 and its corresponding GPU rental. Current pretraining and SFT scripts default to two epochs each. Setup, downloads and other stages take additional time.
No. Eight GPUs describe the author's reference workstation. Single- and multi-GPU paths are provided. Whether a configuration fits depends on batch size, sequence length, architecture, precision and available memory.
Documentation mentions CPU and MPS, with substantial speed and compatibility differences. The reference environment is Linux with CUDA. Windows can add dependency, path and data-loading differences. Establish inference with released weights before attempting training.
The PyTorch entry is commented out in requirements.txt. Check that the installed build supports the intended CUDA environment, along with drivers and device access. Start with torch.cuda.is_available().
Not directly. Pretraining primarily loads text; SFT loads conversations. Similar names and extensions do not imply matching fields, templates or objectives. Inspect loaders, fields and truncation before supplying custom data.
Weights under out are used for inference or a subsequent stage. Resume files under checkpoints also include optimizer state and progress. Use --from_resume 1 for training recovery and the appropriate weight name for chat inference.
web_demo.py scans model subdirectories containing weights under scripts. Copy a Transformers-format model there if it only exists at the root. An individual .pth file is not a complete Transformers model directory.
No. The author's screenshot shows tool names, arguments, returns and answer layout. Arguments can be wrong and tool results need checking. Test invalid inputs, tool failures and result verification separately.
It indicates a closer fit to the training objective, not necessarily factual accuracy, correct reasoning or safer behavior. Pretraining and SFT plots also represent different objectives. Use held-out prompts and record mistakes and repeated outputs.
The code uses Apache 2.0, which generally permits commercial use with obligations including notices and license retention. Datasets, weights, teacher models and external services can have their own terms and need separate review.
Local training and inference can use local files. Experiment tracking such as SwanLab, hosted downloads and external tools introduce network access. Logs can include settings or text. Inspect scripts, logging options and tool configuration before processing private data.
MiniMind-V and MiniMind-O are related projects. dLM and linear-attention directions are experimental extensions linked through project discussions. They should not be presented as default capabilities of the MiniMind-3 text mainline.
KEEP EXPLORING
FROM THE COMMUNITY
Sign in to share your experience and help others find the right tool.
YOUR NEXT DISCOVERY
Scans content to determine if it was generated with AI
AICommit is an OpenAI-powered JetBrains IDE plugin offering AI-driven features like commit message generation, code optimization, explanation, documentation, conversion, and translation.
GPU Everything. Run anything Dockerized. Run autoscale Inference. Save costs 50-90%.
Cerebro is a versatile multi-model platform designed to empower enterprises in the responsible creation, management, and deployment of Generative AI applications 10x faster. It ensures meticulous governance and adherence to applicable regulations.
MAKE YOUR NEXT MOVE
Visit the official website to explore the latest features and get started.
