Researchers at PetroChina's Research Institute of Petroleum Exploration & Development (RIPED) report building a shale oil domain large language model on Qwen2.5-72B and Qwen3-32B base models, together with an LLM-driven agent for shale oil seismic processing and interpretation, describing the work in Petroleum Exploration and Development (Vol. 53, Issue 5, pp. 1289-1300, 2026). The team assembled a specialised corpus of roughly 21 billion tokens from open-access shale oil literature published between 2000 and 2025 plus a domain knowledge graph, and built around 58,000 supervised fine-tuning samples, of which 3,000 were authored and reviewed by more than 20 oil and gas domain experts.
The training pipeline runs in five stages. Continued pre-training inserts trainable task-aware expansion branches after the attention layer of every Transformer block in Qwen2.5-72B, mixing general and domain corpora at a 3:1 ratio that transitions to 1:1 to limit catastrophic forgetting, with a 32,768-token sliding window. Supervised fine-tuning uses QLoRA with NF4 quantisation, double quantisation and paged optimisers, plus a dynamic context-parallel strategy that assigns different parallelism degrees to samples of different lengths. On Llama3-8B with 8xA40 GPUs and 32k sequences, that strategy cut single-iteration time from 525 seconds to 122 seconds, a 4.30x speed-up and a 76.76% reduction in iteration cost; the supervised fine-tuning stage used 128 A100 GPUs for about 15 hours. Preference alignment uses Direct Preference Optimisation on expert-reviewed answer pairs, followed by Group Relative Policy Optimisation with rule-based rewards for numerical tasks and model-based scoring for open geological assessments.
A reflective retrieval-augmented generation loop replaces the usual single-pass retrieve-then-generate chain with generate, reflect, retrieve and regenerate: the model first answers from parametric knowledge, then uses that draft answer rather than the raw question as the retrieval query, so the search targets the specific knowledge gaps the draft exposes. Evaluation on 2,818 expert-selected objective questions and 146 expert-checked subjective questions shows the effect. The Qwen2.5-72B variant scored 86.35 on objective questions and 75.54 on subjective questions, against 72.22 and 15.43 for the base model and 77.32 and 12.12 for GPT-4o; adding retrieval alone lifted the fine-tuned model's subjective score from 20.55 to 74.40, far more than the roughly 5 points contributed by continued pre-training and fine-tuning.
The seismic agent wraps the language model as a scheduling hub in a five-layer architecture: a tool set, the language model, an intelligent service layer built on the Model Context Protocol, a results display layer and a data store. From 1,051 open-source geoscience projects the authors extracted a library of about 6,000 tools spanning 11 geoscience subdomains and 13 function tags, then selected 23 high-confidence tools built on SFM, FaultSeg3D, SeisCLIP, SeismicPro, SeisBench, Pyseistr and GFM. To suppress tool hallucination they generated roughly 60,000 layered question-answer samples with both correct and deliberately wrong calls, and fine-tuned Qwen3.5-27B on them. On a 184-question tool-calling test set covering single-tool calls, multi-tool selection, parallel calls and call refusal, the resulting model scored 94.377 overall against 78.565 for the base model, 92.275 for ChatGPT-5.4 and 90.775 for Kimi k2.6, and 88.00 on multi-tool orchestration versus 52.00 for the base model.
The authors note that existing oil and gas large models remain largely single-model, single-task, and present the agent as extending the language model from knowledge question answering to task execution. Future work aims to fuse core, well-log, seismic and engineering data into a unified cross-scale representation and reasoning system supporting reservoir evaluation, sweet-spot identification, development planning and fracturing parameter optimisation, moving shale oil large models from single-modal, single-task use toward multimodal, cross-scenario collaboration.