A Diagnostic Evaluation of Small Language Models for Offline Writing-Feedback Tutoring in Brazilian Portuguese: A Study on Structural and Pedagogical Adherence under Public-School Infrastructure Constraints

Abstract

The adoption of generative Artificial Intelligence in education has relied largely on proprietary modelsb that depend on broadband connectivity, an assumption incompatible with the material reality of most Brazilian public schools. We report a controlled diagnostic evaluation of eight open Small Language Models (SLMs), executed strictly in a local, offline environment against a metalinguistic rubric derived from Ingedore Koch’s Textual Linguistics: 312 primary inference calls (8 models × 13 canonical scenarios × 3 repetitions) without any grammatical coercion mechanism, together with 100 counter-experiment calls, with categorical comparisons by Fisher’s exact test. Two failure regimes emerge. The first is zero-shot structural non-adherence: 100% typological divergence in the Llama 3.2 family (1 B and 3 B) and in Phi-3 Mini. For Llama 3.2 this behaves as a deterministic formatting bias, invariant both to sampling temperature and to the client transport layer and fully reversible by a minimal in-context demonstration (one-shot), which sets it apart from genuine compositional incapacity; the design is behavioural and does not establish the bias’s origin. The second is pedagogical superficiality in Qwen 2.5 3B-Instruct, the only structurally conformant model in the deployable range. We state its evidentiary status first: this is an exploratory single-model analysis whose instrument is a lexical screen over Koch’s terminology, so its figures corroborate the diagnosis rather than carry it. The model mobilizes the rubric terminology in only about half of the runs (51.3%), and it does so unstably, with a single scenario out of thirteen eliciting it in every repetition; two external teachers judging the same runs for metalinguistic focus credit 66.7% and 64.1% of them, agreeing with each other at κ = 0.72 but with the lexical screen at only 0.28 and 0.43, so the screen is a lower bound on the construct rather than a measure of it. Read against a specialist reference of 65 feedback turns from five experienced teachers, coded descriptively into eight mediation functions (κ = 0.83), the model’s mediation concentrates on format-forced recognition and open expansion, stays well outside the expert corrective core, and never models a repair or reinforces the student’s autonomy. We report this as a pattern and not a measurement: two independent codings of the same responses by external teachers converge on that shape, recording recognition and reflection at identical rates and the two absent functions at zero, while three of the corrective functions occur too rarely in this corpus for their reliability to be estimated. We conclude that, in the regime evaluated here (zero-shot prompting, local CPU-only inference, Q4 quantization, no grammatical coercion), off-the-shelf generalist models are insufficient for fine-grained writing-feedback tutoring in Brazilian Portuguese, which motivates specialized models fine-tuned on curated pedagogical corpora. The boundary is deliberate: a single in-context demonstration reverses the structural failure, so the finding characterizes an operating regime rather than a fixed limit.