How AI-Powered English Practice Made Learning Progress Visible

Context

Why measure learning while it's happening?

Traditional assessment often reveals a problem only after a lot of instructional time has passed. Jinso's working premise is that institutions make better instructional decisions when practice itself produces usable evidence of learning. If a learner is struggling today, evidence from current practice could support action before the next high-stakes assessment.

A closed-loop learning model (5 steps)

1. Institutional objectives: Define the skills and outcomes that matter.
2. Practice: Complete curriculum-aligned practice.
3. Measure: Create evidence from learner interactions.
4. Learn and adapt: Use evidence to adjust instruction.
5. New evidence: Test whether the next cycle improves.

This study evaluates one part of that loop: whether recorded participation in AI-supported practice is associated with estimated change in learner performance.

Methodology & Sources

From conversation to inspectable evidence

The historical data included detailed learner conversations and lesson context. There was no randomized experiment, though, and no common external assessment given at both the start and end of the observation period. Learner conversations and lesson context were turned into CEFR-tagged performance evidence, then task-level observations, then estimated ability. Observations were classified as successful independent performance or unsuccessful, clearly elicited attempts. Insufficient or unreliable evidence wasn't scored.

Study design

For each learner, recorded participation was the count of qualifying assignments during the observed window. A recorded assignment needed at least two learner messages. Estimated change was the difference between the first and last modeled ability estimates in that window. The main model adjusted for starting ability, observation length and classroom.

Who was included

The source data came from four partner institutions using Jinso as part of instruction. Names have been removed to protect partner confidentiality. One institution contributed two analytic cohorts at different levels, so the pooled analysis has five cohorts.

*Institution-level counts are distinct students with at least one exchange submitted for scoring. They are not enrollment totals or analytic sample sizes.

Validation and limits

Checks included computational reproducibility, comparisons between observed and predicted response patterns, and sensitivity analyses across coverage rules, observation windows and specifications. These checks support how the fitted outputs are interpreted. They don't establish accuracy against an external assessment. The design can't rule out unmeasured differences between learners, classrooms or implementations. It estimates association, not causal impact.

Results

Participation and estimated change across cohorts

Estimated association per five recorded assignments, in model-relative baseline standard deviations of first-to-last estimated change. (Optional: render as a forest-style chart with 80% CI bars, built in SVG or HTML with accessible text. The table below must also be in the HTML.)

Four of the five cohort estimates were positive. Cohort C had the clearest positive estimate in the primary cohort comparison, with an 80% interval that didn't include zero. The pooled 80% interval narrowly included zero, and the corresponding 95% interval (−0.039 to +0.149) was wider.

How to read this table: Confidence intervals show a range of estimates consistent with the model and the available data. An interval that includes zero means the analysis doesn't rule out a null association at that interval level. 80% intervals have lower coverage than 95% intervals. They don't add observations or increase certainty. Institutional displays used their original classroom and cohort rules, while the pooled analysis used a harmonized rule and sometimes different samples.

What the cohort-level patterns looked like

• Partner Institution A, lower-level cohort: 89% of learners had a higher final point estimate. Lower participation (2–27 assignments; n=74) had a median increase of +5.99 logits, and higher participation (28–37 assignments; n=67) had a median increase of +6.66 logits. These are descriptive group summaries from the same retrospective model.
• Partner Institution D: A raw descriptive comparison showed average estimated change of +0.21 baseline SD among lower-participation learners and +0.56 baseline SD among higher-participation learners. After adjustment, the higher-minus-lower difference narrowed to +0.10 SD (80% CI −0.08 to +0.27).
• Partner Institution C: Among 141 learners with repeated estimates, 66% had a higher final ability estimate, and the median estimated change was +0.30 model logits. A cohort-specific participation analysis estimated +0.24 logits per five recorded assignments (80% CI +0.02 to +0.45). This cohort-specific logit scale shouldn't be compared directly with the pooled baseline-SD estimates.

Using learner evidence to guide intervention

The practical value of the measurement system isn't just producing a score. It surfaces recurring patterns that academic teams can investigate and address while instruction is still underway. Example: a learner repeatedly leaves out required forms when building questions. In an academic review across 916 question-asking attempts, 28% failed, and 14 of 145 learners failed three or more times across at least two lessons. Teachers review the relevant grammar and tie follow-up practice to current instruction. Later learner work can then be checked for improvement, and the same task-failure rate can be measured again for the same classrooms.

How to read the findings

• Consistent with a positive relationship. The pooled analysis and most cohort estimates point in a positive direction. After adjusting for the variables in the analysis, learners with more recorded participation tended on average to show greater modeled change.
• Not a causal estimate. Learners who participate more may differ in motivation, attendance, teacher support, prior exposure or other factors the models don't fully capture. The study supports an association. It doesn't show that assigning more practice will by itself produce a specific amount of improvement.
• Model-derived outcomes. The outcomes come from retrospective model fits. The reported intervals don't cover every source of uncertainty or model misspecification. Task alignment to CEFR was modest in several cohorts and absent in one primary analysis, so task labels shouldn't be read as direct learner CEFR boundaries.
• Different analysis versions answer different questions. A compatible growth-slope analysis estimated +0.0009 baseline SD per 30 days per five assignments in 516 learners across 92 classrooms (80% CI −0.0088 to +0.0106; 95% CI −0.0145 to +0.0164). The samples and estimands differ.
• Short follow-up. One partner cohort with short follow-up showed weaker learner-level first-to-last change in a separate institutional display. Among 191 learners with repeated dates, 45.5% had positive point changes, median follow-up was 10 days, and median change was −0.111 model logits.

Looking Ahead

The next study should be more prospective, standardized and integrated

The strongest conclusion from this retrospective work isn't that the effect size is settled. It's that there is a signal worth testing more rigorously in prospective implementations. Recommended next steps:

• Collect more data from implementations where Jinso practice is explicitly built into the curriculum.
• Use common external assessments at defined baseline and follow-up points where feasible.
• Set assignment exposure, observation windows and cohort inclusion rules in advance.
• Track implementation variables such as teacher use, scheduling and instructional follow-up.
• Keep reporting uncertainty and sensitivity results alongside headline estimates.

Confidentiality note (small): Partner institutions and binational centers are anonymized throughout. Institution labels only distinguish study cohorts and don't identify participating organizations. Source analyses were completed and synthesized in September 2026.