Kuaishou's own engineers reported their search test's gains

The test

A preprint titled 'Beyond Relevance: Improving User Engagement by Personalization for Short-Video Search', posted to arXiv on 17 September 2024, describes a search-personalization system its authors call PR2, built and deployed by engineers at Kuaishou Technology on Kuaishou itself, a platform the paper describes as having over 400 million daily active users.

What the evidence says

The paper states that all eight listed authors are affiliated with Kuaishou Technology in Beijing, making this a platform's own engineering team publishing its own production results, not an outside academic study of Kuaishou. The methodology section describes online A/B tests run directly in Kuaishou's live search system, trained on production search logs the authors describe as spanning tens of billions of impressions. The headline results, a reported 10.2 percent increase in click-through rate at position 10, a 20 percent increase in video watch time, and a 1.6 percent increase in search daily active users, are self-reported outcomes of Kuaishou's own internal experiment, not figures verified by an outside party or disclosed through any public transparency report.

The sample and the variance

Because the test ran inside one platform's production system on its own traffic, the reported gains describe what changed for Kuaishou's search users specifically, over the specific weeks and, for one long-term test, one quarter of online experimentation the paper describes, and cannot be assumed to transfer to another platform's search stack, user base or existing personalization baseline. The paper is a preprint; arXiv's own site states that hosted materials are not peer-reviewed, and no independent replication or platform transparency disclosure corroborating these specific percentages was found.

What to try next

A platform observer citing this paper should describe it explicitly as Kuaishou's own reported internal test of its search-personalization system, name the specific metrics improved, and avoid presenting the percentages as independently verified or as evidence about search personalization on other short-video platforms. This is a labelling discipline the paper's own affiliations make necessary, not a judgment on the underlying engineering work.

• Does the source label these percentages as Kuaishou's own internal A/B test results, authored by Kuaishou's own engineers?
• Has any figure from this paper been generalized to short-video search on a different platform without support?
• Has this preprint been published in a peer-reviewed venue, and does that version report the same figures?

A platform's engineers publishing their own production results as a preprint is a useful window into how a recommendation system was changed and why, but it remains a self-reported case study of one company's system, not independent evidence about short-video search generally.