User:Michael.C.Wright/sandbox/The problem

One of the hallmarks of an institution in crisis is that, far from preparing for the future, it is barely capable of managing the present.
– Ashley Rindsburg

For years various editors have complained about the lack of active reviewers at en.Wikinews. I am one of them. Below are my notes trying to make sense of the problem.

The problem defined

I don't believe one can truly evaluate a system from within the system (John Boyd). When you're part of the system, your view is restricted by the system’s rules, constraints, assumptions, and inherent biases. This internal perspective makes it challenging to identify structural problems, since your thinking and actions are shaped by the very elements that may need to be questioned or changed.

I do not consider myself fully part of the system yet, as a new reviewer with only a handful of reviews under my belt. Therefore I consider my observations of the system to at least be largely unrestricted by the system’s rules, constraints, assumptions, and inherent biases.

Per Boyd's understanding, outside information and fresh perspectives are needed to reinterpret circumstances, break from patterns, and make better decisions. Boyd believed that stepping outside the system enables a clearer view of its limitations, inefficiencies, and even unintended consequences, which are often invisible from within due to cognitive and operational biases.

I have yet to fully define The Problem. Below are notes I am collecting in order to organize my thoughts around that definition.

Our rate of publication is low and sporadic

This is possibly the most glaring sign of a problem. A news organization that publishes no news articles for several weeks straight is not a reliable source for news.

The EasyPeerReview widget automatically assigns articles to date categories when published. Therefore any errors in the numbers below could be caused by manual publication in which a date category is not assigned.

Our monthly rate of article publication (or not)
Month yr	# of articles published	# of days with 0 articles published
Category:January 2024	5	26
Category:February 2024	0	28
Category:March 2024	10	24
Category:April 2024	9	23
Category:May 2024	5	28
Category:June 2024	17	20
Category:July 2024	1	30
Category:August 2024	1	30
Category:September 2024	10	21
Category:October 2024	TBD	TBD
Category:November 2024	TBD	TBD
Category:December 2024	TBD	TBD

Stats from Wikimedia

Starting in mid-to-late 2008, en.WN had a steady drop in active editors.[1]
Starting in the first quarter of 2016 the net byte difference plummeted.[2]
Starting in the first quarter of 2016 the number of edited pages plummeted in a similar pattern.[3]
New registered users has been mostly trending down since 2006, but steadily since 2020.[4]

Note: For the second and third stats sites above (all are from wikimedia), you can toggle the display of bot edits on or off using the 'Editor type' button in the Filter/split section of the graph (on the left).

Too few reviewers or too few draft articles

We need to know the number of articles that are marked stale while in the review queue to more accurately identify the source of our problem as too few articles developed or too few reviewers. To do this, maybe we should do something like the following test for 90+ days:

When articles go stale in the review queue, use {{stale}}
If the article is abandoned, use {{aband}}
After two days of abandonment, instead of deleting the article, we do the following:
1. blank the content (it remains available in history but not readable otherwise)
2. add a new template + category, something like {{Stale without review}}

In this way we can better track the number of articles that go stale in the review queue without ever being reviewed.

Review and developing markup

Part of the delay in publication comes from the back-and-forth between reviewer and author(s).

Reviewers can't expand articles or add sources and remain uninvolved. If an unsourced statement is found, the reviewer could simply remove it but that might negatively impact the flow of the article or its coherence.

To help alleviate this, I think we should develop a set of templates specifically to be used before publication to allow authors and reviewers to quickly, clearly, and accurately communicate between each other.

I have developed {{Verify}} as one example.

Other useful templates may be:

{{Note}}
{{Copyright issue}}
{{Needs attribution}}

Question how to prevent edit conflicts. maybe better to write questions on article talk and ping the author(s)? Gryllida (talk) 10:48, 26 October 2024 (UTC)

If by edit conflict you mean where two editors click 'save changes' to an article at the same time? If so, I'm not worried about it. I don't see how using in-line markup in the article would increase the risk of edit conflicts. Reviewers should use {{under review}} to prevent other edits to the article and editors can use {{editing}}.

I look forward to the day when we are so busy, we have to worry about edit conflicts. ツ —Michael.C.Wright (^Talk/_Published) 13:26, 26 October 2024 (UTC)

I'm kind of worried as I would prefer talk page format to notes in draft. It means I have a place where to leave a message while keeping draft nice and clean. It also means I can reply to each concern separately using threaded messages on talk page. For my own drafts I would prefer to keep it that way. I believe consensus or a consent from draft author would be needed to change it to the 'verify' templates. Gryllida (talk) 03:14, 27 October 2024 (UTC)

I don't understand your hesitancy here.

It is already common to recommend that authors use hidden, in-line citations to aid the review process. {{verify}} can be the same thing, but not hidden.
I believe we could more effectively communicate between author and reviewer using markup in addition to the talk page.
Eventually requiring authors to clearly cite information will speed up the review process by improving the quality of early article drafts.
How does it benefit the project to prevent us from using {{verify}} due to lack of consensus or consent?
What else do we require an author's consent before doing?

I believe consensus or a consent from draft author would be needed to change it to the 'verify' templates. Why would you like to see consensus for this change, which directly supports core policy, but not feel you need consensus to unilaterally disregard core policy in your own, current, 5W-only review process? —Michael.C.Wright (^Talk/_Published) 14:04, 27 October 2024 (UTC)

Past conversations

One aspect of en.WN culture is the notion of institutional knowledge. I am not a fan, as it represents hidden or exclusive knowledge.

Organizations spend a lot of time and resources developing knowledge and capability. While some of it gets translated into procedures and policies, most of it resides in the heads, hands, and hearts of individual managers and functional experts. Over time, much of this institutional knowledge moves away as people take on new jobs, relocate, or retire.
– Ron Ashkenas, Harvard Business Review

Below are past conversations on the topic of too few active reviewers. The list should not be considered exhaustive.

Also keep in mind the list is inherently biased—I am looking for conversations where someone is complaining about the review system.

2018

'Wait just a bit longer...'

In my opinion, in at least the last six years, the 'just wait' crowd has not yet been proven right.

and I hope you forgive me for feeling unenthusiastic about the prospect of waiting just a bit longer, since that's essentially the argument we WN people have made since the days I was a bureaucrat on this wiki. IlyaHaykinson

I draw two conclusions from that observed pattern: (1) if we could do things that significantly and permanently increased our review capacity per reviewer (or to put it differently, per unit of review labor), we could sustain a significantly increased level of output. Pi zero (in direct response to above comment)

The following conversation echos many still being had.
The best way to prove the bad reputation of this site wrong is an increase of relevant, informative news stories, not useless denial of the obvious misery (a la Baghdad Bob). Any animosity or snobbishness towards the bigger sister is totally unproductive, in a situation where Wikinews is dying from a scarcity of fresh content. Gray62

Proposed solution by Pi zero: tweaking existing processes using 'semi-automated' tools.
The key to tweaking the parameters, as I see it (and as I must have mentioned before), is semi-automated assistance to streamline the various on-wiki tasks, including (though not limited to) both writing and reviewing. It's essential to follow the semi-automated route, where the essential human component is carefully preserved and nurtured, because that human component is part of both the quality, the robustness, and the ideals of the project. Pi zero

2013

It seems to me that the long-term viability of this project would be substantially improved just by attracting a larger readership. CMBJ

There is (averaged over time) as much demand for what we do as our reviewers can supply: increase our review capacity and, with a time lag, demand for review goes up until it saturates the available supply. Pi zero (in direct response to above comment)

I interpret what Pi zero is saying here is that with more active reviewers we'll get more articles written and conversely; with fewer reviewers fewer articles. We've known for years that reviewer capacity is a key to solving The Problem.

2010

Review waits are dismal. Many of our articles are coming from people as-yet too inexperienced to receive reviewer, or who do not have English as a native language. A supply of people regularly writing has dried up. Blood Red Sandman

'Bust' of boom and bust blamed on the lack of AGF
We have a long-term problem here. A (sort of) boom and bust cycle, playing out on a scale of multiple years. Our community dynamic is unstable, resulting in the boom that peaked early this year, and the bust we're in now, because we (necessarily) don't have AGF. No AGF means that unpleasantness doesn't get to hide behind superficial forms. In good times, our authenticity makes this a bright place (quite refreshing compared to WP), leading to more new contributors and more good times. When things go sour, the reverse happens. Boom and bust. Pi zero

quality energizes the project
"If the choice is between ..." There is no such choice. There is no choice between quality control and anything else. Quality control is a sine qua non; without it we'd be an inferior sort of community blog, with no good reason for existing. We want to improve by getting things out the door faster. We want to improve by growing. Quality control is an inherent feature of the landscape where we want to do those things. Put it this way: without quality control, the project would not grow, it would wither and die. Pi zero

Your contention is that [the sine qua non is our quality control. My response to the current "crisis" is to increase the size of the community, while yours is to defend the standard of quality. Neither is necessarily better than the other, but I think mine can lead to yours, while I do not see how yours can lead to mine. Amgine (in direct response to above)

2008

Near the inception of the 'modern review' system

Since the changes to the reviewing system in late July/early August, (which led to compulsory peer reviews) the number of new articles being published appears to have fallen dramatically. It also takes much longer to get an article published than it used to, due to the need to involve a third party. GW_Simulations