The advocacy email included Sophie's
Freexian email address. Some people know
Raphael Hertzog has some connection with
Freexian, which he founded and when they see
Sophie's email address, they will realise there is some connection.
Nonetheless, the inclusion of
Sophie's email address at
Freexian is not the same as a declaring they are a married couple
running a business together.
If
Debian is going to be honest with our community and our values then
we need to declare things like that.
Legal framework for handling cults in France
In 2000, the French parliament passed the
About-Picard Law 2001 to criminalise some of the most extreme
cases of exploitation and brainwashing.
French business records show us that
Freexian SARL is located just north of
Saint-Etienne. That is a small city about an hour away from
Lyon.
I felt that detail in the story is important for another reason. When
the rugby team from
Australia plays in
France, for example, at the recent Rugby World Cup 2023, they
stay in Saint Galmier, which is immediately to the north of
Freexian.
When rogue Debianists attacked my family, was it inspired by resentment
of Australia's rugby team?
History of Freexian antitrust behaviour and unfair competition in business
In the United States, unfair business practices are prosecuted under
criminal law. This is referred to as antitrust law in America or
anti-competitive behaviour or unfair trading in Australia.
In France these are civil law disputes. In the French language, the
term concurrence deloyale is used to describe antitrust
behaviour.
In this email from 2005,
Raphael Hertzog is using the organisation name "Debian" in
the message headers and he is using the names "Ouaza" and "Freexian"
in the message signature at the bottom. In other words, he uses
the names of these organisations interchangeably. He is
exerting influence over other volunteers who are not employees
of his company:
Subject: Re: WTF: Debian security, ex. Linux kernel vulnerabilities
Date: Wed, 21 Sep 2005 08:41:50 +0200
From: Raphael Hertzog <raphael@ouaza.com>
Organization: Debian
To: Martin Schulze <joey@infodrom.org>
CC: security@debian.org, leader@debian.org, debian-private@lists.debian.org
Le mardi 20 septembre 2005 à 16:21 +0200, Martin Schulze a écrit :
> The proper contact would be the security team.
Hi Joey,
I understand that it's difficult to respond to such aggressive mails.
However I believe that the concerns raised are reasonnable.
Andreas he's not the first one detecting very big delays in security
updates. We have all read your various blog posts when you had troubles
with the infrastructure in june... and you're aware that the delays have
been recently mentionned on a LWN article (and now the infrastructure
should be working fine).
It looks like the security team has a real problem in manpower (even if
several people are listed on the team, we need more people which stay
active like you do). And I don't see any evidence that work is done to
resolve this problem.
Please take a few minutes to explain what's planned to fix the various
issues and don't hesitate to ask for help, we have plenty of people
willing to help (maybe you can find skilled volunteers among the
secure-testing team?).
Cheers,
--
Raphaël Hertzog -+- http://www.ouaza.com
Freexian : des développeurs Debian au service des entreprises
http://www.freexian.com
Subject: Events at the DebConf Dinner
Date: Fri, 19 May 2006 10:41:19 -0500
From: Anthony Towns <aj@azure.humbug.org.au>
To: debian-private@lists.debian.org
CC: ted@reactor-core.org
Hi,
As promised, details from the dinner:
Ted was seen walking ...
[ ... snip ... ]
As I understand it,
Amaya Rodrigo Sastre asked
Fabio who the lady was, and was led to believe
that she was a prostitute.
[ ... snip ... ]
One of the volunteers, Ana Guerrero, spoke to Hilda and suggested that
it might be a good idea for her to leave.
[ ... snip ... ]
... and the question of whether she was or was not a prostitute was
not raised. As I understand it, at this point Hilda did leave, however
was encouraged to return by Ted and his long-time friend and co-worker,
John Sokol, a former developer of 386BSD and an attendee at DebConf.
At some point, I believe shortly after she returned, some attendees then
decided that it would be all right if she stayed, but Ted left. At this
point Holger and Moray, as mentioned above, manhandled Ted across the
dining hall to the door, where they were intercepted by John. At this
point the rest of the attendees noticed that something was going on, and a
group gathered around ...
[ ... snip ... ]
at which point Ted told him to "Get off me you fucking Nazi".
[ ... snip ... ]
I escorted Ted and
Hilda back to their seats to separate them from the other attendees who
were clearly angry, and tried to determine what was going on. At about
this time, Amaya and Fabio had a heated argument in Spanish, which Amaya
later indicated involved Fabio claiming that Hilda was not a prostitute,
and that he had never said anything that should give that impression.
I also spoke with Moray and Holger, who by this time were talking with
Bdale and some other developers who hadn't been involved, and indicated
that grabbing people to throw them out of the room was entirely wrong. At
this point they both remained extremely angry at the situation, and had
very little to say. I don't believe resorting to violence is remotely
appropriate within Debian, and had the opportunity to speak further about
this with Moray this morning, and hope to have a similar opportunity to
speak with Holger shortly.
[ ... snip ... ]
The report leaves out one key detail:
Amaya Rodrigo Sastre, who was present at the moment the rumour
started about
Ted Walther bringing a prostitute to the dinner, was actually
sleeping with
Holger Levsen. This is a common pattern throughout the history of
Debianism, the girlfriends are there to start gossip and the
boyfriends get worked up into a rage and attack people to please
their girlfriends.
Shortly after this, at
RMLL in 2006, a group of Debian Developers (joint authors) living in
France or having a strong connection to
France decided to form a local non-profit association under French law.
The name of the association is simply
Debian France.
Please write a list of 5 Debian Developers you would like to kick out of the project.
Enrico Zini becomes known as an attack dog who is used
to spread rumours and inflict punishments on other volunteers.
He is eventually rewarded with a contract from
Freexian after attacking me on multiple occasions.
Think about it for a moment, if your dog bites five people,
the authorities are going to have it put down. When
Enrico Zini wants to hit people with a hammer, they start
talking about whether to use a big hammer or a little hammer.
On 17 April 2011, on the same day Carla and I got married
Adrian von Bidder-Senn died in
Switzerland and it was discussed like a suicide. Read the
detailed history of the death. This death marks a significant
step change in the rudeness of certain people towards my family and I,
leading to various plots and conspiracies in the years that followed
the death.
Think about how horrible it is to recall a suicide like this on your
wedding anniversary each year.
On 5 May 2011,
Raphael Hertzog registered the domain name
debian-handbook.info and began using the name in his
email signature. Nobody has ever made any dispute about this
domain name but they viciously attack other people who register
domain names for our Debian work.
Here is an example of
Raphael Hertzog interacting with Dropbox about secretly
putting binaries into user home directories. The email has
his domain name debian-handbook.info at the bottom.
Subject: [rian@dropbox.com: Re: Dropbox Debian Package]
Date: Fri, 28 Oct 2011 10:45:52 +0200
From: Raphael Hertzog <hertzog@debian.org>
To: debian-private@lists.debian.org
[ ObPrivate: I'm sharing a private mail that I don't want to post
publicly. ]
Hello,
when I packaged nautilus-dropbox, I patched it so that the non-free
software is downloaded and installed in /var/lib/dropbox
instead of ~/.dropbox-dist/. Upsream is unhappy with this because
it means they cannot auto-update the version of dropbox
on user's system.
I have argued using several arguments: first it was not compliant
with the FHS, that software ought to be installed by the admin
and run by the user, and that software should not change
without the user's consent. I added that duplicating the binaries on each
user's account was non-optimal. I also said that it was not good security
wise to have the binaries owned by the user. I might have been a bit
quick in asserting that.
In any case, I wonder what other people think, am I right in
persisting that the official Debian package should not allow
auto-update of the software by installing a copy in $HOME by default?
If yes, what other arguments would you bring forward to convince
them given the response they made to me so far?
Cheers,
PS: The attached mail should never be disclosed.
--
Raphaël Hertzog ◈ Writer/Consultant ◈ Debian Developer
Pre-order a copy of The Debian Administrator's Handbook and
help liberate it: http://debian-handbook.info/go/ulule-rh/
In 2012,
Raphael Hertzog became president of the non-profit association
Debian France. He had this role at the same time that he was
running his private company using the Debian trademark. Both the
private company and the non-profit association were using the
trademark without permission from the wider body of joint authors.
This is one of many examples of the French developers simply going
out and doing something spontaneously without waiting for authorisation.
In the same year, French developer
Lucas Nussbaum was elected for a one year term as leader of
Debianism.
Immediately after being elected,
Lucas Nussbaum announced he was taking a vacation by sending
an email to the debian-private (leaked) gossip network.
Of particular interest in the scandal about
Debian France and
Debian LTS, we can see the trademark@debian.org email
address is also on CC:
Subject: [VAC] 28/04 -> 04/05, French Alps
Date: Fri, 26 Apr 2013 19:15:09 +0200
From: Lucas Nussbaum <lucas@debian.org>
To: debian-private@lists.debian.org
CC: trademark@debian.org
Hi,
I'm taking a little break next week. I'll read email at least once every
two days and deal with the urgent things, but I'll postpone non-urgent
things until after that.
I offer beer + key-signing to anyone willing to visit, but be warned
that this involves 1.5 hour of ski touring to reach where I'll be staying.
(GPS: 45.549, 6.918)
Lucas
--
Please respect the privacy of this mailing list. Some posts may be declassified
3 years after posting as per http://www.debian.org/vote/2005/vote_002
Archive: file://master.debian.org/~debian/archive/debian-private/
To UNSUBSCRIBE, use the web form at .
He shares his GPS coordinates in the email to debian-private.
For all those spreading conspiracy fact about military infiltration of
Debianism, could this be the proof?
In January 2014,
Red Hat announced they had acquired CentOS and given jobs
to the CentOS developers. This created a motive for people to
provide a service similar to RHEL and CentOS but based on Debian.
Somebody who follows
Debianism closely will look at that email and understand there are
some conflicts of interest. Somebody external to the organisation or
somebody reading these documents in the distant future may not be
aware of the conflicts of interest. That is why it is vital for the
conflicts of interest to be repeated in each critical piece of
correspondence where they are relevant. In this case, with the leader of
Debianism also being a French person and likely member of the
Debian France association that was under scrutiny, it may have been
better for
Lucas Nussbaum to delegate the examination of the relationships to
another subordinate who is well outside the clique of French developers.
Raphael Hertzog began using the trademark like this for his private
business
Freexian SARL at exactly the same time he was acting on behalf of
the non-profit association
Debian France in their negotiations about using the trademark in the name
Debian France, becoming a "Trusted Organisation" and handling assets
on behalf of the global body of joint authors. He was aware of various
discussions about the strengths and weaknesses of the strategy
joint authors had developed for protecting the trademark.
Raphael Hertzog was privy to private discussions about the rights
of individual joint authors and third party organisations wanting to
use the trademark for economic benefit. At exactly the same time those
discussions were taking place, he cleverly designed the Debian LTS
service name and began using it to entice other organisations to sign
contracts with his external business.
On 14 April 2014, the French developer
Lucas Nussbaum was elected for a second year as leader of
Debianism.
On 12 May 2014, the Debian Project News email is distributed by
French developer
Cédric Boutillier. These newsletters are distributed very widely
to people who don't subscribe to the email discussion lists.
Cédric Boutillier includes comments about
Freexian's project
Debian LTS:
Subject: Debian Project News - May 12, 2014
Date: Mon, 12 May 2014 22:27:19 +0200
From: Cédric Boutillier <boutil@debian.org>
To: debian-news@lists.debian.org
------------------------------------------------------------------------
The Debian Project http://www.debian.org/
Debian Project News debian-publicity@lists.debian.org
May 12, 2014 http://www.debian.org/News/weekly/2014/08/
------------------------------------------------------------------------
Welcome to this year's eighth issue of DPN, the newsletter for the
Debian community. Topics covered in this issue include:
* Debian members vote to accept a code of conduct
* Registration open for DebConf14
* SPARC removed from Jessie
* Promising future for Debian in embedded systems
* Bits from the Release Team
* Bits from the systemd + GNOME sprint
* Other news
* Upcoming events
* New Debian Contributors
* Important Debian Security Advisories
* New and noteworthy packages
* Work-needing packages
* Want to continue reading DPN?
Debian members vote to accept a code of conduct
-----------------------------------------------
Just after the election of the Debian Project Leader, Debian members
were called by Kurt Roeckx [1], Debian secretary, to vote on a general
resolution for a code of conduct proposed by Wouter Verhelst. This code
of conduct [2] promoting respect, good faith, collaboration, conciseness
and openness has been adopted by Debian Members [3]. It can be modified
via further general resolutions. More details about the results of this
vote can be found on the page of the website dedicated to this general
resolution [4].
1: https://lists.debian.org/debian-devel-announce/2014/04/msg00004.html
2: http://www.debian.org/code_of_conduct
3: https://lists.debian.org/20140428213635.GA10933@roeckx.be
4: http://www.debian.org/vote/2014/vote_002
[ ... snip ... ]
Bits from the Release Team
--------------------------
Regular security support for Debian GNU/Linux 6.0, "squeeze", will be
terminated [17] on May 31, 2014. A new suite named "squeeze-lts" and
containing only two architectures, i386 and amd64, will be made
available with support extended until February 2016 to provide a five
year support cycle. A reminder that the scheduled freeze date [18] for
Jessie has been set for six months from now on Wednesday, November 5,
2014. In the same message, Niels Thykier reported on the Release Team's
architecture meeting [19], held on April 12, 2014, about the status of
the architectures and considering their suitability for Jessie.
Recursive auto-removals have returned, with warnings to the maintainers
of the involved packages prior to removal.
17: https://lists.debian.org/debian-security-announce/2014/msg00082.html
18: https://lists.debian.org/debian-devel-announce/2013/10/msg00004.html
19: https://lists.debian.org/debian-devel-announce/2014/05/msg00000.html
Bits from the systemd + GNOME sprint
------------------------------------
Jordi Mallach sent bits [20] from the Debian sprint where the Debian
GNOME core team and systemd Debian maintainer gathered in Antwerp,
Belgium. Over two days, the ten participants discussed a variety of
topics to do with systemd and GNOME integration in Debian. After some
improvement in the packaging workflow for systemd, version 208 of
systemd has been uploaded to experimental. The GNOME team initiated
several transitions, improved the status of GNOME 3.12 in Debian [21],
and discussed the feasibility of having Debian Jessie shipped with 3.14.
The participants also jointly discussed how to configure and start
display managers, and may have come to a working solution to this
problem, which is complicated by the number of packages providing
display managers and init systems. They also used this opportunity to
sign new, stronger GnuPG keys in order to help the effort to abandon
older and weaker keys [22]. The participants thank the sponsors, most
notably INUITS, which provided the venue, and Debian and its donors for
covering the travel expenses for five of the attendees. A few of the
attendees were kindly sponsored by their employers.
20: https://lists.debian.org/20140501232614.GA29827@oskuro.net
21: http://www.0d.be/debian/debian-gnome-3.12-status.html
22: https://lists.debian.org/20140303181359.GA68761@gwolf.org
[ ... snip ... ]
On 27 May 2014,
Axel Beckert (XTaran) creates a section in the official Debian
wiki server for Freexian SARL's "Debian LTS" service
(wiki log).
For organizations that cannot contribute their own workforce to the LTS team, Freexian — a French company owned by Debian Developer Raphaël Hertzog — offered to act as an intermediary between organizations interested in funding LTS work, and individual Debian contributors preparing LTS security updates. For more information, see Freexian's description of their proposal.
In the same announcement, they pretend that
Debian LTS is being created by volunteers.
We would like to thank the volunteers who joined the LTS team
On 8 July 2014,
Lucas Nussbaum, in his role as leader of
Debianism, publishes an email with a list of four non-profit associations
authorised to hold money and other assets on behalf of
Debianism. The
Debian France association is included on the list.
In November 2014,
Raphael Hertzog admits that the Code of Conduct is used for
censorship. When people ask questions about
Freexian SARL business practices with the Debian trademark /
Debian LTS service name, the developers paid by
Raphael Hertzog can start a pestering campaign sending
messages to "debian-moderation".
Subject: Host COC related discussions on a new list
Date: Wed, 26 Nov 2014 09:37:50 +0100
From: Raphael Hertzog <hertzog@debian.org>
To: Miriam Ruiz <miriam@debian.org>
CC: debian-private@lists.debian.org <debian-private@lists.debian.org>
Hi,
On Tue, 25 Nov 2014, Miriam Ruiz wrote:
> We seem to be moving in circles, with the same arguments and
> contra-arguments all the time. The current recursive pattern we are
> having is:
Indeed, let's see if we can improve this. For example by moving
COC-related discussion to a new list nicknamed
debian-moderation@lists.debian.org.
> a) Person X says something inappropriate, without consciously wanting
> to offend or without acknowledge that whatever they say might be
> offended
> b) Person Y complains about it and say that it's not appropriate
That person Y would reply privately to the person and put
debian-moderation@lists.debian.org in copy. If enough persons do the
same, X is more likely to apologize.
> c) Either person X apologizes or a significant number of other people
> acknowledge that it was indeed inappropriate, then try to turn page
> ang move on
> d) Other people become upset that person X's comment was critisized
> and turn the debate into an abstract discussion about censorship
At the same time, if we have other people who dislike the message
of Y, because they don't agree, the discussion would stay on
debian-moderation (or more likely there would be no discussion at all
because those persons are unlikely to subscribe to debian-moderation
at all).
What do you think ?
debian-moderation@lists.debian.org would only allow subscriptions
by Debian contributors (to avoid external trolls). It could be watched
by listmasters to send official warnings when they see someone has gone
too far.
I'm not sure about the archives. I would like to keep them public but I'm
not convinced that it's necessarily a good idea.
Cheers,
--
Raphaël Hertzog ◈ Debian Developer
Support Debian LTS: http://www.freexian.com/services/debian-lts.html
Learn to master Debian: http://debian-handbook.info/get/
--
Please respect the privacy of this mailing list. Some posts may be declassified
3 years after posting as per http://www.debian.org/vote/2005/vote_002
Archive: file://master.debian.org/~debian/archive/debian-private/
To UNSUBSCRIBE, use the web form at <http://db.debian.org/>.
On 29 January 2015, an official meeting of
Freexian was conducted. The minutes were filed with the local
tribunal and they are
available here. The first thing to note is the appointment of
Sophie Brun Hertzog as a director. Notice that here she is using her
married name while in
Debianism she is one of the women deceiving people with
the use of her maiden name.
The second thing to note from the meeting minutes is the fact
these people are paying themselves a salary and giving themselves
five weeks paid vacation each year. Remember the people who killed
themselves? Remember the people
sleeping on the desks at TU Darmstadt? Did they ever get a vacation or
were they victims of exploitation?
In 2015,
Raphael Hertzog ceased being president of the non-profit
Debian France and became an ordinary member of the board.
Nicolas Dandrimont became the new president of the non-profit
association.
In 2015,
Sophie Brun Hertzog left her career as an industrial engineer and
began working in technology as part of her husband's company
Freexian.
Another blog post from Hertzog tells us
they pay people EUR 75 per hour. The blog post alerts us to
various concerns about the way
Debianism has fallen into disrepute. Hertzog states that to
receive money, "you are Debian Developer or a Debian Maintainer".
In other words, they realize they can negatively impact somebody's
economic situation by disparaging their ability as a Debian Developer.
The same checklist states:
you must respect the Debian code of conduct and respond to queries about your work from fellow community members;
The implication is that Debian's
Code of Conduct gaslighting requires unpaid volunteers to be available
twenty four hours per day, seven days per week or they will impede
your employability. This is a form of blackmail that enables
modern slavery.
When people talk about the
Code of Conduct gaslighting in
Debianism, they try to imitate the language of professional
organisations but what they are really telling us is that we are
not allowed to ask questions about money.
Under copyright law, all joint authors are entitled to an equal
share of every donation. We are all entitled to see accounts for
the donations received, including the names of the donors. All those
details are hidden from us. From time to time, emails like this
appear from joint authors who are blissfully unaware of what is really
happening with the money:
Subject: Debian funding model
Date: Thu, 26 Nov 2015 12:34:26 +0800
From: Paul Wise <pabs@debian.org>
Organization: Debian
To: debian-private <debian-private@lists.debian.org>
On Wed, 2015-11-25 at 22:35 -0200, Thadeu Lima de Souza Cascardo wrote:
> The Conservancy has been put in this critical situation because they
> depended on companies (or their surrogate) for funding.
I wonder what Debian's current funding model is; clearly we have both
corporate funding (mostly of DebConf) and individual donations, but I
wonder what the proportion is, whether we should be shifting more of
that towards funding based on individual supporters and what the
consequences of a shift towards that could be and how it could work.
Could the auditors or DPL comment on this?
--
bye,
pabs
https://wiki.debian.org/PaulWise
In June 2016, people begin a campaign of defamation against
Dr Jacob Appelbaum. Of particular note was an email that
Enrico Zini sent to a journalist demanding that news reports
should not question the credibility of the gossip and they should
present the gossip as fact.
The defamation conspiracy was so intense that people vandalised
Dr Jacob Appelbaum's home in Berlin.
Paul Tagliamonte suggested using the Debian Press Team / Debian
Publicity Team gangsters to spread the defamation as far as possible.
Notice how Tagliamonte is always
top-posting in this email thread. This is against the convention
of replying in-line that is normally followed in
Debianism. The top-posting is often a sign that somebody is sending
the emails from a mobile device while at their workplace. In the case of
Paul Tagliamonte, his workplace was the White House.
This is significant because
Freexian subsequently employees a member of the Debian Publicity Team
who they can use as a weapon of defamation against commercial rivals.
Subject: Re: Expulsion of Jacob Appelbaum <error>
Date: Tue, 21 Jun 2016 07:37:43 -0400
From: Paul R. Tagliamonte <paultag@gmail.com>
To: Russell Coker <russell@coker.com.au>
CC: Debian Private List <debian-private@lists.debian.org>
Seems like a thing the Press team could help coordinate.
On Jun 21, 2016 7:27 AM, "Russell Coker" <russell@coker.com.au> wrote:
http://www.itwire.com/business-it-news/open-source/73441-appelbaum-banned-from-debian-events-after-sexual-misconduct-charges.html
Well the DPL has decided to go public about it after all.
In future I think it would be better to have a plan for these
things. To have an argument on this list about whether information
should be released while the DPL is sending it all to a journalist
isn't the way things should go.
On 21 June 2016 8:57:56 PM AEST, Jonathan Dowland <jmtd@debian.org>
wrote:
>Please do not CC me, I am subscribed to the list.
>
>On Tue, Jun 21, 2016 at 08:04:01PM +1000, Russell Coker wrote:
>> Mattia suggests that it's already public that he was expelled for
>anyone who
>> knows how to interpret it.
>>
>> Why can't we just state it outright instead of leaving clues in
>Wikipedia?
>> The social contract says that we won't hide problems. Why are we
>trying to
>> hide this problem?
>
>I don't think Debian has taken ownership of whatever is written on
>Wikipedia,
>but we certainly aren't hiding it: It's there on the NM page in plain
>sight.
>There's a big difference between hiding something and making an active
>effort
>to draw attention to it.
>
>The action that has been taken has been for the sake of the safety and
>well
>being of other Debian members, NOT as some kind of white-knight
>publicity
>stunt. There's no *need* to shout it from the rooftops as that
does not
>further the former goal.
--
Sent from my Samsung Galaxy Note 3 with K-9 Mail.
Here is one of the messages that
Enrico Zini sent to a journalist:
Subject: On coverage of Abbelbaum being "banned" from Debian
Date: Wed, 22 Jun 2016 09:34:50 +0200
From: Enrico Zini <enrico@enricozini.org>
To: andrew.matler@itwire.com
CC: debian-private@lists.debian.org
Dear Editor in Chief of iTWire,
you may want to do something about this article by Sam Varghese on
Debian revoking membership of Jacop Appelbaum:
http://www.itwire.com/business-it-news/open-source/73441-appelbaum-banned-from-debian-events-after-sexual-misconduct-charges.html
While the first part is factually correct in its DPL quote, the article
ends with baseless hints of Debian and Tor having fallen victims to
manipulations by GCHQ psyops.
[ ... snip ... ]
At the same time, I was employed by one company in the UK and I was also
running another company based in the UK. The UK was still part of the
European Union. My employer, the other company and
Raphael Hertzog's company
Freexian SARL were competitors.
In May 2017,
Chris Lamb and I both went to the
OSCAL conference in Tirana,
Albania for the first time. The
OSCAL conferences were run on weekends as a social / hobbyist
activity.
It was a very relaxed event and one of the Albanian organisers,
Elio Qoshi, who was 24 years old, brought his girlfriend. I
only discovered about this a few months later: the girlfriend was only
15 or 16 years old. Here is a picture where
Freexian employee/contractor
Chris Lamb and
Elio Qoshi are pictured together at the
Red Hat table.
I think it was July 2017 when other women told me about the
underage scandal. Another few weeks later, the man responsible,
Elio Qoshi, resigned from his role as a
Fedora ambassador.
Red Hat, who are responsible for
Fedora, did not give any warning about the situation
to other volunteers. See the reports about the
Albanian female whistleblowers for more details about how the
scandal was exposed.
While I am a native English speaker, my knowledge of French is better
than my knowledge of German. My home in
Lausanne was much closer to the headquarters of
Freexian SARL in Saint-Étienne,
France.
Looking at the map, we can see how the business centres of
Lyon and Geneva are very close to each other. Consultants and
small businesses from cities throughout the region are competing
against each other for the same pool of clients.
Freexian SARL has a lot of French clients in the region and they
also recruited some clients in
Switzerland, including CERN, Wingo, Roche and Adfinis AG, a notable
Red Hat partner.
Various leaks show that
Chris Lamb, an employee or subcontractor of
Freexian SARL began engaging in antitrust or unfair trade practices.
The relationship between joint authors of
Debian is a peer to peer relationship. The Debian Project Leader,
who was
Chris Lamb, does not pay us and therefore does not have any right
to give orders to the rest of us. Nonetheless,
Chris Lamb would regularly impose upon people and he would spread
defamatory rumours about people who refused to obey him.
In January 2018, I went to St-Cergue for the event
Rencontres Hivernales du Libre. This annual event is in a mountain
village just north of Geneva. I think this is when people first realized
I had relocated to the French-speaking region and started plotting
against my family and I. It is an event where people do technical
work and ski:
In March 2018,
Anisa Kuci, one of the
Albanian female whistleblowers sends an email thanking me for my
support and confirming that
Elio Qoshi, with the underage girlfriend, is the root cause of all
harassment and abuse problems:
Subject: Re: "free travel"
Date: Mon, 5 Mar 2018 23:51:28 +0100
From: Anisa Kuci <anisakuci9@gmail.com>
To: larjona@debian.org
CC: Chris Lamb <lamby@debian.org>, Daniel Pocock <daniel@pocock.pro>,
leader@debian.org, antiharassment@debian.org
Hello Chris, Daniel, Laura,
Thank you very much for being so supportive.
I read the comments on the thread and to be honest I am really sad that
Elio [Qoshi] said that. It is not true at all.
They (Elio & Redon) pretend to support women but on the other hand their
behavior towards many of us shows the opposite.
Daniel I feel bad because you have encouraged and helped not only me,
but so many other people, no matter if they are Open Labs members or
not, and also all the attendees from Kosova to learn new things, to work
and improve their skills and knowledge. They are doubting your good
intentions just to remove the attention from the shady things that they
are doing.
The free travel comment is really offensive to me and i feel it should
be offensive to every woman who is part of the community.
I have been contributing and supporting Open Labs since its early days,
and I have put a lot of effort and time, I do this because I believe in
what it is meant to stand for and without waiting something in exchange,
but the situation lately has been not very positive. Daniel has been
present by chance in few cases where situations have been very hard to
go through.
I would definitely like to talk to any of you and tell you more about
everything that is happening here, its fine to me whether it is a video
call, call or just emails.
Please tell me what would be more convenient to you.
King greetings,
Anisa
On 5 March 2018 at 18:10, Laura Arjona Reina <larjona@debian.org> wrote:
Hello
El 5 de marzo de 2018 17:40:00 CET, Chris Lamb <lamby@debian.org> escribió:
>[Adding antiharrassment to CC]
>
>Daniel Pocock wrote:
>
>> If Elio or anybody else has made any other comments like this on the
>> private members channel or Telegram and you want to discuss them with
>me
>
>[..]
>
>Anisa, please feel to drop Daniel from any replies you wish to make, if
>you even wish to do so.
>
>(Daniel, thank you for your concern but we have got it from this point
>onwards. There will be no need for you to reply further on this
>thread.)
>
>
I may be silent but I'm listening (reading here and also subscribed
to the OpenLabs Forums, and up to date with the (public) English
posts so far).
Ping me if you need anything.
Best regards
--
Laura Arjona Reina
https://wiki.debian.org/LauraArjona
Sent with K-9 mail
Notice the condescending comments from the
Freexian SARL employee/contractor
Chris Lamb. He is trying to downplay the seriousness of the abuse and
deter
Anisa Kuci from making any written reply about it. She ignored
him and she wrote the inconvenient truth at that moment in the history
of the scandal.
The Debian
Code of Conduct gaslighting and the Swiss laws on
criminal speech are very similar: they protect dirty men like these
and they punish the people like me who took a risk to protect the
dignity of our female colleagues.
In July 2018, I went to Strasbourg, in
France for the
Rencontres Mondiales du Logiciel Libre
(
official 2018 RMLL web site). Other developers and Debian users
saw that I was making an effort to establish connections in the
French-speaking
Debian and open source communities. Some people seem to resent
the competition from an English-speaker (anglophone).
In reprisal and to frustrate my attempt to compete in the region,
Chris Lamb, who was still a subcontractor of
Freexian SARL, had formally recruited other joint authors, including
Enrico Zini, to engage in harassment and defamation.
From: Chris Lamb <lamby@debian.org>
To: da-manager@debian.org
Subject: Requesting the demotion of Daniel Pocock (pocock)
Date: Sun, 05 Aug 2018 14:26:23 +0100
Dear Debian Account Managers,
I am writing to you to request that Daniel Pocock (pocock) be demoted
from DD to DM.
[ ... snip defamation ... ]
Like each month, here comes a report about the work of paid contributors to Debian LTS.
Individual reports
...
Chris Lamb did 18 hours.
...
Chris Lamb,
Enrico Zini,
Raphael Hertzog and
Freexian have never paid anything to the rest of us. They simply have
no right to give us orders or change the priorities of our work or tell
us what we can and can't speak about.
The collusion between multiple people to exert control over somebody
without paying that person is a form of criminal exploitation.
The defamation tactics used to exert control over people also serve
to disrupt the competition between
Freexian SARL and the companies operated by
Chris Lamb's victims. Therefore, as well as being a case of defamation
and exploitation, it is an example of antitrust behaviour or unfair
competition.
DebConf18 was taking place at the same time
Chris Lamb was engaged in this conspiracy to start rumours about
my family and I.
Enrico Zini, a future employee of
Freexian, gave a talk on 3 August 2018 on the topic
"Multiple People". In the talk he hints at the gay power politics that
is wrecking
Debianism and humiliating healthy people. It looks like
Chris Lamb sought to exploit these tendencies to add weight to
his vendetta against my family and I.
At the end of November 2018, I attended the UN Forum on Business
and Human Rights in Geneva. This was at the very moment the defamation
gang was hard at work trying to create the impression that I am some
kind of neanderthal or cave man. A video of me asking a question at the
UN in Geneva sent the
rogue Debianists into a frenzy:
Earlier this year, the
Debianism elections had only one candidate.
She refuses to comment on being the wife of a male developer
while insisting that she wants to promote
diversity. This is disturbing because
Debianism has a history of using these women as stooges to spread
gossip about political rivals. As the women are wives and girlfriends,
they are considered to be disposable. Nobody cares if these women
burn their own reputations by spreading lies because they can
replace them with other women.
Some of the most dramatic evidence of this pattern is the email from
Paul Tagliamonte. At the time, he was employed in the White House under
the administration of
Barack Obama. Here is that email again:
Subject: Re: Expulsion of Jacob Appelbaum
Date: Tue, 21 Jun 2016 07:37:43 -0400
From: Paul R. Tagliamonte <paultag@gmail.com>
To: Russell Coker <russell@coker.com.au>
CC: Debian Private List <debian-private@lists.debian.org>
Seems like a thing the Press team could help coordinate.
On Jun 21, 2016 7:27 AM, "Russell Coker" <russell@coker.com.au> wrote:
http://www.itwire.com/business-it-news/open-source/73441-appelbaum-banned-from-debian-events-after-sexual-misconduct-charges.html
Well the DPL has decided to go public about it after all.
In future I think it would be better to have a plan for these
things. To have an argument on this list about whether information
should be released while the DPL is sending it all to a journalist
isn't the way things should go.
On 21 June 2016 8:57:56 PM AEST, Jonathan Dowland <jmtd@debian.org> wrote:
>Please do not CC me, I am subscribed to the list.
>
>On Tue, Jun 21, 2016 at 08:04:01PM +1000, Russell Coker wrote:
>> Mattia suggests that it's already public that he was expelled for
>anyone who
>> knows how to interpret it.
>>
>> Why can't we just state it outright instead of leaving clues in
>Wikipedia?
>> The social contract says that we won't hide problems. Why are we
>trying to
>> hide this problem?
>
>I don't think Debian has taken ownership of whatever is written on
>Wikipedia,
>but we certainly aren't hiding it: It's there on the NM page in plain
>sight.
>There's a big difference between hiding something and making an active
>effort
>to draw attention to it.
>
>The action that has been taken has been for the sake of the safety and
>well
>being of other Debian members, NOT as some kind of white-knight
>publicity
>stunt. There's no *need* to shout it from the rooftops as that does not
>further the former goal.
--
Sent from my Samsung Galaxy Note 3 with K-9 Mail.
Subject: Personal mail
Date: Sat, 29 Dec 2018 02:15:17 +0100
From: Laura Arjona Reina <larjona@debian.org>
To: Daniel Pocock <daniel@pocock.pro>
Daniel
This is a personal mail.
I'm very worried about you, your feelings and your health.
I would like to have a calm meeting with you (if you also want) and listen and maybe try to mediate but I really cannot these days; I'm in the middle of family visits. Also there is the language barrier (I'm much worse at listening/speaking that writing English). And my brain mostly in another place.
But I fear something bad (worse) happens to you or to any of the people involved in the situation.
She was right about that:
Adrian von Bidder-Senn had died on our wedding day, the
email was sent on the anniversary of
Ian Murdock's suicide and
Lucy Wayland died four weeks after Laura sent the email.
I probably should have seen this much earlier but I'm bad at that, and everything happens at a speed that is very hard for me to follow. But your last mail to -project triggered an alarm in me. I know that hackers' world is very fragile and even without knowing them I felt deeply sorry for A.Swartz and Ian Murdock, maybe because I've had that bad temptation of finishing with everything several times in my life, and also I've seen people in that situation too (but thanks God they/we managed to stay alive after all).
Please, stay alive and look for a safe place where brain can calm down and stop engaging in blue/black thoughts.
I am not a professional and can only talk about what worked in my case and the 2 other 'crisis' I've been witness.
I know it's difficult to escape from the pain, from how we see things, from those blue/black thoughts. OTOH try to calm down, switch context (e.g. stop any contact with the persons involved in the pain) and avoid the things that trigger bad thoughts some times help. Not stay alone much time, even not very close friends may be better than staying alone. Some times professional aid is needed (at least a general doctor and a psycholog were able to help me in recovering sleep, removing some black thoughts or fears, and become to a calmer situation in which my brain and soul work better).
I know our relationship is not of friendship in a traditional way (we don't know in person, never talked about private life for example) and I don't know even if you would trust in my words. I hope you receive them as I send: sincere, trying to empathise with whatever you are feeling, and just to tell that personal safety is first and please try to keep yourself alive and calm. No matter how difficult is the situation, with time and maybe third party help it can be improved.
I've offered my house several times as working space for a web team sprint that never happened. I've also invited some (very few) people to stay here if they want to visit me or Madrid. I extend this offer to you and your family. I can grant different topics of conversation and humble day to day life that may help to just take a break. My kid is 9 years old but good at chess (and soccer and videogames). I'm good at cooking and can share some recipes. Here's sunny, too, and that usually helps, too.
I don't know what else to say, except "take care", and "if I don't answer further mails it means probably that I'm far from phone and busy with family, but you are in my mind and I hope we can talk when I am back".
Kind regards
--
Laura Arjona Reinahttps://wiki.debian.org/LauraArjona
Sent with K-9 mail
At the time I received that email, I had already known for some weeks that
people had been encouraged to play psychological tricks on me. However,
I could also see the possibility that
Laura's email was completely innocent. I could also see a third
possibility, that is, the possibility that while
Laura may have thought she was sending the email in good faith, the
people who control her may have been using her to lure me into a
discussion that could be dangerous for one or both of us. While it
sounds paranoid, that is exactly how I've seen these groups using women
over and over again over the years.
When we talk about
diversity we need to look at the facts. We have seen numerous cases
over the years where women are used and exploited for the purposes of
spreading defamation and blackmailing male developers.
What about the companies that employ people to engage in
Debianism? If the controlling companies are not employing women in the
first place, why should women work on
Debianism for free?
The rate offered to paid contributors is the same for all (75 EUR/hour), it’s based on a correct rate for independent contractors in western Europe. If the rate is very high for your own country, then be happy to be able to invoice Freexian at this rate and use this opportunity to work less (for money) and contribute more to Debian on your (now copious) free time.
Raphael Hertzog is telling us that he pays people the same salary
whether they are located in
France or in
India. Does the same parity hold true for gender? Does every woman
receive the same payment as every man?
I took FSFE to court. This is my story ... A female colleague and me had dared to discuss wage transparency and gender pay gap in the office. ... our boss Matthias was beyond furious ... “there will be consequences” ... My aforementioned female colleague, who had also backed me up, was fired just a few days later. ... fired all the full-time women in the office in the two months leading up to the elections ... He even texted telling me I should answer my phone, for my own better. Even after my lawyer warned him to terminate all attempts to communicate with me and send someone else to pick up my work laptop, he came in person to my house, and was very irritated that I was not alone.
If
Sruthi Chandran insists on making
diversity the pillar of her election campaign then we need to make
a laser focus on the hiring decisions and the firing decisions of the
controlling corporations who suffocate
Debianism today. In most organisations, the work history of employees
and volunteers is private but in
Debianism it is all public. The
Debian Publicity Team gangsters have taken a key role in publicly
humiliating volunteers and our families so it becomes particularly relevant
to look at the credibility of the people in that gang.
Most of the time, when new developers get involved in
Debianism, their goal is to create a package or improve some of
the tools in Debian itself.
Anupa Ann Joseph appears to express no interest in such technical
work. We need to be suspicious when a woman wants to join Debian only
to join a team known for spreading gossip. Somebody may be controlling
the woman and planning to use her as a cyberweapon.
Later in 2021, we can see that
Donald Norwood was used to spread defamation and libel under
the name of the
Debian press team. He was subsequently removed from the team.
commit 285a2280c465fb58549d6155ac663c71e9b4afbd
Author: Anupa Ann Joseph
Date: Wed Aug 30 09:13:44 2023 +0200
Confirmed events for DebConf23 is on our website https://debconf23.debconf.org/talks/ #debian #debconf #freesoftware #dc23 #debconf23 #kochi #debconfkochi #debianindia
diff --git a/content/2023/08/1693386801.md b/content/2023/08/1693386801.md
new file mode 100644
index 00000000..10889b9d
--- /dev/null
+++ b/content/2023/08/1693386801.md
@@ -0,0 +1,7 @@
+Title: Confirmed events for DebConf23 is on our website https://debconf23.debconf.org/talks/ #debian #debconf #freesoftware #dc23 #debconf23 #kochi #debconfkochi #debianindia
+Slug: 1693386801
+Date: 2023-08-30 09:13
+Author: Anupa Ann Joseph
+Status: published
+
+Confirmed events for DebConf23 is on our website [https://debconf23.debconf.org/talks/](https://debconf23.debconf.org/talks/) #debian #debconf #freesoftware #dc23 #debconf23 #kochi #debconfkochi #debianindia
Many organisations are now giving payments to
influencers and similar
people who promote their products or services on
social control media.
There are two key differences between the work done by regular
influencers and the work being done by
Anupa Ann Joseph.
First of all, the regular
influencers normally create their own content spontaneously. Looking
at all the announcements
Anupa Ann Joseph has committed, she is making tweets about other
peoples' speeches and content.
Secondly, the regular
influencers normally have thousands of their own followers. They help
bring new followers to a brand.
Anupa Ann Joseph is sending tweets to people who already follow
Debian directly.
Therefore, on both counts,
Anupa Ann Joseph does not add any value. She is just doing a chore,
cutting and pasting announcements from other people into an existing
group of followers and she is receiving huge benefits in return for
work.
I would like to apply to change my status in Debian to Debian Developer, non-uploading.
I have been part of the Debian community for past 3 years. I have worked on organizing the local track in DebConf 20 and DebConf 21, volunteered in the debian-publicity team for MiniDebConf Gaming edition and MiniDebConf India. Organized MiniDebConf India 2021, and also organized Debian release parties locally for Buster release and online for Bullseye release. I would like to be able to contribute more towards publicizing Debian in India and strengthen the diversity in community, it will also help us organize DebConf India.
For real developers, male or female, the "non-uploading" status is a
sham that makes them feel inferior to the gangmasters. A lot of the women in the
Debianism cult have been "parked" with this
"non-uploading"/"non-developing Developer" status.
This dubious stratification of volunteers reduces the incentive for the
women to go further in the development of their technical skills. Many
of the people given this status simply languish at this level and
never progress to becoming a "real" Debian Developer, whatever that
means today. As a non-developing developer, they get a minor certificate
they can use for job hunting and they don't try to progress any further.
I have worked with Anupa Ann Joseph on DebConf and MiniDebConf publicity, and also other publicity tasks
since September 2020 and I consider them as having sufficient technical competence and sufficient social
knowledge of the project.
I have worked with Anupa Ann Joseph on organizing Debian Release parties,
online Debian conferences (minidebconfs and DebConfs), and preparing a
successful bid for DebConf in India, for many years and I consider her as
having sufficient technical competence.
She also coordinated printing and shipping T shirts for these events.
They all make reference to her "technical competence". When male
developers apply to have the same recognition, they are expected to publish
copies of their source code to prove their technical competence. In the
case of
Anupa Ann Joseph, we are being asked to trust that she has
technical competence without seeing the code anywhere.
If some random girl in
India uses her account on
social control media to spread rumours about
Dr Richard Stallman (RMS) or some other senior male developer,
nobody is going to pay any attention to that girl. If you give the
girl a big title like Debian Developer then it makes her
gossip look more important. But gossip is still gossip, no matter
how many titles the girls are giving themselves.
Look past the Debian Developer title and remember this is
just one of millions of twenty-something year-old girls in
India who have never published any source code.
When other volunteers ask for funding to travel to DebConf, the
bursaries team usually wants to look at
contributors.debian.org to find their history and decide if the
funding is justified. Yet this ghost has received numerous free trips
without her name being listed there at all.
If
Anupa Ann Joseph was listed on
contributors.debian.org we would be able to look there and see a direct
link to her Salsa profile, we could browse her history in Git and we
would see the cut-and-paste social media posts and nothing else.
It seems to be better for her if people can't find a direct link to that
activity in the usual place.
During the current
Debianism elections, people have been talking about how to get
women to do work for Debian. Many people only do work for Debian if
some company is paying them. The companies that pay those people haven't
chosen to employ many women. When they do employ women, those companies
have a tendency to hire their wives and girlfriends.
Voting for
Sruthi Chandran and her hot air about
diversity does not change the way all these other companies decide which
women, if any, to employ.
Once again, does
Raphael Hertzog really pay the Indian girl
Anupa Ann Joseph EUR 75 per hour as he promised when he
launched his business? Or is there some reason this girl is
different from the male Debian Developers?
Going back to the death of
Adrian von Bidder-Senn, we can see that his wife,
Diana von Bidder, also works in computing. Why didn't they
ever try to make her a Debian Developer like the other wives?
It looks like she has a PhD and therefore she is not in the same
category as the other women who are easier for the men to control.
On top of that, they prefer to recruit "female developers" while
they are still single, young and naive about their status in the
workforce. Married women and women with children are totally
invisible to these internship programs.
Was
Anupa Ann Joseph really a
Freexian employee all along and did she receive free trips paid for
out of Debian bank accounts prior to them adding her to their public
staff list?
Anupa attended Debian Publicity team meeting and is moderating and posting on Debian Administrators LinkedIn group.
It looks like this is the first time her name appears in a report.
Therefore, it looks like
Raphael Hertzog has bought an existing member of the
Debian Publicity Team.
Remember, in May 2011,
Raphael Hertzog bought the domain name debian-handbook.info
which includes the trademark debian. Why doesn't the
Debian Publicity Team publish a "Statement on Raphael Hertzog"
accusing him of stealing the trademark? It seems that if you pay
money to people in the
Debian Publicity Team you are less likely to become a victim of
defamation. This is how conspiracy and blackmail works.
In 2025, she has
a DebConf25 speaker profile. We can see her in the video for the
Publicity Team BoF.
Anupa Ann Joseph is there to operate the computer displaying slides
for the speaker. She is invited to speak once for about thirty seconds.
The room is almost totally empty. Nobody cares what these people have to
gossip about but we all have to pay for their flights and hotels anyway.
In France, young men and women compete to enter universities and find
public sector jobs using a process known as the Concours,
which is based on merit. We don't see that in Debian.
There are thousands of young women in France who would have been
willing to do exactly the same job at
DebConf25. Think about all
the carbon emissions wasted when somebody is brought from
India to do chores like cutting and pasting.
Looking at it from another angle, if the
Freexian insiders hadn't given this
Indian girl a "Debian Developer" certificate, it is very unlikely
that any other free software organisation would give her conference
tickets and speaking opportunities. The whole free software ecosystem
has descended into farce when it comes to the status of women.
At some point, either
Anupa Ann Joseph will be in a relationship with a male
developer or she will leave this nonsense because she will be
in a relationship with a man who doesn't have time for
Debianism. If she hangs around too long she will eventually
become the subject of inappropriate gossip like
Molly de Blanc.
Remember the story of the Boy who cried wolf? Each
woman can only be used to attack one or two men and then she loses
all her credibility. That is one reason we have this nonsense
about hiding the identity of women who are used to make silly
rumor-complaints about male colleauges. The fact is, after a
woman has been used to start some rumors, she is cast aside and
replaced with another woman.
After the men use a woman like this, in some cases, the woman
simply disappears and in other cases, the woman ends up having
a pregnancy with somebody and her reputation for gossip gets
a reset because she is a wife now.
Look at the case of
Anisa Kuci in the world of
GNOME Foundation. Why did they park this girl there with
an anonymous profile in the staff list? Everybody knows but
nobody is allowed to say it out loud.
This isn't meant to be a report about
Anupa Ann Joseph herself. This is a report about how cults like
Debianism use and abuse vulnerbale women. There is nothing new about
this.
Anupa Ann Joseph is one of those women who appears to become
quite tame, obedient, submissive and in awe of the Google employees
in Debian. That is just what cults
are looking for. Look at the example of the
DebConf6 fight, which revolves around the rumor that
Ted Walther's partner at the dinner was a prostitute. In
fact, she was the local dentist.
Look at it from another angle. If the
DebConf25 was your own small business, would you use your own
funds to pay for this girl to fly from
India and operate the slide projector for you? Or would you
just pay a local woman to do that task?
People are starting to raise serious ethical concerns that places in
Google Summer of Code have been used to mentor the
Freexian recruits. In the case of women, it is even more sinister
because the
Outreachy funding is coming out of Debian's own bank account.
Now have a fresh look at the way
Sruthi Chandran and
Molly de Blanc have been signing up various people to do various tasks.
Sruthi Chandran and
Molly de Blanc do not personally understand the tasks they are
assigning to people. This is a huge concern for the reliability and
security of the Debian software:
Subject: Re: Google Summer of Code 2018
Date: Sun, 21 Jan 2018 20:25:22 -0500
From: mollydb <deblanc@riseup.net>
To: debian-outreach@lists.debian.org
I mmissed this on the application before! We need 2-5 administrators for the application. Who else wants to be one?
Cheers,
Molly
Even the more talented interns could see that Debian had
descended into a farce:
Subject: Re: Upcoming Outreachy Round
Date: Sun, 5 Aug 2018 09:49:55 +0000
From: Kira Obrezkova <kiraobrezkova@gmail.com>
To: Molly de Blanc <deblanc@riseup.net>
CC: Debian Outreach Team <outreach@debian.org>
Dear Molly,
I thank you for your job during Outreachy round. At the same time I
have some feedback. The main thing I want to note is that you do your
work not so good. You haven't responded to me and also during the last
round of Outreachy you have provide almost to no response to
applicants of Outreachy FSF project. I asked some of them and they
said that you haven't helped them:
https://lists.fsf.org/archive/html/esd-translators/2018-03/index.html
. Though, you were a mentor of this project.
At the same time, I can't see where you were useful for me. You
haven't answered to me, you haven't answered to applicants. So, it
would be great if you can improve your usefulness.
At the same time, I want to thank my mentor Thorsten Alteholz. He did
a great job helping me and I was able to find a new job that fits my
interests well.
Sruthi Chandran and
Molly de Blanc do not understand the tasks they are signing people
up to do. How many people did they incorrectly sign up to tasks that
may have negative consequences for the rest of us?
SiteGround is a technology company founded in 2004 that helps individuals,
entrepreneurs, and businesses build and grow their online presence.
Our goal is to provide customers with the tools they need to create, manage, and grow their online businesses from a single platform.
Over the years, SiteGround has built a strong engineering culture focused on
performance, reliability, innovation, and customer experience.
This continuous focus on technology and quality has helped us support millions of websites and businesses worldwide.
Tell us more about SiteGround's product lines?
SiteGround has evolved far beyond traditional web hosting.
Our platform includes web hosting, managed WordPress hosting, cloud infrastructure,
Site Tools, website-building solutions, ecommerce platforms, professional email services,
email marketing products, and AI-powered solutions.
These products serve a wide range of users, from individual creators and small businesses to agencies and larger organizations.
From a technical perspective, our products are built using a diverse technology stack
that includes PHP, Python, Go, TypeScript, React, containerized environments, and cloud technologies.
How about SiteGround's relation to open source?
Open source has always been an important part of SiteGround's engineering culture.
Many of the technologies that power our products are built on open source foundations,
and we actively support the open source community through sponsorships, community involvement,
and contributions to projects that help power the modern web.
We strongly believe that open source software promotes innovation, transparency, collaboration
and accessibility. We are also strong supporters of the WordPress ecosystem.
One of the advantages of using open source software is the ability to collaborate directly with maintainers,
contribute improvements, and tailor solutions to real-world needs.
An improvement from SiteGround's Martin Bodurov has already been merged into Kiwi TCMS.
One of the biggest challenges is the breadth and diversity of our product portfolio.
Our teams test everything from hosting infrastructure and website management tools to
ecommerce solutions, email marketing platforms, and AI-powered products.
These products are built using different technologies and often rely on complex integrations
and shared infrastructure. A change in one area can potentially affect multiple systems,
which makes regression testing, coverage visibility, and traceability especially important.
Another challenge is balancing the speed of delivery with the level of quality and reliability
our customers expect. As our platform continues to grow, maintaining confidence in releases requires
a combination of structured testing practices, automation, and close collaboration between
engineering and QA teams.
How do teams at SiteGround approach testing?
Testing starts with understanding the requirements and acceptance criteria for a feature.
Before execution begins, we define a lightweight test plan that outlines the areas we want to validate,
potential risks and the overall testing scope.
During testing, we maintain testing notes and a testing journal where we document findings, observations
and scenarios that deserve additional investigation. Both the test plan and the testing journal go
through a review process, helping us share knowledge, improve coverage, and ensure consistency across the team.
The approach combines structured validation with exploratory testing techniques,
allowing us to adapt as we learn more about the feature.
Detailed test scenarios are created and maintained in Kiwi TCMS. These scenarios serve both as
validation artifacts and also as long-term documentation and a knowledge base for the organization.
As features mature, many of these scenarios become candidates for our Playwright automation.
Automation plays a central role in our testing strategy.
We maintain extensive automated coverage across different layers of the testing pyramid,
including unit, integration, API, end-to-end, visual, accessibility, and performance-related testing.
Automated regression suites are continuously executed through our CI/CD pipelines,
helping teams identify issues early and maintain confidence in every change.
The combination of structured manual testing, peer reviews, automation,
and continuous regression automation execution allows us to maintain quality while supporting a fast development cycle.
What other technologies does testing at SiteGround involve?
Testing at SiteGround relies on a diverse ecosystem of tools that support automation,
reporting, traceability, and continuous delivery. Our end-to-end automation is primarily
built with Playwright, while reporting and execution visibility are provided through
Allure Report and our CI/CD pipelines. We also maintain dedicated checks for visual regressions,
accessibility validation, and website quality metrics using tools such as Lighthouse.
To support fast feedback cycles, our automation infrastructure is optimized for containerized
execution on Google Cloud Platform, allowing us to execute large regression suites efficiently and at scale.
Where does Kiwi TCMS fit into SiteGround's overall testing infrastructure?
Kiwi TCMS serves as the central repository for testing knowledge and traceability within our organization.
While our automation execution, reporting, and CI/CD processes are handled by other tools,
Kiwi TCMS acts as the place where testing knowledge is documented, maintained, and shared across teams.
It stores our testing scenarios, helps us understand what has already been automated,
and provides visibility into overall test coverage. By maintaining a clear relationship between manual test design
and automated validation, it helps us ensure consistency as our products and automation suites evolve.
Kiwi TCMS integrates into our broader testing ecosystem and it is the source of truth for testing knowledge.
It provides traceability between requirements, documented scenarios, and automated validation,
while also serving as a valuable source of context for engineers and AI-assisted development workflows.
Why did you decide to use Kiwi TCMS?
Before selecting Kiwi TCMS, we evaluated both commercial and open source test management solutions.
While many products offered a wide range of features, we were looking for a solution that was flexible,
practical, and aligned with the way our teams work.
Kiwi TCMS stood out because it provides the core functionality we need without unnecessary complexity.
Rather than focusing on heavily marketed features that we would rarely use,
it offers a clean and efficient approach to test management, traceability, and knowledge sharing.
Its open source nature was another important factor. We value the flexibility that open source software provides,
whether it's adapting workflows, building integrations, or contributing improvements back to the project.
This gives us confidence that the tool can evolve together with our testing needs.
Tell us how you've integrated Kiwi TCMS with Playwright and Allure Report
One of our goals has been to maintain strong traceability between documented test scenarios and automated validation.
The scenarios stored in Kiwi TCMS often serve as the foundation for our Playwright automation.
To preserve this connection, we include Kiwi TCMS test case identifiers in the Allure Report metadata
of our automated tests. This provides a consistent mapping between documented scenarios and automated coverage,
making it easier to understand which tests validate which requirements and features.
Having this relationship available directly in the test metadata helps us identify coverage gaps,
keep documentation aligned with implementation, and navigate easily between automated tests and their corresponding
Kiwi TCMS scenarios.
Over time, we have also built custom integrations around this model.
One example is a custom Playwright reporter that can read Kiwi TCMS identifiers from the Allure Report metadata
and correlate automated execution results with the corresponding test cases in Kiwi TCMS.
The exact implementation continues to evolve, but the primary objective remains the same:
maintaining a clear connection between test design, automation, and execution results.
Tell us more about your internal "Kiwi TCMS API Client" script
The internal "Kiwi TCMS API Client" started as a simple wrapper around the well-structured Kiwi TCMS API.
Initially, it was created to support integrations between our automation ecosystem and Kiwi TCMS,
helping us automate parts of the test result synchronization process and reduce manual effort.
Over time, its role evolved beyond automation integrations.
Because Kiwi TCMS serves as our central repository of testing knowledge,
this API client became a convenient way to expose that information to other tools and workflows.
More recently, we have also used it as part of the tooling that supports AI-assisted development and testing.
By giving AI tools controlled access to test scenarios and testing knowledge stored in Kiwi TCMS,
we can provide better context when generating test cases, automation code, or other testing-related artifacts.
This internal client remains intentionally lightweight, focusing on simplifying access to Kiwi TCMS data
and enabling integrations without requiring every other tool to interact directly with the API.
Tell us how does using AI in testing fit in relation to Kiwi TCMS?
Yes, AI has become an important part of our daily engineering and testing workflows.
We use tools such as Codex, Claude Code, and Cline to assist with a variety of tasks,
including test case generation, Playwright automation development, code reviews, documentation,
technical research, and day-to-day development tasks. These tools help us accelerate routine work
and allow engineers and testers to focus more on quality strategy, risk analysis, and problem solving.
Kiwi TCMS plays an important role in this process because it serves as our central repository of testing knowledge.
The scenarios and documentation stored there provide valuable context for both engineers and AI-assisted workflows,
helping ensure that generated test assets remain aligned with documented requirements, expected behavior, and existing coverage.
For example, when working on automated tests, AI tools can help us generate or refine Playwright implementations
based on the scenarios stored in Kiwi TCMS. Having access to well-structured testing knowledge
significantly improves the quality and consistency of the generated output.
Rather than treating AI as a replacement for testing expertise, we see it as a force multiplier.
Combining AI-assisted development with well-maintained testing knowledge in Kiwi TCMS allows us to
work more efficiently while maintaining consistency and quality across both manual and automated testing efforts.
If you like what we're doing please help us grow and sustain development!
Due to the increase in AI-generated security vulnerability reports, it is time for some changes in how GNOME manages vulnerability reports.
These policy changes intentionally do not distinguish between reports that contain AI-generated content and those that do not. Following the same rules for all vulnerability reports is simpler than having two different ways of doing things. Reporters rarely disclose AI use, and it’s nice to not have to guess whether the issue report is AI-generated or not; it’s normally obvious, but not always. Also, vulnerability reports that are not discovered by AI are becoming increasingly rare. Non-AI reports are now moderately unusual, so it really doesn’t make sense to optimize for them.
Reduced Disclosure Deadline
Traditionally, I have applied a 90 day disclosure deadline to all security issues reported to GNOME Security. 90 days is an industry standard timeline, but it doesn’t work particularly well for GNOME. In practice, almost all GNOME maintainers handle vulnerability reports in one of two ways:
The project maintainer fixes the issue quickly, typically within 1-3 weeks after it is reported.
The project maintainer does not fix the issue at all. The issue report eventually reaches the 90-day disclosure deadline, at which point I unset confidentiality.
The 90-day deadline is intended to allow project contributors time to fix the issue before it becomes public, but in practice, maintainers do not actually make use of most of this time. I disclose the issue report and request a CVE when it is fixed or when the disclosure deadline is reached, whichever comes first. Once a CVE is assigned, contributors who are not regular project maintainers will sometimes attempt to fix it. Accordingly, keeping the issue reports confidential for 90 days only introduces a delay that is not useful.
Some other projects, notably the Linux kernel, have implemented an immediate full disclosure policy for issue reports that seem to be AI-generated, on the basis that a vulnerability that can be discovered by AI is presumably already known to attackers. But this policy seems pretty extreme, and is certainly unkind to maintainers who might feel pressured to urgently fix the issue. Immediate disclosure would not work well for GNOME.
Instead, I will switch to a 30 day disclosure deadline for issues reported on August 1, 2026 or later. This seems like a good compromise. The shorter deadline would probably work better for GNOME even if not for the increase in AI-generated issue reports.
Procedure for Projects that Prohibit AI-Generated Content
If a project prohibits issue reports that contain AI-generated content, I will no longer forward security issues reported to GNOME Security to the project’s issue tracker, since the overwhelming majority of vulnerability reports contain AI-generated content and would violate the project’s policy. Instead, I will immediately close the issue report in the GNOME Security issue tracker, then ping the project maintainers to let them know about the existence of the report. If you prefer to receive vulnerability reports in your project’s issue tracker, then please change your project’s AI policy to make an exception for vulnerability reports.
Unfortunately, GNOME maintainers don’t have access to confidential issues in this issue tracker, and GitLab does not allow CCing individual developers on confidential issue reports. I had been planning to adopt immediate disclosure for these issues only, but perhaps we should instead expand the permissions to allow all GNOME developers to see the issue tracker. Opinions welcome.
Moving On
I have been managing GNOME security issue tracking since November 2020. (Thank you to Red Hat for supporting this work.) Security tracking is largely a secretarial duty: I keep track of issues when they are reported and when they are closed, disclose them when the deadline is reached, and request CVEs when appropriate. It is not a huge amount of work, but I am getting tired of it, so it’s time for a change. I will discontinue tracking newly-reported security issues on November 1, 2026. During November, I will focus only on tracking issues reported prior to November 1. By December 1, all disclosure deadlines for that set of issues will have been reached, and I will be done.
Currently nobody else is tracking GNOME security issues. If you are an experienced GNOME community member and you are interested in taking over this work, let me know and I will help you get started. (Security tracking is not a good task for newcomers.)
This may also be an opportunity to improve our tracking infrastructure. I use a wiki page, but this is fairly primitive and requires considerable manual upkeep. It’s easy to forget to update the page when an issue report is closed, for example. Ideally, we would replace the wiki with a proper web app that dynamically updates based on the actual state of the issue.
I use Home Assistant, and for text-to-speech (TTS) I’ve been running Piper through Wyoming Piper.
Piper is a fast, local neural TTS engine originally built for the Rhasspy project and now maintained by the Open Home Foundation. It’s designed to run entirely offline, even on modest hardware like a Raspberry Pi.
Wyoming is the open protocol Home Assistant uses to talk to voice components like TTS, speech-to-text, wake word, voice activity detection (VAD, which decides when someone has started or stopped speaking) over the network, so any service that speaks Wyoming can be plugged in as a satellite. Wyoming Piper just wraps Piper so it can be served this way.
Both work well and I have no complaints about reliability. My issue is quality: the German voices aren’t great. Piper depends on open datasets for training, and good open German speech data is scarce, so the German models lag behind the English ones. I also run TTS locally on my desktop for event reminders, so voice quality matters to me beyond just Home Assistant.
Looking for something better
I wanted better output quality, so I started looking at alternatives and found Crane, a Rust inference framework built on Candle.
An “inference model” is a trained neural network used to actually produce output like text, speech, an image, rather than to learn from data (that’s “training”). An “inference framework” is the software that loads such a model and runs it efficiently: managing GPU/CPU memory, batching requests, and exposing an API around it. Piper and Crane are both inference frameworks.
Crane already had Qwen3-TTS support, and its Serena voice’s German output sounded noticeably better. I also wanted to try Voxtral-4B-TTS-2603, Mistral’s open-weight TTS model, so I added support for it. Voxtral TTS produces expressive, natural-sounding speech across 9 languages including German, with low time-to-first-audio and streaming support. It is a good fit for a voice assistant that needs to start speaking quickly.
Adding Wyoming support
Once Voxtral was working in Crane, I built crane-wyoming, a standalone Wyoming protocol server, so Home Assistant could use these models as its TTS service. To make that possible, I added the Tts trait and the surrounding TTS abstractions to Crane, since there was no stable interface for driving a TTS model on its own, separate from Crane’s full inference engine (tokenizer/LLM/VLM machinery). Those abstractions have since been merged upstream: lucasjinreal/Crane#44.
crane-wyoming depends on Crane only for that Tts trait and the concrete model types it needs to construct, not Crane’s engine crate. So it carries its own small TTS-only model runtime (one dedicated worker thread per loaded model) and its own on-disk response cache.
The project grew into a small Cargo workspace. Besides the Wyoming server itself, it now has cw-say, a standalone CLI client for scripting.
It also has sd_crane_wyoming, an output module for speech-dispatcher. speech-dispatcher is the common Linux TTS abstraction layer that screen readers like Orca, and other accessibility tooling, talk to. It launches output modules as subprocesses and speaks to them over stdin/stdout, using its own line-oriented, SMTP-style protocol. sd_crane_wyoming translates that into Wyoming requests against a running crane-wyoming server. That way, the same server process and cache serving Home Assistant can also serve the desktop. After registering it in speechd.conf, spd-say -o crane "..." works. So does anything else built on speech-dispatcher, like Firefox’s “Read Aloud” or Orca itself. All of it gets the same voice quality as Home Assistant, without running a second TTS backend.
What’s next
Speech-to-text. I’ve added Qwen3-ASR support for utomatic speech recognition (ASR) into Crane. This needs to be wired in crane-wyoming next. Also Voxtral-Mini-4B-Realtime-2602 is interesting.
VAD. Crane already has a Silero VAD implementation. Adding it for STT is straight forward.
Wake word. Once VAD is in place, add Open Wake Word support or similar.
With all of that implemented, you’d have a complete self-hosted Wyoming voice stack with no cloud dependency.
Current limitations
The catch is that you need a GPU to run it well.
If you only need TTS for occasional things like reminders, short announcements, running it on CPU with caching is enough, since repeated phrases just get served from cache instead of resynthesized.
All of this is for advanced users and hackers right now. There’s no polished packaging yet. Systemd units exist for both system and user services, including socket activation, but you still have to build from source.
Another week, another saturday post recaping things. :)
RHEL10 migrations
Bunch more things reinstalled with RHEL10 this last week. Made
some good progress. We are soon going to be down to the 'tricky'
ones that will require an outage. So, there will likely be
an outage or two in upcoming weeks to knock those out before
Fedora 45 branching.
Fedora 45 Mass rebuild
The mass rebuild for f45 started this last week and seems to be moving
along fine. Of course s390x is the slowest arch, but thats not
unexpected.
I did manage to update all the builders and reboot into the latest
kernel before the mass rebuild started, along with updating to
koji 1.36.1. So far no builders have dropped off or failed that
I am aware of, which is nice.
DNS and geoip
This last week we noticed that out dns geoip setup wasn't updating
correctly and had some pretty old data in it. This may have been
causing some network blocks in some regions to go to proxies
that are... not in those regions. ;(
Thanks to work from Vit Smolík, it's now updating correctly.
So, for some fedoraproject.org services hopefully some folks will
see improved performance with web application access.
I also added memory to some proxies and removed some from the
EU zone that were not really in EU.
17 July 1882 is the birthday of my great grandfather, the ANZAC hero
Sgt Robert Pocock. He had already completed three years military
service before the war but when the empire called, he re-enlisted and
came to the UK in a boat.
Coincidentally, 17 July is also the date that nominations close
in the
Clacton-on-Sea by-election.
On 13 August, the residents of
Clacton-on-Sea are being asked to rise to the occasion and vote for
the candidate who will best represent the interests of the region,
the nation and the Commonwealth.
In other words, should people make a tactical vote for a man with
a bin on his head or a strategic vote for the great-grandson of an
ANZAC hero?
Launching the Pocock-on-Sea campaign
There were so many news reports after the previous member resigned
from his post. The date of his resignation, 7 July, coincides with
the anniversary of the London underground terrorist bombings from 2005.
One of his colleagues,
Ann Widdecombe, was very regrettably killed the following day.
Ever since then, news reports discussed the egos of novelty
candidates. If the Essex Lion was to appear on the ballot paper, I suspect
it would have an equally good chance of winning. I have yet to see a single
report about the real, day-to-day issues facing residents of the district.
In a parody of this situation, I have chosen the codename
Pocock-on-Sea for the operation to unleash an Australian
upon London's Establishment.
Nigel Farage and Boris Johnson both endorsed Australianism
The outgoing candidate and the former prime minister have both publicly
called for the replacement of UK policies with Australian policy.
Clacton residents now have the opportunity to take up the nearest pair
of scissors and cut out the middleman. Why pay for
Nigel Farage to copy Australian policies when you can simply have
a real Australian like me?
Many people in
Clacton-on-Sea are deeply concerned about the risks of
social control media for their children. If this online culture is not even
safe for the people who created it, how can it be safe for kids?
Mental health, in general, is a big issue for the region.
The Deep Blue State
There is a lot of news about the Deep State and the Establishment at the
moment. Are they real?
People are invited to read my blog post
Google, FSFE and Child Labor and the efforts made by IBM Red Hat to censor
the blog. A legal panel made a verdict of harassment, stating that my company,
my family and I are the victims. (evidence bundle)
Here are the comments I made at the United Nations Forum on Business and
Human Rights in 2018. These comments were made before the sale of Twitter to
Elon Musk.
Release Candidate versions are available in the testing repository for Fedora and Enterprise Linux (RHEL / CentOS / Alma / Rocky and other clones) to allow more people to test them. They are available as Software Collections, for parallel installation, the perfect solution for such tests, and as base packages.
RPMs of PHP version 8.5.9RC1 are available
as base packages in the remi-modular-test for Fedora 42-44 and Enterprise Linux≥ 8
as SCL in remi-test repository
RPMs of PHP version 8.4.24RC1 are available
as base packages in the remi-modular-test for Fedora 42-44 and Enterprise Linux≥ 8
as SCL in remi-test repository
ℹ️ The packages are available for x86_64 and aarch64.
ℹ️ PHP version 8.3 is now in security mode only, so no more RC will be released.
I’ve tried several gaming oriented Linux distributions and found them all to be… well, to be frank, bloated and unstable. I seek the most minimal experience, secure, up to date, with a comfy desktop environment for a small screen. Most distributions are full of preinstalled stuff that you will never use, some of which often runs in the background, quietly eating away at your battery life. I also really like the ability to “tab out” of my game and have discord, browser, or anything else immediately available, without unreliable 3rd party plugin managers for steam that are broken every single time you want to play a game (yes I’m looking at you deckyloader). Simply I want my handheld to be ready to game, no matter how often I use it. Daily, or once a year, without having to deal with mandatory updates breaking everything.
For the longest of time I used Fedora KDE that would boot straight into the gamescope steam. Relatively recently I acquired a new mini laptop where I went with Fedora minimal + Niri + Noctalia and I love this setup. This article describes my configuration that can be applied to any laptop, or a handheld gaming console, with small differences. And I’ll be writing it as I apply it to my GPD Win4. Please note that this configuration is not for a beginner windows-refugee. You’d be better off with KDE if you need a lot of floating windows on many displays, or if you’re a beginner.
Linux Distro
Why Fedora? Because it’s one of the most stable Linux distributions and at the same time one of the most up to date distros with the latest toys available. We will be using minimal install, without any desktop environment. (Note that it doesn’t even install wifi drivers so you will need ethernet, or a usb stick to install those.)
Grab yourself a Fedora Everything iso and apply it onto your USB stick/dongle/goober of choice with Fedora Media Writer(available in your linux software center, flathub, etc.)
The good old fedora installer will guide you through everything, just select Zero checkboxes on the software selection screen :) Minimal. For laptops or other portable devices, use LUKS encryption. Otherwise default btrfs partitions are fine.
The first steps post install
You’re gonna boot into basic tty command line interface. If you have ethernet available, install your wifi drivers.
If you don’t have ethernet, you’re gonna have to find the rpm and shuffle it over with a usb stick.
Now the next thing that I configure is mounting my NFS share which I can then use to easily transfer some files, such as my do-it-all script which enables all my repositories necessary, installs all rpms, enables flathub, installs my flatpak apps, and clones my chezmoi configuration files for everything. At this point my install is done. I’ll help you with yours though :)
A little disclaimer - I am using a handful of copr repositories which are “user generated” packages without the same security and quality assurance that the main Fedora repository enjoys. Kind of a “use at your own risk” situation - you should always verify the author and use their packages only if you trust them. I trust those that I use (some of which are my own).
Install Everything
rpmfusion
RPMFusion repositories are necessary for other packages as well, but also useful for multimedia and nvidia. Enable them either way.
Fedoratricks is a collection of scripts that we (the Fedora Discord) have put together to help new users with the basic Fedora setup and diagnostics - to help us in providing support to you.
Enable the copr repository:
sudo dnf copr enable rhea/fedoratricks
Install the package
sudo dnf install fedoratricks
Use it to enable rpmfusion:
fedoratricks rpmfusion install
You can now use it to install multimedia/codecs, or even nvidia drivers if that’s what you have. Simply run $ fedoratricks --help to figure it out :)
Manual steps - take it from the horses mouth directly:
Terra repository is our source of noctalia packages. Beware though, it is known to cause issues if you leave it enabled. We will restrict it only to install the relevant packages.
Here is a complete list of things that I personally install on my handheld (both laptop and gaming) - feel free to use it to experiment with chezmoi, wezterm, neovim and ble.sh - or delete the things you don’t want.
Wezterm solves a few issues for me which many of the GTK based terminals cause.
Enable its copr $ sudo dnf copr enable wezfurlong/wezterm-nightly - otherwise remove it from the below command.
Another one included below is neovim - it’s great, you’re welcome :D No seriously though, you can remove it as well as npm fd-find ripgrep if you do not want to use it. If you do, you will need to also sudo npm install -g tree-sitter-cli and if you want tips on nvim plugins, message me.
Some of these packages are simply dependencies for other things, or necessary applications to exist in a minimal non-DE setup, and it also includes niri which we’ve omitted so far.
Install your desired flatpaks. For gaming, since that’s our focus here, you’d want at least protonplus (and probably a web browser - the above didn’t include one)
flatpak install protonplus
XDG configuration
This is by far the most tricky thing to do on this kind of install. Necessary packages should already be there, now you need to configure them - Arch Wiki reference
(Create its directory and) add into the following file:
Assuming you did install it. I won’t go into the details of why I have these, alas this is my preferred configuration. (I don’t like menu autocomplete.)
You can simply create an alias for this in your .bashrc for that btrfs-assistant doesn’t work out of the box without a DE. This is my workaround. Execute it and use btrfs assistant to configure your snapshots for both root / and /home, after you reboot and have some GUI :)
alias btrfs-alias='xhost +; pkexec btrfs-assistant; xhost -'
Auto-start Niri
Add at the end of your .bashrc
if [ "$(tty)" = "/dev/tty1" ]; then
echo "niri?"
niri --session
fi
Configure Niri
A must-have is autostart for noctalia shell. Optionally configure some keybinds and what not.
mkdir -p ~/.config/niri
Grab (wget) the default config and shove it into ~/.config/niri/config.kdl
You should probably run niri validate after messing with your config file.
Some things you can add or configure, I’m sure you can figure it out what’s what based on this example:
Technically all of the above can be done without a single reboot in tty - but you are now ready to reboot into your niri session.
After you reboot into niri/noctalia, configure your noctalia using its gui settings and under the Colour Scheme “tab” create templates for qt, gtk, niri and kcolorscheme
You will need to add include "noctalia.kdl" into your niri config for these to be read.
Then run the qt6ct application and simply select the noctalia theme and confirm. You should be all set now. If for some reason dolphin doesn’t use your theme, you may have to set it there manually. Dolphin is a special snowflake like that.
steam
I personally execute the entire Steam in gamescope on my handheld with the deck bigpicture mode. You can auto-start it as well. (Just keep in mind that Steam will update itself in the background so it may take minutes for it to actually show up if it’s updating.) Add into your niri config:
Since we’re still waiting for valve to fix gamescope touch input and nointro, you will need to run Steam without gamescope if you need the touch input. And you can use a black frame webm to get rid of the intro video as well, grab it and shove it into: ~/.steam/root/config/uioverrides/movies/deck_startup.webm
I just wanted to create a playlist I can listen to that will most likely have a
solid selection of songs I'm going to hear August 13/14/15. I ended up with a
prob and stats model that has produced close to 90% accurate predictions (citation needed, see "Context" below).
Back in February this year (2026) my Uncle Tim
and I caught the first 2 nights of the opening run of
moe's "Born to Fly" tour, at Higher
Ground in Burlington, VT.
It was great. My only note: I wish we had realized sooner that it was 3 nights,
not 2.
Going into that (unforgettable) experience I felt pretty confident in my
knowledge of moe's catalog. They've been around for about 35 years and primarily
tour, so that's a lot of potential music to keep up with. I was mistaken. There
were a lot of songs they played those two frigid Burlington nights that I wasn't
familiar with. And that's cool, ya know? You're allowed to play music I'm not
familiar with.
At the time I didn't feel like I got everything I wanted out of it.
Because of the hard work of the tapers like Phil
Hernandez I've
been able to listen to those two nights on repeat (and the third night we
missed) and it's even better than what I remembered. Which was, to be completely
honest, already pretty fucking great.
OK let me stop burying the lede. This update is about the weeks of work I've put
into some data analytics/modeling/probability software. Basically "setlist.fm
but better". I'm going to do my first "follow the band" trip in August and I
wanted to be sure I was familiar with enough of their recent touring catalog to
not get taken by surprise again. It's just how I enjoy music. Don't knock it.
So I made a few things.
My objective was to create some kind of script that will generate a pool of a
few dozen songs that are most likely to play on the 3 night run I'm catching
them in August. All I wanted was to create an actual playlist (on the computer)
that I can listen to that probably covers most of what I might hear (songs I
wasn't prepared for).
It turns out once you build that foundational model and data pipeline you can
build a lot of other prob and stats stuff with very little extra effort. And
when you add an LLM assistant, you can accelerate that like whoa.
The Scorecard is a prettier and more useful
version of the first iterations of what the model was producing. This version
keeps history and shows you how the predictions change over time as more results
come in and the model is able to improve its predictions.
Here are examples of two nights side by side. You can already see that there's a
lot of data in this. It's a lot more interesting if you load the page and
explore it yourself.
The model ingests data from 3 sources.
Archive.org API
Setlist.fm API
Machine vision parsing Instagram posts
Data ingestion is weighted in that order. We prefer to build nightly data based
on taper uploads in the moe. collection on
archive.org. We supplement that with data from setlist.fm
when archive is lacking. And if all else fails we fall back to me personally
feeding a Python parser data from their Instagram
page.
Speaking of data. Did you notice that pretty little building icon? Let me show
you a closer screenshot:
As we parse new data we update the database. That means that when the
predictions are updated we're able to automatically update all churn tables to
give you links to recordings we found on archive.org when we were building the
data set. Obviously this information is not available for dates that haven't
passed yet :-)
The churn dashboard starts at the beginning of June because… I had to start
somewhere. And at the time I was having to manually do a lot of data wrangling.
Eventually my automation was upgraded enough to backport to 2020. It seemed like
a fine place to stop. There's no reason I couldn't extend it back further. I
just don't need data that old for what the model is doing.
Speaking of which, what exactly is the model doing?
The model runs hundreds of thousands of simulations, using time based weights,
decays, and other "dampening" methods to predict future set lists. It's just not
reasonable to predict a 10 track set any given night. They have scores of songs.
Literally hundreds if you're being particular.
That's where The Songbook comes in.
If you filter The Songbook to just 2026, and only
include Anchors, Core Rotation, Regular Rotation, and Deep Cuts you
still end up with 100 songs:
What we can do is use probability and statistics to create a model for how
they historically write up set lists, and refine our predictions based on that.
We can incorporate a lot of attribute into this, such as venue type, if we're in
"a run" (several nights back to back), and even things like estimated set list
length and historical data. I'm not including all of that in the model yet,
it's a work in progress. Some things like learning average song/jam length are
coming in the next iteration. That greatly influences what we could expect to
hear in a given night. You wouldn't put 4 songs that tend to descend into 30+
minute jams into the same prediction for a given night in a small coffee-house
venue. You can also start to look for common segues and absolute exclusions.
Here's the context. That "90%" accuracy quote was not a lie, but also not the
whole truth. You read this far, strap in for some basic math.
Earlier I also said that we don't make 10 song predictions when we're building
predictions. We'd never get anything close to useful from that. We actually
build 30 song pools when we make predictions. Each of those has a probability
normalized from 0->1. On average given a recall@30 we "predict" 8 songs. The
model reports its nightly "expected" rate at about 20-25%, and most of the time
it's spot on. That cherry-picked screenshot above says we called 86% of the
show. But we picked 30 songs. There were only 7 songs played that night
Side note, that little greek building icon with the 1 next to it means there is 1 recoding on archive.org for that show, you could listen to it right now
When the model says 25% expected, it means it thinks about a quarter of the
songs in the prediction list, the 30-song pool, will show up in the actual
playlist. A lot of the time that turns out to be right. 20-30% of 30 is in the
6-9 song range.
But we didn't produce an artifact that had the exact number of songs in the show
that night.
I said recall@30 earlier, that's a tuning parameter of the model. If we make
the number there larger and larger our "hit rate" will increase and approach 1.0
(100%). That's because we would be producing a 100-song prediction pool.
Do you recall what I said earlier about the song book? If you filter to just the
most likely stuff this year, there they have already played 100 songs. It's a
coincidence that recall@100 would get us close to "100%" correct. But it's a
handy way to explain what this model does and DOES NOT do.
The model DOES NOT produce 90% accurate 10-song set lists
The model DOES produce pools of 30 most likely songs, of which about 25% tend to
show up. And sometimes, of those 6-9 songs that might play (it's never the top
6-9% by probability), more than half are actually played.
Here's another short recap of the last week from me.
Another shortish week as I was off monday, but still of course
a lot going on.
Aarch64 hardware fun
So, we have a bvmhost-a64 that runs 10 buildvm-a64's for koji builders.
A while back it refused to boot with a memory error. Reseating memory
got it booting again, but this week when I tried to reinstall it it failed
to boot again. At first we thought it was a bad memory stick and pulled one
to get it booting again. This took it from 512G memory to 384 (The sticks
are in pairs so it was 2 down). Since the mass rebuild for fedora 45 is
next week, I didn't want to run with fewer builders, so I had a stick
moved from a staging host. Adding that and... it didn't boot again.
Turns out on further inspection that it was the memory slot itself on
the motherboard that is bad. No memory in that slot works. :(
The vendor wanted us to ship the machine back for testing/repair/replacement.
(Normally they would replace/repair on site, but this machine was past
the time when they do that).
So, on to plan "B". I repurposed a buildhw we had to be a new bvmhost-a64
and reinstalled all the builders are we are back up and running. Of course we
will be down one builder, but that is much better than being down N buildvm's.
RHEL10 reinstalls
Made some progress on rhel10 migrations, although less than I would like.
I reinstalled all our bvmhost-x86's that run builders (and all the buildvm-x86
vm's that are on them). I hope to get a number more next week. I'm hopefull
I can do them without causing any outages, will do my best.
AWS proxy instances ssh problems
Someone wanted to debug / look at something on one of our proxies and noted
that they couldn't ssh into it, even though they were in the right groups
and should have been able to. I could not either. Login via root was still
possible, but none of our users. Nothing seemed wrong with our config, ansible
hadn't changed any in that area in quite a while. ssh config looked normal,
nothing odd in /etc/ssh/sshd_config* files. It wasn't even getting past
ssh, just saying 'ssh keys failed'. It affected all the proxies we have
that were running in aws. I did see some odd selinux denials, but setenforce 0
did not get things working.
Finally, I happened to do a ps to make sure sshd was running in the right
selinux context and found it. It was the ec2-instance-connect package.
It installs a systemd drop in for ssh that adds it's own AuthorizedKeysCommand
option, which completely overrides ours thats in config. Additionally,
removing that package causes sshd to... be disabled and stopped.
Luckily I was logged in as root and able to start/reenable it.
I filed https://bugzilla.redhat.com/show_bug.cgi?id=2498870 on this.
Another Flock to Fedora conference has come and gone, and like last year, this one was held in Prague. Unlike last year, I was not in the middle of moving across the country (again) so I was able to attend, thanks to my employer.
As always, it was great to see so many familiar faces and meet new folks face-to-face. To those of you who weren’t able to make it, you were missed. And, as always, I spent a lot of time in the hallway track talking to people, getting a sense of what everyone was working on and interested in.
Day -1
The day before the conference started, there was a sponsorship dinner. Although Bex did all the work getting the paperwork to the correct people at Microsoft, he wasn’t able to make it to Prague in time for the dinner so I was sent. I arrived on Saturday morning after a quick connection through Dublin, which gave me plenty of time to get settled in and resist taking a nap. I spent the dinner chatting with Kevin Fenzi and Jef Spaleta, and while I can’t remember all the topics, curling was definitely mentioned.
Day 0
I volunteered to help at the check-in desk the morning of the first day, which I felt went very smoothly (the new label makers were a nice addition). It was nice to help out, but it was also a great way to match names I’ve seen on Matrix to faces as folks arrived. After my shift, I got sucked into the hallway track until lunch.
After lunch and a bit more hallway track, I went to the “PR-based Gating for Fedora: Can We Make It Work?” workshop from František Lachman. There was a lot of discussion in and around the Fedora contribution workflow which I have lots of thoughts about, but I felt there was a rather widespread desire to make things better (even if the exact way we do that isn’t clear). Lots of people who were not me brought up keeping the specfiles in one repository rather than forty thousand or however many git repositories we’re up to. In any case, I’d really like a nice pull request workflow for Fedora where I can’t mess up updates, and where we can all share the tooling we build around packaging.
I spent the rest of the day in the hallway track, doing some last minute preparations for my talk on signing, and preparing for the joint Microsoft talk with Reuben and Bex. I was happy to meet some of the Red Hat folks working on cryptography and signing, and I’m hopefully somewhere down the line we can all do approximately the same thing for signing content.
I got dinner with Bex and Reuben at some place that served North Carolina style BBQ, and it was pretty good (especially with kimchi on top!).
Day 1
This was the first day of recorded presentations. I went to the usual “State of Fedora” address, followed by the Fedora Council and FESCo panels. I thought it was interesting (but sadly unsurprising) to see downward trend of contributors, and I’d be interested to see a further breakdown of who’s leaving. I have plenty of not-backed-by-hard-data ideas about why this is happening, but I do hope it leads to a stronger focus on (and acceptance of) improving the contribution experience - the general feeling in the hallways, as I mentioned earlier, makes me somewhat optimistic.
After lunch and a bit of hallway track, I went to the “Secure by Design: Aligning Fedora with the EU Cyber Resilience Act (CRA)” workshop by Jaroslav Řezník and Roman Zhukov. A good portion of it was a run down of what the CRA entailed and how the roles it describes map into Fedora. After that I spent a bit of time preparing for my talk. My talk went well, I think, except the live demo didn’t entirely work (gpg2 + gpg-agent + gnupg-pkcs11-scd is very finicky and I forgot a setup step). With that stressful event out of the way, I was able to relax a bit at the dinner party, chat with numerous folks, and fill up on the “appetizers” they brought out in vast quantities. Big props to the event organizers, the weather was great and I really appreciated the open space and variety of food options.
Day 2
It was hard to believe it was already the final day of the conference, but I think at this point I was also feeling pretty worn out. I went to Justin’s “State of the Fedora Kernel” talk, and was glad to hear that the GitLab workflow I helped build before I left wasn’t absolutely terrible. I made some last minute edits to the slides for our “Two Years In: Accelerating Microsoft Contributions to Fedora” talk (where I was happy to have Bex and Reuben do most of the talking), then helped present that talk. Afterwards I went to the “What’s new in Fedora CoreOS” talk and managed to chat with Jean-Baptiste Trystram and Joel Capitao about signing and Konflux, which we’ll hopefully get sorted out in the next couple weeks (in the staging environment, anyway). Hopefully we’ll also be able to get Fedora CoreOS images into the Azure community gallery alongside the Cloud images.
The lightning talks were all enjoyable, and I’m really impressed some folks even managed to make up slides for theirs and nothing went terribly wrong (great work everyone). There was time for a bit more hallway track, and then I went to the Contributor Recognition Program, which concluded the presentations for Flock 2026. I spent the evening catching up with old and new friends, chatting about ideas on improving various bits of Fedora infrastructure, and how to make the contributor experience better. People were already leaving for DevConf (or home) at this point, so if I didn’t get a chance to say goodbye, I’m sorry and I hope we’ll see each other next year!
Day 3
It was an uneventful trip back home, thankfully.
I’m looking forward to put all the work I’ve done on improving Fedora’s signing infrastructure through its paces, to get support for PQC done, and I have a few ideas on what to work on next. Hopefully some of them work out and don’t lead to too many people screaming at me. Flock is a great event to get excited about the next year of work and to test the waters on wild ideas, so I’m really glad I was able to make it this year.
RPM Fusion shipped Kodi 22-beta1 for Fedora 44. This is the version
with PR adding HDR support under Wayland merged.
It's a bit weird seing a beta version submitted as an update to stable branch, but hey, it works.
Thanks Leigh!
It works OK under GNOME session with HDR enabled. The UI elements are bit oversaturated
when playing High Dynamic Content, but the videos themselves are pretty. At last. This is
the year of HDR on Linux Desktop.
If only GeForce Now would catch up with the times…
The second quarter of 2026 is over, and so in this post we’d like to highlight the top Fedora Quality contributors who helped us maintain the quality bar for Fedora during this time period. Fedora wouldn’t be a high-quality distribution without its community. Every single person who helped us detect and resolve issues, or verify that things work as expected, deserves our gratitude, thank you!
If you haven’t participated yet in testing Fedora, perhaps you’d like to give it a try? We gladly welcome everyone. Please look at our Fedora Quality homepage.
Testing proposed updates
When software packages are updated in Fedora (bringing bug fixes and new features), they are not released to end users immediately. They first go to the updates-testing repository, where they undergo automated testing, and also await manual feedback from human testers. This feedback can be provided through Bodhi, either by using its web interface or CLI tools, see instructions. Alerting package maintainers by posting a negative feedback with a problem description can stop the update from reaching general audience and causing issues to all our users. Testing proposed updates is a simple, yet vital process for keeping Fedora releases of high quality during their whole lifecycle. It is used both for already stable and in-development Fedora releases.
…and also 358 other testers who commented on less than 10 updates each, but 750 comments combined!
1 If a person provides multiple comments to a single update, it is considered as a single comment. Karma value is not taken into account.
Test days participation
Test Days are events which are partly focused on testing Changes planned for an upcoming Fedora release, but they also regularly test important areas of the Fedora distribution, like upgrades, internationalization, graphical drivers, desktop environments, kernel updates, and others. The upcoming and past events can be seen in our Testdays app.
Test period: Q2 2026 (2026-04-01 – 2026-06-30) Contributors: 25 Test cases executed: 76
This post announces "simple gals", or simpleGals stylized. This is my first
new semi-serious open source project in a long time. Like
bitmath I had an itch I needed to
scratch. And that itch was the apparent lack of a modern day "simple HTML image
gallery generator".
Back in my day, on Mac the Photos application had this nifty feature. You
could export a gallery to a simple HTML bundle with forward/backward nagivation.
It looked a lot like they used the DocBook XSL stylesheets to generate a simple
thing that worked really well. Speaking of which, I still have an example of
thing I"m talking about.
The Grand Library is one of the first big planned on graph-paper projects I
made in Minecraft, back when I was in college and it was still IN ALPHAv1.1.2_01. According to the wiki, this screenshot must have been taken between
the ass-end of
September and October
30th, 2010.
You can see the rest of that gallery
here with (most of) the full
build from beginning to end.
I think I had a screen recording, or screenshots of, a giant library burning
(after we backed up the world files). It brought the server (probably my old
macbook) to its knees. It was rough, buddy. But that's what we did back then: we
mined, crafted, and burned shit down/blew shit up. Not much has changed really.
That Apple generated HTML image gallery is the inspiration for simpleGals. It's
so stupid simple. Rows. Columns. Pagination. Click to view full size. It's
perfection.
simpleGals is trying to be a very simple command-line driven static HTML image
gallery generating tool. simpleGals just like to have fun, it doesn't want you
getting bogged down with all the tedious overhead associated with fancy gals,
running software that has to get patched, or paying another subscription.
simpleGals ain't like that.
simpleGals isn't for album management. You feed simpleGals directories of images
and in return you get some simple HTML files with thumbnails.
simpleGals inserts some simple javascript for quality of life enhancements. But
exactly 0 user-facing functionality requires javascript. The functionality
degrades gracefully to a simple point and click adventure, just like the old
days.
simpleGals has a full TUI interface for configuring projects and modifying
properties. The sgui tui can rebuild the gallery, just as well as the batch
simpleGals command. There are progress bars, because I was feeling fancy.
Also, you get actual image-previews IN THE CONSOLE when you're setting
captions/alt text, or toggling individual "include in gallery" options.
For over a year now, Log Detective has provided an analysis of failed package
builds in Copr.
Relatively recently, we have also integrated our service with Packit.
Now, you can use our log summarization algorithm with your own
agent, using our new MCP server.
Rather than relying on a remote service, the logs are all processed locally.
Installation
Installing Log Detective MCP server is as simple as pip install logdetective-mcp.
In order for your agent to have access, you need to follow relevant guidelines.
For example, adding the server to Claude Code:
claude mcp add logdetective -- logdetective-mcp
For Pi, you would first need to install appropriate MCP extension,
such as pi-mcp-adapter.
Using Log Detective MCP
After installation, the tool is ready for use.
Testing with Claude Code, on a simple example of failed build from Copr,
has revealed savings in the token budget, compared to the naive approach
of the agent reading the log file directly.
Analysis without Log Detective MCP:
Analysis with Log Detective MCP:
The agent response is also substantially faster.
How does the extract_log_snippets tool work
The MCP server provides a single tool, extract_log_snippets, derived
from tools used by our production agent.
Unlike our service, only general heuristics are supported at this time.
That being said, for many use cases, they are sufficient.
Just like our agent, the tool uses an updated fork of Drain3,
which I have published on PyPI.org and maintain.
The extract_log_snippets tool exposes several parameters to your agent,
controlling granularity and removing irrelevant snippets.
Parameter
Type
Default
Description
max_clusters
int
8
Maximum number of snippets to extract.
max_snippet_len
int
2000
Maximum character length per snippet.
skip_patterns
dict[str, str]
null
Map of names to regex patterns. Matching chunks are excluded before clustering.
The default values were chosen to minimize impact of the tool on your token
budget, while still extracting enough useful data on the first attempt
in most tested scenarios.
When used in skills, it is recommended to highlight that the tool provides
more benefit when utilized with files with line count over 200.
It is also mostly useful for working with one message per line, although
the tool does have a simple heuristic for extracting multi-line messages.
The Fedora Package Review Process is clunky, archaic,
and not on par with what we expect when contributing to Open Source projects in
this century. We all know that, and we all want it to improve. That being said,
we need to realize what is currently our main bottleneck. Even though the
process is not friendly to new contributors, they are doing just fine - at all
times, we have hundreds of new packages in the queue. Our biggest
problem is our inability to effectively review them.
I don’t think we talk about this problem enough. That’s why it felt so
validating to hear Miro Hrončok voice my exact thoughts during the
Flock to Fedora 2026 keynote.
In this blog post, I am going to elaborate on the ideas that we (mostly Miro)
came up with, shooting shit in the hallway after the session.
Proposing new packages through PRs
This is an obvious one, we talked about the same idea with
Zbigniew Jędrzejewski-Szmek at Flock to Fedora 2025. It
is a necessary prerequisite for any potential improvements, which will allow us
to have a workflow that contributors are familiar with, inline code comments,
CI/CD, and other things that are not possible in Bugzilla.
Proposing new packages through PRs would be trivial to implement if we
had all Fedora packages in a monorepo. Which we don’t, and we probably
don’t want to have. And even if we wanted to have, it would require
massive changes throughout the ecosystem.
As a workaround, we discussed having an intermediate repository on the
forge.fedoraproject.org into which we would only propose new
packages. It would have Packit CI enabled, and therefore every
proposed package would automatically get a scratch build and a test suite run on
top of it. Currently
supported tests are rpmlint, rpminspect, and license-validate.
We know that adding new tests is easy, as I am currently working on support for
fedora-review.
Of course, a final approval from a fellow package maintainer would still be
needed. Once accepted, we would merge the PR and automatically create a new
DistGit repository and import the package. Then we would delete all data from
the intermediate repository to keep it clean.
Bulk review
The review queue is and always has been in hundreds. Many of the packages are
dependencies for something else, and people have no motivation to review them
separately. And even if they do, it’s not always easy to test them on their
own. This currently leads to accepting broken packages that nobody tested or
ignoring the tickets completely.
We discussed the possibility of proposing multiple packages within one PR. They
could be reviewed, tested, and accepted all at once.
For such PRs, we could automatically create a new project in Copr and build the
packages in the order they were committed. We could nicely use
Copr’s build batches feature here. If multiple packages were
added in one commit, they could be built in parallel. If the contributor decides
to force-push into their PR, we would wipe the Copr project and start over.
Auto import into DistGit
There is no reason why the contributor would have to manually run
We can request all the DistGit repositories and branches for them. And once they
are created, we can automatically import the packages. There are some open
questions though. How can the contributor signalize what branches they want?
Should we use the git history from the PR? Can we use Forgejo actions to trigger
the requests and imports?
Proposal vs prototype
I realize this is not a formal proposal but merely a blog post on my personal
website. That was an intentional decision. We’ve been discussing and
bike-shedding this topic for years, and yet we don’t have much to show for it. I
am writing this article mainly not to forget the ideas we’ve had, and even
though I am interested in your thoughts, this is not an RFC. I already started
implementing a prototype, and soon I’ll record a demo for you. Then, I’ll start
bothering you and asking for feedback.
In my opinion, it doesn’t have to be perfect. We just need to kick this off,
implement something small that works, and improve it as time goes.
Maybe I should also say that I am not aiming to replace the current
Package Review Process. My goal is to provide an
alternative version of this process and allow contributors to choose which one
they want to follow. Then, someday in the future, if the alternative turns out
to be popular, possibly deprecating the old review process.
At the beginning, I struggled to find reasonable use cases for this tool.
I maintain software, which involves a lot more communication and coordination than actually writing code.
When people ask me what I do, I often half-jokingly reply that I read and write a lot of emails.
How can AI boost my productivity when I spend 80% of my time essentially talking to people?
Where is the fun in replacing the remaining 20% of actually crafting code with more talking, this time to half-competent robots?
In time, I found ways to use AI that felt productive.
And, ever so hypocritically, not only at work.
But at what cost?
I am supporting an industry that regularly harms open source projects such as Fedora, helps destroy the planet and uses stolen data.
Moreover, I’ve become reliant on a proprietary tool.
Is my AI-boosted contribution to Fedora worth it?
Despite my moral dilemma, I still love my job.
I am a long-standing, well-known Fedora contributor, working for the most part on whatever I feel is needed, earning a competitive salary.
In theory, I could go look for another job where I would not be motivated to do this, but I wouldn’t be able to keep doing the thing I love.
I try to make the best out of this situation and, despite my initial distaste, use the tool to improve the project.
So at the end of the day, I close my eyes and think of Fedora1.
However, the implications of embracing AI are not just impacting me.
The nature of my work means I’ve made hundreds (thousands?) of small open source contributions here and there.
And sometimes, when I use AI to deliver those, it kinda feels like bringing a chunk of meat to a vegan BBQ.
The people on the receiving end of my contribution for the most part don’t care about my job sustainability or IBM shareholders, nor should they.
Once, I used AI to contribute to a Fedora packaging project.
It was reluctantly reviewed by another long-standing, well-known Fedora contributor (who happens not to be employed by Red Hat and is not well-compensated for this work).
When talking to them, I realized that they were uncomfortable reviewing such a change.
I made them uncomfortable by choosing to use AI for this.
My employer made me uncomfortable; I passed it on to a volunteer.
What an outstanding open source citizen.
I appreciate the irony of this; and yet, I am no slop generator.
I understand what I submit and I put my name on it.
I disclose the usage for transparency, because it matters.
I don’t just drop a vibecoded patch on an open source project.
If you have an AI policy, I read it and respect it.
So it pains me deeply when our carefully considered contributions are outright branded AI slop or LLM hallucinations, and the person bringing them is evaluated solely on the basis of the tool they used.
Especially when such judgment is made by people I respect2.
No, I am not a tool. Please don’t treat me as such.
PS On a lighter note, here are some examples of AI usage that somehow eliminate this problem for me:
Debugging a problem — in our team we’ve been very successful in showing a failing test to Claude and telling it that it started failing with Python 3.15. While burning thousands of tokens and wasting gallons of drinking water it can usually successfully determine what caused the failure. We were able to determine this ourselves in the past, but this actually boosted our productivity. We can then report the problem to upstream after we have verified the find.
My own/my team’s semi-internal tooling —3 the omnipresent bunch of random scripts I am not proud of and which would desperately need a rewrite but ain’t nobody got time for that. Just slop ‘em. If it works, it works. If it doesn’t, roll-back. Nobody needs to see this code anyway. Excellent for AI — I am still kinda killing the planet but at least I don’t shove it in your face.
Reporting to management — somehow people keep asking me what I did. There’s no dilemma in providing AI-generated reports to the same people who asked me to use AI. If nothing else, it demonstrates how progressive I am with it.
Reviews for me — when I don’t use AI to generate code, but rather ask it to review my design and implementation, nobody is forced to deal with my AI usage. For example, I wrote this blogpost myself, then asked AI for feedback on typos, grammar, tone, voice, argument, rhetoric, structure, flow…4
And perhaps even more so, my mortgage and the food on my table. ↩
And precisely because of that I choose to not link those cases here. This is not about naming and shaming. ↩
This m-dash was copy-pasted from websearch results by a human. ↩
If nothing else, at least the model pretends it appreciates my sarcasm. ↩
A reminder that I am no longer here, but am instead here. The new RSS feed is here. If you're still reading this for some reason other than being on Dreamwidth, please update your feed.
"Today's the fourth of july.
Another june has gone by."
(appologies to Amiee Mann).
Here's another short recap of the last week from me.
I was off Thursday, and Friday was a holiday, and I'm off next monday too,
so this was a short week.
aarch64 builders and vmhosts reinstalls
I spend a bit of time reinstalling our aarch64 bvmhosts. These are the
machines that run all our buildvm-a64 instances. You would think this would
be a trivial task, but of course not.
When we got these machines as part of the datacenter move last year,
we couldn't get them to pxe boot from their 25G network. So, we ended
up patching some 1G connections to them to provision them. Then those
links were removed. So, I needed to fix the issue this time.
Turned out it was just some settings in the network card eeprom/settings.
To adjust it, I had to build a kernel module, load that, then poke
at the settings with a tool. Quite a pain, but luckily only a one
time thing.
After that, they pxe booted fine and were easy to reprovison with rhel10.
Except, then I hit the next thing: One of them had a bad memory stick
in it. We had hit this before, but after reseating all the memory
it came up ok, but that memory just decided to croak this time.
So, dc operations folks did a bunch of testing and isolated the bad
memory. Should be on the way in to get a replacement now. Until
the replacement arrives that machine is down memory, so a few buildvm's
on it are shutdown. Shouldn't matter too much.
Staging openshift workers network
Last week we moved a number of servers to balance power in racks.
That went fine, but networking folks noticed that 3 of the machines
were not properly using 802.3ad/lacp. That is, they were only connected
on one interface. These machines were our staging openshift workers.
It took me quite a lot of poking around to see what happened and how to fix it.
Openshift has a lot of ways it configures network and it was not clear at all to
me the flow. I did finally figure it out though: I had installed them with
net.ifnames=0 set. This meant they had eth2 and eth3 interfaces that were the
active ones. After the install, they booted without that and so the interface
names changed. The new ones didn't have any config, so it just picked the
first one and ran it's ovh setup on. So, I had to go in and setup NetworkManager
to know about the new interface names so it would bond them, then the openshift
setup script would just take that bond device and setup on it.
This is a report created by CLE Team, which is a team containing community members working in various Fedora groups for example Infrastructure, Release Engineering, Quality etc. This team is also moving forward some initiatives inside Fedora project.
Week: 29 June – 3 July 2026
Fedora Infrastructure
This team is taking care of day to day business regarding Fedora Infrastructure. It’s responsible for services running in Fedora infrastructure. Ticket tracker
This team is taking care of day to day business regarding CentOS Infrastructure and CentOS Stream Infrastructure. It’s responsible for services running in CentOS Infrastructure and CentOS Stream. CentOS ticket tracker CentOS Stream ticket tracker
This team is taking care of day to day business regarding Fedora releases. It’s responsible for releases, retirement process of packages and package builds. Ticket tracker
Continued investigation into building a Fedora container base image using Konflux.
Continued work on remaining Pagure -> Forgejo migrations.
F45 release cycle is set to begin soon, starting with Mass Rebuild in the middle of July. This will require some prep work next week.
RISC-V
This is the summary of the work done regarding the RISC-V architecture in Fedora.
Discussion with Scaleway (a cloud vendor in France) for potential Fedora Koji builders in their Paris datacenter. To be coordinated via RISE.
Hardware
Coordinated shipping another “K3” hardware to DavidA (one of the Fedora RISC-V maintainers). This will be used as another RISC-V Koji builder. Sponsored by CLE.
Milk-V Titan hardware is now available to buy. A handful of machines are being shipped to a couple of RISC-V engineers, including for Fedora use.
Community work
Fedora Omni kernels for Muse Pi Pro hardware discussion with Trevor from Baylibre and Jason (Fedora RISC-V kernel)
Marcin (hrw) Juszkiewicz continues to chip away at the Fedora RISC-V tracker
Fedora Omni kernel work continues, with support for Muse Pi Pro, K3, and more: Jason Montleon and Jennifer Berringer
Other:
Discussions on ‘fedora-devel’: “How can we improve the Changes Process?” thread
A lot of internal discussion about 2FA for packagers
Several internal and upstreams meetings
AI
This is the summary of the work done regarding AI in Fedora.
Aurelien Bompard improved the This Week in Fedora script to include the CLE status reports
QE
This team is taking care of quality of Fedora. Maintaining CI, organizing test days and keeping an eye on overall quality of Fedora releases.
Lots of PTO: psklenar on extended PTO, kparal and adamwill each took some post-travel PTO days. Also post-event travel, bureaucracy (trip and expense reports) and decompression cut into work time for most
Compose-critical package script now in MVP state – see pull request, it works and does useful stuff but needs more refinement
Version 8.6.0alpha1 has been released. It's still in development and will soon enter the stabilization phase for the developers and the test phase for the users (see the schedule).
The RPMs of this upcoming new version of PHP 8.6, are available in remi repository for Fedora ≥ 43 and Enterprise Linux ≥ 8 (RHEL, CentOS, Alma, Rocky...) in a fresh new Software Collection (php86) allowing its installation beside the system version.
As I (still) strongly believe in SCL's potential to provide a simple way to allow installation of various versions simultaneously, and as I think it is useful to offer this feature to allow developers to test their applications, to allow sysadmin to prepare a migration or simply to use this version for some specific application, I decide to create this new SCL.
I also plan to propose this new version as a Fedora 46 change (as F45 should be released a few weeks before PHP 8.6.0).
Installation :
yum install php86
⚠️ To be noticed:
the SCL is independent from the system and doesn't alter it
this SCL is available in remi-safe repository (or remi for Fedora)
installation is under the /opt/remi/php86 tree, configuration under the /etc/opt/remi/php86 tree
the FPM service (php86-php-fpm) is available, listening on /var/opt/remi/php86/run/php-fpm/www.sock
the php86 command gives simple access to this new version, however, the module or scl command is still the recommended way.
for now, the collection provides 8.6.0-alpha1, and alpha/beta/RC versions will be released in the next weeks
some of the PECL extensions are already available, see the extensions status page
tracking issue#342 can be used to follow the work in progress on RPMS of PHP and extensions
the php86-syspaths package allows to use it as the system's default version
ℹ️ Also, read other entries about SCL especially the description of My PHP workstation.
GNOME developers have long used G_GNUC_CONST, which expands to __attribute__((const)), to annotate GObject _get_type() functions, despite knowing that it is incorrect to do so. const functions by definition have no side effects, but _get_type() functions actually have a side effect the first time the function is called: they initialize the type. Why apply an incorrect annotation to these functions? Because it makes the code faster.
Although this was long known to be incorrect, it worked fine in practice… until now. Regrettably, Sam James has discovered that GCC 16 may optimize away the type initialization, resulting in crashes. This is our fault for providing the compiler with wrong information about our code, so it’s time to audit your use of const attributes to remove them from _get_type() functions. Most GNOME programs use these attributes only for _get_type() functions, but if you use it in more places, then check to make sure those functions are actually const, as defined by the GCC documentation.
Sadly, there is no suitable replacement attribute for _get_type() functions. Two decades ago, Behdad requested a new idempotent attribute for expressing the desired semantics, but nobody has implemented it.
Time for a recap of the last week in my #fedora infra space
(here in longer form).
Overall this week was a bunch of catch up from being away at flock,
along with recovering from Jetlag.
RHEL10 migrations
I scheduled an outage on thursday for the last of the external
colo virthosts to get a RHEL10 reinstall. I had tried to do it
in the last outage, but I wasn't able to do it in the normal
way that preserved all the guests data. This time I managed
to get it done, but it took longer than I would have liked and
I hit a number of fun issues:
Using a remote console I can't hold down shift to get a grub
menu, so I had to hit escape at the right time.
rdp for Fedora is not great still. gnome-connections is still
the winner, but even it has quirks.
The /boot/efi partition still did not want to let me reformat
it as part of the install. I thought it was because there
was a spare in the raid1 for it, but I grew it to use that
spare and it still didn't help. So, I ended up just deleting
it and recreating it.
Finally all reinstalled, reprovisioned and all the guests
brought back up.
There's still a number of hosts to move, and we are down
to trickier ones, but we will look at another outage probibly
week after next to do more.
2fa in Fedora
Fesco decided to require all provenpackagers to enroll a 2fa
token. There's a lot of talk about expanding that to packagers,
or even everyone.
I thought I would share some info around this for interested
folks who may not have seen it in all the various threads
or tickets:
The Fedora account system frontend (noggin) supports enrolling
TOTP tokens. Once you enroll one you must use it to login to
the account system, to get a kerberos ticket, and to login
to any web application with your fedora account.
You can (and should!) enroll more than one token. As far as
I know, there's no limit to number you can add. You can also
disable or delete tokens, so long as you still have one
token enrolled. ie, you can never delete the last one.
Any enrolled, non deactivated token will work to authenticate
you. So, it's good to have at least one saved off to a safe
place in case you loose access to your primary token.
If you somehow loose access to all your tokens, the recovery
process is to mail admin@fedoraproject.org and you will be asked
to prove you are you. This may be a gpg signed email (with the
key associated with your account), using your ssh key associated
with the account, or other means.
So, please make sure you have a backup token to avoid that. :)
TOTP is pretty common and has been around a long time. Lots
of software is capable of storing your secret and displaying
tokens.
s390x builds backup and koji session bug
On friday morning our koji s390x builders were getting swamped.
It was the same story with a lot of large builds taking up builders
so other things couldn't get in. Or so I thought at first, but on
digging there was another problem. It was caused by a koji bug
(already fixed upstream, but not released yet). This caused
tasks to get freed and just sit there and not progress.
I have applied the patch and updated our hubs, and then went
through and reassigned all the tasks I could see that were still
having problems. Let me know if you see any I missed.
I also pulled in 2 more builders to the general build pool
One I had set to only do kernel builds a while back when we
were trying to get security updates out fast.
One was a compose host, but I think we can be fine with one
less compose host.
So adding those two helped process the backlog too.
I've also filed another request for more resources, but so far
it hasn't really gone anywhere. ;(
Some PTO next week
Next week, I am going to be taking thursday off and friday is a holiday
in the US, and I am taking off the following monday too.
Of course I'll still be around, but possibly doing other things. :)
How to migrate Fedora OS from one system to another, across network or (better) a Thunderbolt connection, without cloning the whole disk contents, saving SSD life.
Background
When migrating a Linux installation from a source drive to a target drive of a different size, traditional block-by-block tools like dd don’t allow to migrate to a smaller drive, and also waste SSD life by writing unused blocks. This guide describes a simple approach in which different disk sizes can be used, and the process is performed over network (or Thunderbolt), so that it’s not necessary to place two physical disks in the same device. The target system is exactly the same as the original, with all partition and filesystem UUIDs staying the same, which means no post-migration OS configuration is necessary. The time and SSD wear is minimized by pre-shrinking partitions and skipping unallocated disk space entirely. The default Fedora Workstation disk layout is supported, including LUKS encryption of the system partition.
This guide expects the following disk layout:
/dev/nvme0n1p1 — ESP (most often FAT)
/dev/nvme0n1p2 — Boot partition (most often Ext4)
/dev/nvme0n1p3 — System partition (most often Btrfs, optionally encrypted using LUKS)
For simplicity, this will not try to skip copying all unused blocks, just most of them. The first two partitions will be cloned in raw mode. The third partition will be shrunk first (thus saving lots of blocks from being copied, and also supporting smaller target drives), and then cloned in raw mode.
Migration steps
Shrink the system partition on the source system
On the source system:
Delete all necessary data, prune cash dirs, empty the trash, etc.
Make sure that filesystem consistency of your System partition is OK: sudo btrfs scrub start -B / This will take a couple of minutes, but you can check the progress with sudo btrfs scrub status /, if needed.
Boot to a Fedora Workstation Live system from a USB drive.
Run Gparted. In it, unlock the system partition, if encrypted. Then shrink the partition to a reasonable degree. For example, if the partition has 950 GB (from a 1 TB disk), but only 300 GB is used, you can shrink it e.g. to 350 GB. That still leaves enough free space for regular usage if you had to boot it for any reason, and saves 600 GB from being copied needlessly. This operation might take some time, because any data blocks currently present after the new size limit will need to be moved (therefore don’t be too aggressive in setting the limit). If your partition was encrypted, make sure that both the LUKS encryption container and the filesystem inside it correctly show up as resized.
Unmount all partitions (if any), lock the encrypted partition (if applicable) and close GParted.
Erase the target system
On the target system:
Ensure that this is really the system you want to completely erase (and replace with the OS from the source system).
Ensure that the disk size is sufficient to fit all partitions from the source system (now that the system partition on the source system has been shrunk).
Boot to a Fedora Workstation Live system from a USB drive.
Unmount all partitions (if any). You can use e.g. GNOME Disks for this.
Erase the whole drive (make sure you’re using the correct system): sudo wipefs -a /dev/nvme0n1
Discard all blocks on your drive (better for your SSD longevity): sudo blkdiscard -v /dev/nvme0n1
Configure power saving
On both systems:
Disable automatic suspend. You don’t want that to happen when doing disk operations. gsettings set org.gnome.settings-daemon.plugins.power sleep-inactive-ac-type nothing gsettings set org.gnome.settings-daemon.plugins.power sleep-inactive-battery-type nothing
Disable screen blanking and locking. You probably want to see the screen output at all times. gsettings set org.gnome.desktop.session idle-delay 0 As a bonus, you can also disable screen dimming: gsettings set org.gnome.settings-daemon.plugins.power idle-dim false
Establish an Ethernet/Thunderbolt network link
If you have both systems connected to the network and intend to use it, skip the Thunderbolt setup section below. Instead figure out the IP addresses of both systems (using the ip address command) and go to the Connectivity test.
Thunderbolt setup
If you want something faster than a regular network, have Thunderbolt ports on both systems and a fast USB-C cable ready, you can use that instead. With Thunderbolt, you can transfer data with 10 Gb/s speeds or more, compared to the common 1 Gb/s of a standard Ethernet.
On both laptops, after connecting them through a Thunderbolt cable, locate the network interface name: ip link Look for an interface like thunderbolt0.
Assign static IP addresses to them:
Source system: bash sudo ip addr add 192.168.5.1/24 dev <interface_name> sudo ip link set <interface_name> up
Target system: bash sudo ip addr add 192.168.5.2/24 dev <interface_name> sudo ip link set <interface_name> up
Connectivity test
Let’s use a well-named variable for both numbers, so that we don’t mix them up later. On both systems, run:
You should see both systems pinging the other system correctly.
If you want to test your connection speed (e.g. to pick the best USB cable), you can use iperf3.
On the target system, run the server: iperf3 --server
On the source system, run the client: iperf3 --client $TARGET_IP (you can add --format G to see the speed in bytes instead of bits)
Replicate partition table from source to target
This will make an exact copy of the source system partition table (including partition UUIDs) on the target system. Then we will adjust it to the different disk size.
On target system, start listening for data and save it:
nc -l -p 2000 > table.txt
On source system, dump the partition table and send it over:
Now that the target system is a complete clone of the source system, you can expand the System partition to fully utilize the remaining disk space. In Gparted, unlock the system partition, if encrypted. Then enlarge it according to your preferences. If your partition was encrypted, make sure that both the LUKS encryption container and the filesystem inside it correctly show up as resized.
Reboot the target system
You can now reboot the target system, and you should see your OS and data exactly the same as on your source system.
Check the filesystem consistency of your System partition:
sudo btrfs scrub start-B /
This will take a couple of minutes, but you can check the progress with sudo btrfs scrub status /, if needed.
If there are no problems detected, the process is now complete.
Boot problems
In case the target system has a very different hardware from the source system (e.g. a different brand of a graphics card), the drivers might be missing from the current kernel initramfs and you can see a black screen or kernel errors. In that case, reboot the system again, press F8 repeatedly during boot to get into the GRUB bootloader menu, and boot the rescue Fedora option, instead of the default one. That should hopefully boot using the generic image (which should contain all known drivers), and then you can regenerate all the kernel images using your current hardware setup with: sudo dracut --regenerate-all --force
Mentions of other cloning approaches
Using dd This works fine when cloning to a larger disk (if you fix the partition table afterwards), but unused blocks are written needlessly (wearing down SSD), and migrating to a smaller drive is problematic.
Native Btrfs streams using btrfs send and btrfs receive This perfectly copies only used blocks of the main partition, but it only submits a single subvolume. Which means there’s some extra work needed to make sure exactly the same subvolumes are replicated from source to target. Also, this works above the LUKS encryption, which also needs to be created manually on the target (while keeping IDs, etc).
Using Clonezilla Clonezilla will copy LUKS partitions as raw data, because it can’t see inside. Furthermore, it doesn’t support cloning to a smaller drive by default (there are some tweaks in Expert Mode, but you need to repair the GPT backup table manually anyway).
Sparse-copying the whole disk You can clone and restore the whole disk image with sparse blocks support, as seen in this guide. With a little tweaking, this approach could potentially be used even for my use case. One would need to make sure all the unallocated blocks are freed and really contain zeroes (which means running fstrim or blkdiscard and testing if it zeroed it out), and the partitions would need to be shrunk first in case of migrating to a smaller drive. The GPT backup header would need to be fixed (the same way as in the main article). So this might be a viable alternative approach for those who know how to tweak the steps.
Ampere Altra Q80-30 processor (80 cores at 3.0GHz)
RAM
128 GB (8x 16GBHMA82GR7CJR8N-XN)
GPU
AMD Radeon RX6700XT
NVME
Lexar LM9702TB ADATASX8200 Pro 1TB
Motherboard
ASRock Rack ALTRAD8UD-1L2T
PSU
MSIMPGA850G (850W)
Case
Endorfy 700 Air
USB3
no-name USB 3.2/10Gbps controller (PCIe x4)
To be fair, I should mention that this is a server motherboard, not a desktop
one, and Altra systems were never meant to be desktops (despite companies
selling them as such). Naturally, the list of tested/approved devices (Qualified
Vendor List (QVL for TLA fans)) is quite short and for Ampere Altra systems, it
does not contain AMD Radeon GPU cards. They can be made to work, but this often
requires additional effort.
The extra USB 3.2 controller allowed me to have more USB devices than the
motherboard alone supported, and gave me some 10Gbps ports for connecting
external NVMe drives.
The whole system was running just fine* under Fedora 42–44.
The first issue
Have you noticed the small “*” at the end of the previous paragraph? The system
I used was not quite Fedora — I had to use my own, self-built kernel.
You see, the PCI Express controller in the Ampere Altra has some issues. Let me
quote the description of
the Ampere Altra erratum 82288 patches:
Per Altra family erratum, PCIE_65 may cause invalid addresses to be generated
on PCIe mmio writes, impacting certain device types, notably AMD GPUs, and
thus the Altra family is not generally compatible with those device types.
And longer description from patch itself:
PCIe device drivers may map MMIO space as Normal, non-cacheable memory
attribute (e.g. Linux kernel drivers mapping MMIO using ioremap_wc). This may
be for the purpose of enabling write combining or unaligned accesses. This can
result in data corruption on the PCIe interface’s outbound MMIO writes due to
issues with the write-combining operation.
The workaround modifies software that maps PCIe MMIO space as Normal,
non-cacheable memory (e.g. ioremap_wc) to instead Device, non-gathering memory
(e.g. ioremap). And all memory operations on PCIe MMIO space must be strictly aligned.
So, to have a working Linux system, I had to rebuild the kernel on every package
update. Which usually meant “weekly”. Each Monday or Tuesday, I would update
the local copy of the Fedora kernel package repository and build it using my own
versioning scheme, like “7.0.2-200.fc44.pcie65.6”. The “pcie65” part reminded me
which patches I had applied, and the “6” was a counter for the patch rebases.
I cloned the repository from GitHub and then rebased patches, adapting them
whenever they needed work. The side effect was that I often used a newer kernel
than the official Fedora release — there is a “stabilisation” branch in the
Fedora kernel package repo where the soon-to-be-pushed version is present. So,
when Fedora had 6.19.y kernel, I had 7.0.z one.
So many cores, not enough speed
As I wrote in my previous post,
having eighty CPU cores does not mean that the system is a good, fast desktop machine.
AMDGPU started failing
As I mentioned above, to get my AMD Radeon RX6700XT running properly I had to
alter kernel with the out-of-tree patches. It worked, I could play some games,
watch videos with hardware-assisted video decode acceleration.
Until one day, around the Linux 7.0 release, when it started to fail. Running a
game ended with:
kernel: amdgpu 0000:03:00.0: Fence fallback timer expired on ring vcn_dec_0
kernel: amdgpu 0000:03:00.0: Fence fallback timer expired on ring vcn_dec_0
kernel: amdgpu 0000:03:00.0: Fence fallback timer expired on ring vcn_dec_0
Over and over again. Watching YouTube videos became impossible due to 720 out of
750 frames being dropped, etc.
Normally I would start to bisect the kernel to find out where the problem is.
But I was running a tainted kernel due to PCIE65 patches so who knew where the
problem actually was…
Let’s get Nvidia
I bought an Nvidia RTX 2060 graphics card and put it in place of the AMD Radeon.
It turned out that if I wanted to use it with the nouveau kernel driver I
still needed PCIE65 patches applied…
So I tried default Fedora kernel with Nvidia binary driver. And it worked fine.
Video decoding was accelerated, some games under Wine worked as well.
But then I started FreeCAD. And OrcaSlicer. And in both cases I got crash and exit…
It turned out that there was no org.freedesktop.Platform.GL.nvidia in Flatpak
repositories for AArch64. And I used both of those tools quite often.
Powering up the old x86-64…
At that point, I gave up. And booted my x86-64 system, which had been powered
off all that time. There were a lot of cables to move, some new ones to arrange,
and now I have both “wooster” (Ampere Altra) and “puchatek” (Ryzen 5 3600)
systems running under my desk.
Moving from 80 cores to 6 cores (12 threads) was a weird experience. A much
smaller number, yet things work fine. I can load all threads and the music still
plays. All games from my Steam library are playable. A working FreeCAD allows me
to finish designing cases for my home projects and I can 3D print prototypes
straight from OrcaSlicer.
The “wooster” system stays powered on, churning through RISC-V package builds.
It may be weak in single-thread, but it flies when it comes to multi-core load.
Conclusion
As for the Ampere Altra, I am not planning to repeat this experiment. Another
AArch64 desktop attempt would require a completely new hardware platform. And I
have no plans to spend over twenty thousand PLN to buy an Nvidia DGX Spark system.
This is a report created by CLE Team, which is a team containing community members working in various Fedora groups for example Infrastructure, Release Engineering, Quality etc. This team is also moving forward some initiatives inside Fedora project.
Week: 22 – 26 June 2026
Fedora Infrastructure
This team is taking care of day to day business regarding Fedora Infrastructure. It’s responsible for services running in Fedora infrastructure. Ticket tracker
This team is taking care of day to day business regarding CentOS Infrastructure and CentOS Stream Infrastructure. It’s responsible for services running in CentOS Infrastructure and CentOS Stream. CentOS ticket tracker CentOS Stream ticket tracker
Attended/interacted with some Folks contributors (remotely attending and chat in matrix rooms)
Created other tickets for .stg. stream el10 staging signing service (network ports for fedora-messaging stg rabbitmq hosts and two new service principals) – https://redhat.atlassian.net/browse/CS-3414
Prepared staging host for centos-stream signing (https://redhat.atlassian.net/browse/CS-3414) but now blocked with a fedora infra staging issue (rabbitmq) so tried the whole day to debug but need someone from Fedora infra side to solve the issue there
Riscv64 discussion about a new p550 hardware firmware upgrade
Tested and validated that RH network team opened needed port for rabbitmq.stg.fedoraproject.org
RISC-V
This is the summary of the work done regarding the RISC-V architecture in Fedora.
F45 rebuild started — we’re trying to first focus on language toolchains — need to refer to Miro’s lightning talk from Flock on how they handled Python bootstrap
F44 is being kept up2date (there were some big updates, including GCC)
Kevin did the legwork to move away from old dist-git on fedora.riscv.rocks to forge.fp.o
Another K3 builder in Fedora Koji — another member added. (This might sound like a tame update, but all compute power is helpful / important to keep the build momentum :-))
Discussed some details of riscv64 in primary Koji ticket
This team is taking care of quality of Fedora. Maintaining CI, organizing test days and keeping an eye on overall quality of Fedora releases.
Flock & DevConf attendance, lruzicka gave a talk on openQA test investigation, adamwill gave a lightning talk with cle about Fedora CI improvements, several of us were involved in a workshop/roundtable about the viability/desirability of moving all testing to dist-git PRs, lots of productive side conversations etc., many about AI – trip reports / blog posts coming
Lots of work to enable testing of the huge OpenSSL 4 update for Rawhide, investigate a significant bug that showed up, and eventually bypass the test failures for that bug as it’s proving not to be possible to resolve in a reasonable time (the update needed to get merged)
Continued tech debt / enhancement work on blockerbugs, testdays-web and issuebot
Forgejo
This team is working on introduction of https://forge.fedoraproject.org to Fedora and migration of repositories from pagure.io.
PTOs & travel
On Zabbix staging, now have a working template to monitor the Forgejo runnerhost VM and the runners themselves
Continued work on Private Issues (private comments refactoring)
UX
This team is working on improving User experience. Providing artwork, user experience, usability, and general design services to the Fedora project
Updates to translations of Fedora Documentation are again available. As announced on March 3rd, the unavailability of translation updates was due to the migration of the translation repositories and necessary tools from Pagure to the Fedora Forge. It took longer than expected but we are pleased to report this undertaking came finally to the end.
Long time no blog, once again - as always, I'm mostly posting on Mastodon now, so follow there if you're missing the Content. This is a bit big, though, so it goes here!
I was in Prague for Flock 2026 and Brno for Devconf.cz 2026 recently. Didn't have any issues with travel, fortunately. I was in Prague a day early for a Red Hat "face-to-face", which went fine. Had a fairly quiet/jetlagged dinner with Kevin and Tomas on the first night, and a nice dinner with Lenka Segura, Kashyap Chamarthy, Cristian Le, Frantisek Lehman and Laura Barcziova on the second night; most of them I was meeting for the first time in person, which is always good.
Flock 2026
Flock started for real the next day with workshops. I started with "The Future of Fedora Atomic", about how we can reach the goal of all the Atomic images being based on a shared bootc base container, then went to "Forging Fedora Project’s Future With Forgejo". In the afternoon I went to "PR-based Gating for Fedora: Can We Make It Work?", which was about the idea of moving all automated testing/gating to run against dist-git pull requests (instead of mainly against updates, as is the current case).
These were all more "talking shops" than "workshops", really, but they at least mostly produced interesting conversations and concrete ideas. In particular, Cristian and I were able to determine a set of priorities for Fedora CI from the "PR-based gating" session - there was a lot of discussion of potential issues and concerns further down the road of the potential migration, but in the end we found it pretty clear that we need to improve the reliability of the existing tests and pipelines, and the ease of interpreting the results, and that's uncontroversial work that can be done first.
There was also a request from Petr Khartskhaev that we make it possible to run the openQA tests on dist-git pull requests. This had already been requested by Mo Duffy before. I've resisted doing this for all dist-git pull requests because I know at least some would fail when changes to multiple packages need to be grouped and tested together. The update system allows us to group builds into updates, but dist-git does not yet have any way to mark multiple PRs as a logical group for testing/promotion. However, someone suggested a better idea: do it by request, not for all PRs automatically. So I decided to go ahead and do it. Instead of doing another workshop, I hid in the corner of the "Languages in Floss" session and bodged up a working prototype, which is deployed to staging openQA. If you comment /openqa test on a dist-git pull request, tests on it will run in staging openQA. I'm currently working on getting the results reliably reported back to the pull request.
We had a team dinner that evening, again good to meet people and a good mix of work and social chat. At some point in there I met our relatively new RH team member Jaroslav Groman in-person for the first time, which was great - he's been doing excellent work on digging out from under our tooling tech debt and it's great to have him aboard. Peter Sklenar unfortunately wasn't able to come this time.
Day two was session heavy. There was an opening keynote track with Jef's "State of Fedora" address, live Fedora Council and FESCo meetings (effectively), and a Hummingbird talk from Stef Walter. The State of Fedora produced a couple of very talked-about sets of statistics, one showing Fedora (and friends) usage climbing solidly and one showing Fedora contributor numbers declining worryingly. The usage numbers are great, of course - especially the rapidly-growing numbers for KDE and our awesome downstream distro friends (the uBlue-verse, Asahi, Bazzite et. al.) We definitely need to dig into the contributor numbers some more, see what's going on, and whether and what we need to do to reverse the trend. There are a lot of interesting questions about whether it's Red Hatters, community maintainers, or both who are declining, and how this relates to other trends like the CVE tsunami, mushrooming language ecosystems, the rise of Flatpaks / Snaps / AppImages and so on.
The Council and FESCo sessions touched on those topics and several others, though interestingly not the AI Desktop proposal which I was kinda expecting to hear a lot about. I kinda tuned out and hacked through some of them to be honest. Stef's talk gave a good clear overview of Hummingbird - I kinda knew it going in, but it's always good to have it summarized quickly and clearly (at least I thought so).
I unfortunately didn't know about the Lunch and Learns (somehow missed them on the schedule), or else I would have gone to some! But still had plenty of fun/productive lunches with various folks. In particular I met Vít Smolík (smoliicek) in person, which was great. He's been helping out with the data team and infrastructure team and is doing some great work.
In the afternoon I went to the "Packit and Fedora: The CI Story Continues" talk, which was a good summary of the work to rationalize the mess of different systems we have/had testing pull requests. It's much better than it was before. Then I went to "Fedora Server – What you can expect from the next two releases", which was great because it clearly explained the idea of the "Home Server" spinoff which I hadn't really been clear on. And of course it was good to see Peter Boy and Emmanuel Seyman again. Alexander Bokovoy also explained his latest authentication stuff, which as always I didn't entirely understand but filed under "sounds like Alexander has it under control"...
I skipped another session to do some hacking, then went to "The Engineer’s Guide to Design", which was really interesting - I like going to slightly off-the-beaten-path talks. I don't really work on front-end UX stuff a lot, but it was still interesting to hear a perspective from someone who's been both an engineer and a designer on the impedance mismatches that can happen and the basic concepts it's useful for both sides to know about the other. Then I saw "Artifact Signing in Fedora", where Jeremy Cline gave some background and an overview of his ongoing project to rewrite the Fedora signing server, which is badly-needed work that we're really grateful for.
In the evening we had the official party, at a really nice outdoor food court with a reserved space for the conference. The organizers brought over so many appetizers I barely needed to use the meal ticket, but managed to force down some tacos nevertheless. Had a great time chatting with various folks, then later headed to a Belgian beer bar for more drinks with Justin Forbes and several others.
On the final day I started with "The Packager's Guide to openQA Failures" of course - Lukas Ruzicka (my openQA henchman) did a great job covering openQA failure analysis from the perspective of a packager, and I contributed a few notes here and there. We had a good crowd who seemed really interested, which is always great news. After that I saw Kevin Fenzi's "scrapers gotta scrape scrape scrape" talk on all the fun we've been having with scraper networks flooding our infrastructure. I know some but not all of it beforehand, and of course Kevin explained it well and the audience was very engaged. Plus I got to tell my story about the time I thought a dastardly new scraper network had figured out how to evade Anubis, but it turned out that the call was coming from inside the house (i.e. I had done a slightly silly thing in openQA which made it effectively DoS Koji...)
After the coffee break I saw "Fedora Test Days - a11y", which was actually a somewhat wider talk from a couple of RH folks working on accessibility testing about their current testing and future plans. It was really interesting and it was good to be able to speak with them briefly about the possibilities of using openQA for this. I stayed in the same room for "Two Years In: Accelerating Microsoft Contribution to Fedora", where Jeremy Cline, Brian Exelbierd and Reuben Olinsky covered some Microsoft's (much-welcomed) contributions to Fedora and also some stuff about Azure Linux.
After that was "Upgrading Fedora Infrastructure from Nagios to Zabbix" - this migration has been ongoing for a while but was much-needed as a modernization and also a better architecture. I found it very useful as I do have plans to add more detailed monitoring of openQA and the talk was very helpful in letting me know how to get started with that.
After lunch came lightning talks. I had proposed one about "new stuff we did in Fedora CI lately", which kinda overlapped with some of the full-length talks in the end, but it still got a lot of votes, so Cristian Le and I went up second and did a rapid redux of the Packit consolidation work, improved result displays, optimizations and improvements to the generic tests, addition of rmdepcheck and so on. My voice was giving out by this point but we just about got through it. There were a lot of other great talks and everyone managed to come in under time, which was impressive.
After that was a Fedora Mindshare session which I half-followed and half-hacked/dozed through (was starting to get tired at this point!), then the "Fedora’s Contributor Recognition Program" session where much-deserved awards were handed out to Ankur Sinha, Fabio Valentini and Justin Forbes. I was on the voting panel for this so it was great to see the culmination, and the trophies contributed by the Nairobi GNU/Linux Users Group were awesome.
I almost forgot to mention the whole time I was struggling with some sort of wifi driver bug - it seems there was a troublesome AP or something at the hotel which caused my laptop to crash constantly. And the hotel wifi was terrible, and I only had 5GB of data on my phone, so I couldn't really rebase to F44 to avoid it. Lots of fun.
Devconf.cz 2026
That was the end of Flock; things wound down and I had dinner with...some people...somewhere (possibly the third Vietnamese of the trip? Things are getting fuzzy). The next day was a welcome quiet day traveling from Prague to Brno - train and bus, no problem except waiting for the bus was very hot. I stayed at the Hotel Vaka, which I've never been at before, but it's quite nice. The whole event was a bit weird because Moto GP (the motorbike equivalent of Formula 1) moved their Brno race to the same weekend Devconf.cz would usually be on, and took all the hotels, so devconf was hastily moved to Thursday/Friday. Hotels were still hard to get and I only just managed to grab this one at a decent rate. I was able to rebase my laptop finally and stop worrying about wifi crashes, and had a nice quiet pizza dinner at Doe Boy.
So a shortened devconf started with the opening session, then another Hummingbird keynote, this time with Valentin Rothberg as well as Stef Walter. It added a bit of detail compared to Stef's Flock talk, and there wasn't anything else on. Then I saw "OpenShift CI: What if we stopped retesting everything all the time?", which was an interesting talk about the tradeoffs involved in doing automatic retests of complex merge chains in a big project with lots of PRs trying to be merged all the time (OpenShift). It wasn't directly applicable to anything I do, exactly - Fedora updates don't quite map to PRs in a git repo - but in a more general sense it was useful in suggesting methods for thinking about this kind of complex tradeoff and how to measure the impact and efficiency of testing processes.
Next I saw David Duncan's "From Laptop Chaos to Fedora Cloud: Quadlets and Containers", mainly because it was David, but it turned out to be a good talk about a way to make a relatively complex multi-container side project buildable and deployable the same way on your laptop and in The Cloud, using systemd quadlets. I've dealt with this general area before a few times. David made a solid case that quadlets are a good approach, in his usual fun and personable style.
After that I saw Zbigniew's "New security features in systemd" - honestly I missed some of this one, but got the gist of why systemd is trying to modernize various mechanisms here. Then I went to "Beyond the Screen: A Deep Dive into Linux Accessibility for Developers" by Vojtech Polasek, which was one of my highlights of the week - a really good explanation of how computer interaction really works for blind people, and what properties applications should have (and avoid) to make them usable. This is obviously very important for testing purposes.
Next I went to "Stop Looking for the Perfect Prompt: The Design-First Workflow for Coding Agents" to get my corporate mandatory minimum AI Content(tm) - it was actually a pretty good and accessible talk on different approaches to LLM-based feature development. I still don't really use LLM code generation heavily (for a start, I spend so little time actually sitting at a blank screen typing significant amounts of code that it's not really worth worrying about), but it's good to keep up with the latest ideas about how to do this kinda thing. Continuing with the AI theme I took in Tomas Tomecek and Laura Barcziova's "How AI helped us ship updates in a Linux distro", which was a practical talk on the system behind Hummingbird and the actual approach it takes to (sort-of) AI-driven package builds.
After that I think I did some Fedora booth cover for a while. Lukas Ruzicka and Vojtech Trefny were holding the booth down for most of the weekend, but I stopped by and did an hour here and there to give them some relief. It's always a lot of fun chatting to people about Fedora, other distributions or stuff that has nothing to do with Linux at all. Lukas had set up a Framework laptop with a MIDI keyboard attached to show off that it's pretty practical to do audio creation on stock Fedora kernel and audio stack these days; Vojtech and I had absolutely no idea how to use it, so we had lots of fun trying to talk about it to people and eventually telling them to come back when Lukas would be there...
In the evening we had the conference party, which was at the same outdoor swimming pool it's been at for a couple of years(?) now. It's a great venue - you can grab some food and a drink and then relax in the shade under some trees. I wound up sitting round with a really interesting mixed group of folks (Red Hat and non-Red Hat, some big cheeses, some medium-size cheeses and some fresh faced...cheese curds? Help, my metaphor is falling apart) most of the night, it was a great evening. Walked back most of the way to the hotel with Stef Walter, setting the world to rights as is the tradition after a few drinks at conference parties...
On the second day I started with "From Podman to Production: Building Trusted Container Images with Konflux on OpenShift" by Vladimir Sokolenko, which was probably the most understandable and useful Konflux talk I've seen so far. I think I then maybe did a bit more booth cover(?) and some hacking, then wrapped up with a nice three-talk track in the same room - "Identical Testing Environments from Laptop to CI with tmt and Testing Farm" by Cristian and Petr Šplíchal (covering their work to make tests more reproducible from Testing Farm to your local system), "systemd-sysext in Production: What We Learned Extending /usr Without a Package Manager" by Brian Exelbierd and Daniel Zaťovič (a really good retrospective on their experience using sysext in the real world to ship additional / alternative software on Flatcar), and "Local package layering on bootc systems with DNF5" by Evan Goode (explaining his design and plans for providing a convenient, dnf-ish interface to "overlaying" packages on bootc-based installs). I met Evan in person for the first time at the party, and it was great talking to him. I hope his work on this goes well - we really need it for the glorious bootc-based future.
After that was the traditional wrap-up session with the trivia quiz (I won a t-shirt!) and then I said bye to a lot of people and melted off (it was extremely hot) to the train station...to find that my train to Vienna was delayed by nearly an hour. Ah, well. Eventually made it to my hotel in Vienna (which was incredibly nice, just wish I'd stayed there longer...) and had an excellent dinner at Iki (highly recommended if you're in the area). Got in a couple of swims in the nice-but-small hotel pool before and after sleeping, then had a Vienna take on avocado toast (interesting!) for breakfast and headed off to the airport, and that was another trip in the books.
My uncle Tim (We're both Tim Case, he's Tim H. Case, yes, THC, I'm just TC)
just mailed me two more issues of The Hemp Coalitions "Almost a Newsletter"
newsletter.
I just scanned issues 1 and 2 from volume 4, published in 1994, we now have 3 issues up the archive!
I love my uncle Tim a lot. I moved out of NY State with my family when I entered
my early teens. That meant that growing up he and I never got to have the kind
of relationship we wanted to have, that didn't happen in earnest until just
maybe 3 years ago.
Tim H Case is a character. And if you live in the Troy/Albany/Cohoes region of
NY you almost assuredly have encountered my Uncle
Tim.
He's also goes as "Fake Trey" or "Troy Pistachio". He's the Tall skinny ginger
with a pony tail that goes to his hips.
I'm going to show this page to him later and embarrass him �� 💚
Archiving this stuff from his past is one of my ways to connect with him and strengthen our Tim-ly bond 🤜� Tims Forever 🤛�
The full THC collection is available on the archive in this list I made. You can also just search for hemp-coalition-newsletter:
In the weeks leading up to that release (and since then) I have posted
a series of serieses of posts to Mastodon about key new features in
this release, under the
#systemd261
hash tag. In case you aren't using Mastodon, but would like to
read up, here's a list of all 27 posts:
I intend to do a similar series of serieses of posts for the next
systemd release (v262), hence if you haven't left tech Twitter for
Mastodon yet, now is the opportunity. My series for v262 will begin in
a few weeks most likely, under the
#systemd262
hash tag.
On Saturday May 30th 2026, the XV edition of P.I.W.O. Poznań Free Software Fest was held in Poznań, Poland.
P.I.W.O. is an event with a long history. Between 2004 and 2018, it was organized by various student associations at the PoznaÅ„ University of Technology. After 2018’s XIII edition (superstition much?), it entered a long hiatus that lasted until 2025, when members of the newly-formed Knyfyrtel PoznaÅ„ Hackerspace decided to bring it back. The reactivated event proved a huge success, with the XIV edition bringing in 197 attendees. As such, ambitions for 2026 were rather big.
Teamwork makes the dream work
This year’s edition was the result of combined efforts of three PoznaÅ„-based organizations:
Thanks to Webrains, for the first time in its history, the event was held at the Adam Mickiewicz University in PoznaÅ„ – namely, the uni’s Faculty of Mathematics and Computer Science. Also a historical first, the XV edition was granted honorary patronage by the President of PoznaÅ„ City Council.
Busy, busy day
The event spanned almost the entire Saturday, opening at 9:30 am and closing at 7:35 pm. The agenda was tightly packed, featuring 3 lecture tracks with 24 talks, 2 workshop tracks with 8 activities, plus a LAN Party track with 3 tournaments, as well as “free play” sessions. There was also a quiet corner where, thanks to the “Honte” group, attendees could relax and learn to play Go.
Throughout the years, one of P.I.W.O.’s hallmarks was the lunch break, with the attendees being provided free pizza. This year couldn’t be any different, with some 66m² (or 710 ft²) of pizza being delivered to the venue. (It disappeared frighteningly quick.)
The Fedora Community was represented by:
Zbyszek Jędrzejewski-Szmek, who gave 2 (!) talks
Mat Holmes, who volunteered as a photographer
Kacper Skrzyński, who handled various urgent tasks
Artur Frenszek-Iwicki, who gave a talk and held a workshop session
Historical moment: creation of SPOIWO
This year’s P.I.W.O. served as an opportunity to announce the creation of SPOIWO – Sojusz Przyjaciół Otwartego i Wolnego Oprogramowania (Alliance of Friends of Open and Free Software). This diverse coalition of social and technology organisations, cooperatives and open source businesses signed a declaration in support of digital sovereignty. The alliance was formed in response to the ever-deepening of public institutions’ dependency on closed platforms and their loss of control over data and communication.
The signing ceremony also featured speeches by representatives of SUSE Poland, Polish Linux User Group, “Internet. Czas działać!� Foundation and PLZ Spółdzielnia.
Exceeding all expectations
The event wrapped with 452 attendee badges issued, which exceeded even the most daring of expectations. (The number also explains why the pizza disappeared so quick.) The bar for the next edition is set high, but so are the organizing team’s ambitions.
If you’d like to learn more about the event, watch the live stream recordings, or subscribe to news so you won’t miss the announcement for the XVI edition – you can do all of that on the event’s website, at piwo.sh.
Dear testers, we're happy to announce Kiwi TCMS version 16.1!
IMPORTANT:
This is a minor version release which includes security related updates,
several improvements, database migrations, new API methods and updated translations.
I have a 5G modem in my Lenovo x13s and t14s. I run Fedora on them, currently Fedora 44. I have had issues in the past when traveling. The 5G modem works fine when in the US but will not connect to any provider when roaming. It has worked after some time in the UK and Australia, but not in other countries. I use Google Fi for phone service, which allows roaming with the same data allowance as in the US.
After some digging during my current trip, when it wasn’t connecting, I think I have found the issue and a workaround to ensure I can connect. Google Fi uses two networks, T-Mobile in the US and Three UK when roaming. It also uses T-Mobile for roaming. The issue seemed to be that the towers were rejecting the SIM card, attempting to register using T-Mobile’s default APN that seems to be baked into the SIM card as a default profile. The ModemManager logs looked like the following
ModemManager[1603]: <msg> [modem0] 3GPP packet service state changed (unknown -> detached)
ModemManager[1603]: <msg> [modem0] 3GPP packet service state changed (detached -> unknown)
ModemManager[1603]: <msg> [modem0] 3GPP packet service state changed (unknown -> detached)
ModemManager[1603]: <msg> [modem0] 3GPP packet service state changed (detached -> unknown)
The trick that got it to connect was to make a change to the default profile used to connect to the phone tower was to run an mmcli command: “mmcli -m 0 –3gpp-set-initial-eps-bearer-settings=”apn=h2g2,ip-type=ipv4v6” Which forces the sim to use the Google Fi APN and not the T-Mobile one
For quite some time I wanted to explore
openshell. I naively thought it would be
easy to set up on my workstation that has an AMD GPU, together with llama-cpp
as an inference server.
After following the tutorial, I was immediately stuck even with running the gateway in a podman container.
I guess I’m too old for this stuff because I let Claude Code to investigate the problems.
This is a report created by CLE Team, which is a team containing community members working in various Fedora groups for example Infrastructure, Release Engineering, Quality etc. This team is also moving forward some initiatives inside Fedora project.
Week: 15 – 19 June 2026 (Flock Week!!)
Fedora Infrastructure
This team is taking care of day to day business regarding Fedora Infrastructure. It’s responsible for services running in Fedora infrastructure. Ticket tracker
Helped getting elnbuildsync moved into stg openshift (ongoing).
A couple more services had OS upgrades to Fedora 44.
Some more MirrorManager problems.
Scrappers found another ostree repo. on kojipkgs, limited dir. indexing to stop the attack and then searched for any other ostree repos. Hopefully the last time it happens.
CentOS Infra including CentOS CI
This team is taking care of day to day business regarding CentOS Infrastructure and CentOS Stream Infrastructure. It’s responsible for services running in CentOS Infrastructure and CentOS Stream. CentOS ticket tracker CentOS Stream ticket tracker
Finish rebuilding/bumping some pkgs for ppc64le libvirt/qemu-kvm stack on el10 (https://gitlab.com/CentOS/infra/tracker/-/work_items/1928)
Start to work on debuginfo origin servers refresh with el10 instances (https://gitlab.com/CentOS/infra/tracker/-/work_items/1892)
This team is taking care of day to day business regarding Fedora releases. It’s responsible for releases, retirement process of packages and package builds. Ticket tracker
Migration at 95%, the only thing left to migrate is majorly fedora-scm-requests, bug fixes, and continued releng operations.
Migration: archive-repo-manager from pagure
Fix: Missing permissions on releng/fedora-comps
Bug Fix: Mass tagging script for verification bits
F45 Change Sets reviews are in check and f45-python side tags are merged.
Regular releng operations.
QE
This team is taking care of quality of Fedora. Maintaining CI, organizing test days and keeping an eye on overall quality of Fedora releases.
We’ve started searching for new owners/maintainers of Packager Dashboard and Oraculum
Forgejo
This team is working on introduction of https://forge.fedoraproject.org to Fedora and migration of repositories from pagure.io.
Completed the Flock To Fedora 2026 presentation on Fedora → Forgejo migration efforts #588, packaged and deployed Forgejo 15.0.3 #620.
successfully ran the PR fix doctor across all migrated repositories #601 to address merge issues with Pagure imports.
Enhanced runner infrastructure with the addition of a Testing Farm runner option #619 and docker-slim type runner #568, and improved security by deploying oauth-proxy to secure the Forge metrics endpoint #571.
EPEL
This team is working on keeping Epel running and helping package things.
Updated caddy in rawhide, resolving 14 CVEs
Ongoing onboarding of Pedro
UX
This team is working on improving User experience. Providing artwork, user experience, usability, and general design services to the Fedora project
With Flock this week, got to see how all the final designs turned out it in person!
Emma delivered her talk on ‘Why You Should Use Open Source Design in Open Source’ at Flock
Great progress on the F45 wallpaper in the final stretch!
If you have any questions or feedback, please respond to this report or contact us on #admin:fedoraproject.org channel on matrix.
درود بر دوستان، همراهان و اعضای خانواده بزرگ طرفداران فدورا امروز با افتخار پانزدهمین سالگرد تأسیس وبسایت FedoraFans می باشد. پانزده سال پیش، با هدف ترویج فرهنگ نرمافزارهای آزاد و متنباز، آموزش لینوکس فدورا و ایجاد فضایی برای تبادل دانش و تجربه، این وبسایت فعالیت خود را آغاز کرد. آن روزها شاید تصور نمیکردیم که […]
This post discusses tools reluctantly written with AI assistance. If you don’t entertain
using them under any circumstance, and think even reading about them legally compromise
your ability to reimplement them yourselves, stop reading now
Today’s post introduces dbranch, a tool to update a Debian package in unstable, and rebuild downstream branches (currently supports Ubuntu PPAs and stable proposed-updates; backports support could be easily added once I have a package that needs it). Eagle-eyed readers might notice the naming similarity with my previous tool ebranch; and the credit for the name and inspiration eventually came from Jens Petersen’s fbrnch.
I was looking forward to this year’s Flock - Fedora Project Conference. I had to cut my vacation short and return from my river trip a day early.
I arrived in Prague at noon on Sunday, just in time to attend František Lachman’s workshop “PR-based Gating for Fedora: Can We Make It Work?�. We all agreed that we want all contributions to happen through pull requests and only after all tests pass. However, we also realized it is not easy: gating for multi-package PRs will be difficult (unless we have a monorepo). Proven packagers will need special handling (can we work on this later?), and first, we need the new Forgejo—everyone is looking forward to it. Coincidentally, later at Devconf, I spoke to Marcela, who mentioned that SUSE already has Forgejo for all packages and that it does not scale as they expected.
But the important message from Miro was: “we should not force maintainers to use a new workflow; we should make it pleasant and easy to use so they want to use it on their own.�
After this workshop, I checked into my hotel and later returned to the venue. We had a passionate conversation with Aleksandra, Miro, and Karolina during a card game. We then moved with several other people to a nearby pub that served Belgian beer. I had only one glass, but I felt very dizzy and tired, so I returned to the hotel sooner than the rest of the gang.
On Monday, I joined Collin for breakfast, and we discussed the dark corners of RPM building.
Then I moved to the venue and listened to Jef Spaleta’s “State of Fedora� talk. Fedora’s usage is growing, and KDE is working enormously well, but the decline in Fedora’s contributors is accelerating. We have to figure out why and take action, rather than just setting up plans without execution.
The “Fedora Council Strategic Proposals� session was great. Two highlights stood out:
“It’s not the new generation that is different. It’s you who is different,â€� said Aleksandra. “They talk about changing a wallpaper; we are talking about burnout.â€�
“I would not join Fedora if it were stable. I want to improve it.” Jef Spaleta
The subsequent “FESCo Q&A� was less vibrant but provided a great opportunity to learn about members' views. For newcomers, Kevin’s remark that FESCo’s role is sometimes to say “No� and stop things was likely interesting. However, FESCo cannot drive new initiatives; individual contributors must do that.
Then I watched Stef’s talk about Hummingbird; this was not new to me as we closely cooperate on agentic packaging.
I also observed the talk “Packit and Fedora: The CI Story Continues�, as it summarized what my team accomplished with Fedora CI. There were many examples, and the feedback was positive. I appreciated that Matej managed to substitute for Nikola, who got sick at the last minute.
Several ideas in the Q&A highlighted that PRs are not mandatory and that maintainers do not always follow upstream in a timely manner.
I attended “The EU CRA vs. Community: Why You’re Safe, and How Stewards Help�. The CRA was new to me, and the session was packed with concrete information. You may check the most interesting slides in my Mastodon post. The biggest takeaway is that open-source maintainers do not need to worry, but they should be prepared that companies using their software will start email-bombing them in August. There was a nice guide on how to politely reject them.
I took a break and talked to people in the hallway, sharing remarks with my team members. I talked with Kashyap about bootstrapping RISC-V; he was thankful for our support in Copr. We drafted next steps, only to find that Kashyap’s colleague already did that while I was on vacation.
After this, I was quite tired, so I headed to the hotel for a quick nap and shower to get ready for the Official Party, which took place at Manifesto—a local market with various street foods. I met many familiar faces as well as new people.
On Tuesday I listened to Adam’s “RHEL 11 is branching from Fedora 46: What this means for youâ€� talk and appreciated his honesty in answering “I do not knowâ€� to several questions. Later in the hallway, we discussed whether branching RHEL during the Fedora branching phase is ideal, and why we don’t branch during the beta phase instead.
I then moved to Kevin’s talk, “Scrapers Gotta Scrape Scrape Scrape�. This topic was familiar, as we are also fighting scrapers, and Jiri from my team helped package Anubis and its dependencies for Fedora. However, I learned several new things, and the follow-up Q&A was fun. If you think people ranting about scrapers only wrote inefficient web applications, you should definitely see a recording of this talk.
I attended Frederick’s talk, “Machine-readable package lifecycle information in repository metadata�. He spoke about their DNF plugin that can set various End-of-Life dates for different packages. This was extremely helpful, and since we wanted to do something similar in RHEL, I immediately emailed the DNF team to introduce them to Frederick.
After the talk, I bowed to Frederick, as he is reportedly the person behind the migration of AWS Linux to Fedora.
I listened to Microsoft’s talk, “Two Years In: Accelerating Microsoft Contribution to Fedoraâ€�. It was interesting to see how much Microsoft uses Fedora. The highlight was that the presenters were from different teams within Microsoft and were largely unaware of each other’s work.
I then talked to people in the corridor, including Bex, who introduced me to two students from Jihlava interested in a high school internship.
The highlight was definitely the lightning talks. They were strictly 5 minutes long (including notebook setup). I learned about current Lenovo support for Linux, how Miro sped up the last Python mass rebuild, and I reported on the current status of the SPDX migration, noting that the next steps will proceed without me. Fortunately, Max started a SIG that wants to take over the work. I shared some information with him that didn’t fit into the lightning talk. Jiri presented his achievement of packaging Goose for Fedora and even provided a live demo!
By this point, I was quite tired, so I passively watched the presentation of the Contributor Recognition Awards. Then, I went into the city to meet a friend, and we attended the theater performance Tahle země je naše.
Tip: If you want to see any mentioned talk, you can find it in the schedule and then rewind to the specific time in the Streams. The edited and cut talks are usually available weeks later on Fedora’s channel.
DevConf
DevConf CZ is huge even. With over one thousand participants and eleven tracks - it is huge. As this is a conference in my hometown I volunteered to help organizers: I chaired one of the rooms and acted as a fire marshall. That includes overseeing A/V tech. Help speakers to connect to video and mic. Make sure they do not run with that. Remind them the time.
This year, I chose a room with Lightning talks. There was 15 minutes for each talk and 5 minutes for a break. And it was a blast!
I chaired the Thursday morning and Friday afternoon. One of the best delivered talks was from new junior who was speaking for the first time - yet she overperformed more senior people.
“AI accelerated learning. Mentorship directed it. Real systems grounded it.” Kaja Prokopova @ “From Sysadmin to Software Engineer: A Non-Traditional Path into Tech”
Anežka Müller speaks about how she organizes a small one-day, one-track conference, Python Pizza. You can use her recipe for free https://docs.python.pizza/
James Freeman with his theory about technology dopamine culture.
Jan Jurca spoke about tests on the filesystem that produce different results if you use the filesystem for some time. You can artificially simulate the usage of filesystem and then run the benchmarks using Filestorm https://github.com/janjurca/Filestorm
See the interesting slides.
I did a small cameo at lightning talk by my student Peter Stefunko in his talk “From SBOM to Dependency Stacks: Making Software Structure Visible“ where he tries to visualize the state of the operating system using famous XKCD comics “Dependency�. You can check his work at github.com/peter-stefunko/rpm2xkcd2347.
I did not attend Thursday’s party as we had an anniversary with my wife and preferred my wife over all of you. 🙂
I met many old faces, former colleagues. And the biggest surprise was the booth of Free Software EU where I found the Czech edition of the book Ada & Zangemann that I helped to translate. I did not know it was already printed - that was because the print was finished that morning. I then called Tomas Stary who took the work from me and who I never met - only to find that he is standing beside me in the queue for ice cream 🙂
Mathias post about the new print https://mastodon.social/@kirschner/116770321504331386
Half of the batch. Fresh from print.
Later we discussed with Tomas, his publisher and Lucie from RH additional steps to advertise the book.
Last week I had to extend complicated, unfamiliar Helm chart. While
I've discussed approach with my carbon-based coworker, actual
editing and extending was done by Claude Code.
Refactors and renamings were tedious, but Claudius did them in no time.
During the review we decided to significantly change the way
chart is organised. Claude executed the changes perfectly.
If I had to fully understand the working of the chart, it would
have taken me a good part of the week. Making the changes – another
day or two. Significant rework midway – I'd probably be too demotivated to
do it.
With AI assistant, I have finished work in two days. Tested, simplified,
cleaned up.
But I have no more skills than I had before. I do not understand this
Helm chart much better. I've spotted a thing or two during the review,
but in no way I have a fuller comprehension. AI let me borrow expertise without acquiring it.
Next time when I'll have to work on this chart, I will use Claude again.
I've got softly locked-in in using AI assistant to work. And I'm not happy with that.
Now, my employer sees this as an acceptable tradeoff. Paid for some tokens,
unlocked couple of my pricy senior-level hours to do other stuff. It balances.
Subsidising Scam Altman and his ilk is a money well spent from this perspective.
We are at the point where we offload more work to so-called AI. Those
are just tools. You know what more primitive tools we had before? Assemblers.
Then compilers. No one writes machine code by hand anymore. Maybe someone
despairs because of it, but people commonly writing binaries are more likely
to be dead by now.
No one treats high level compilers as fad and stuff kids these days do. It's
just a tool, expected to work and dissappering into the background.
There's one important difference. We do not have to pay for the compilers,
thanks to work of the GNU Project last century (gcc) and Apple more recently (llvm/clang).
But at this point we are paying for tokens. I expect that as we
cannot fathom working with any codebase without a compiler now, a LLM-driven
assistant will be required to tackle bigger projects in near future.
To be free, we need to be able to run assistants at reasonable cost.
Free is reasonable. Not paying through the nose for electricity is reasonable.
We can hope for good quality chinese models to be available for free.
We need a free toolkit revolution again. It was GNU Compiler Collection
for the previous generation. Would it be Kimi now?
And we need the proprietary AI bubble to pop and rationality to return.
Way back in September I apologized for the gap since my last update. I am, once again, sorry it’s been so long since I posted an update. I recently presented at talk at Flock 2026 - if you prefer consuming updates via video where live demos fail, you can find that on YouTube.
This post is something of a summary of that talk, but the short version is Siguldry has all the features Sigul has, plus some. That means we can start deploying it in the coming weeks.
PKCS#11
In the last post, I noted I had implemented server-side support for PGP signing. That’s gone now, I’m pleased to say, because I came across a much neater approach. I implemented a small PKCS#11 module inventively called siguldry-pkcs11.
I can’t take credit for any of the ideas I’m about to describe, since I stole them from multiple other projects. Our friends over in Flatcar have to sign things as well, and they use a key stored in Azure Key Vault. The way they use it is that they implemented a minimal PKCS#11 module. That was my primary inspiration, but I also recently spent some time building a drop-in replacement for pesign-daemon and the idea of exposing the signing interface over a Unix socket that can be mounted into isolated build environments was another concept I borrowed.
For those who aren’t familiar, PKCS#11 is a standard that is made up of a C interface. To implement it, you create a shared object file with a few well-known C functions used to discover your implementations of the interface. Users load your shared library, call a well-known function to get a table of functions where you provide your implementations for the various features. What’s nice about this is that common libraries like OpenSSL can speak to PKCS#11 modules, so you can make any tool that uses those common cryptography libraries use your implementations.
PKCS#11 is a fairly broad specification that covers not just signing, but also encryption/decryption, digest calculation, random number generation, and key management (creating, modifying, deleting, etc). What I realized while poking around Flatcar’s implementation is that you actually don’t need to implement everything. You can, instead, stub out all the functions you don’t want to implement and have them return the handy “function not supported” error code. The list of functions I opted to not support is extensive.
In fact, all I implemented was the functions to authenticate, list keys, and sign. However, that’s enough to make it possible to use the Siguldry server with tools including (but definitely not limited to): rpmsign, ostree gpg-sign, cosign (if built with PKCS#11 support), gpg2 (with the gnupg-pkcs11-scd service), sq (once it merges its PKCS#11 support), systemd-measure, systemd-sbsign, pesign, and perhaps most humorously, ssh. All these tools know how to speak to PKCS#11 modules so they “just work” with Siguldry. Any new tool people make will also likely just work.
One thing that’s neat about this approach is that we sit between the tool requesting the signature and the signing server. This means we can hash the content client-side and requests to the server are very small: just a hash, the algorithm used for the hash, and the key name. No more sending a 4GiB ISO over the network only to have the server hash it, that all is done before the request starts.
The last thing that’s really cool about this approach is that if Fedora decides it needs a new signing server and spends some big bucks on a fancy hardware security module, it will definitely ship with a PKCS#11 module of its own so we can start using it with our existing tooling, no development required. If Siguldry stops being the right choice for Fedora, it’s really easy to swap it out.
Unix Socket API
The other thing I did was to implement a Siguldry client that binds a Unix socket and proxies requests from the local system to the remote Siguldry server. I did this because you can expose the socket into locked down environments, such as RPM build environments, install the PKCS#11 module into that environment, and sign things. Right now Fedora uses this general approach for SecureBoot, except it’s specific to the pesign utility. While I’d prefer to disconnect the signing event from the building of a package, it’s important to support the current workflows and this was a neat way to do that which is generic across signing tools.
It has another advantage. Since the Unix socket is set up using a systemd socket unit, we can sandbox the client proxy, and also configure it with the necessary credentials to connect to the Siguldry server and unlock signing keys. This lets you, if necessary, allow anyone to sign content without needing to manage secrets themselves. The secrets can be encrypted with systemd-creds and bound to hardware. All we have to do is ensure the keys that are configured that way get a special flag when queried via PKCS#11: the “protected authentication path” flag.
siguldry-fedora-autopen
The primary way Fedora uses its signing server is via an AMQP consumer that automatically signs things. Robosignatory is the current tool used, but it needs some significant changes to work with this new workflow. In particular, we don’t want to sign content serially, but the fedora-messaging Python library commonly used to interact with the AMQP broker doesn’t handle concurrency well. And, since we now call out to the particular tool used to sign content rather than sending it to the server, the consumer is primarily a mapping between message topics and those various tools.
All those reasons led me to re-implement it, and since I’ve got no imagination with names, it’s called siguldry-fedora-autopen. It uses the lapin AMQP client with Tokio to manage hundreds or thousands of RPMs/ISOs/containers/etc signing operations at the same time. This means builds should no longer languish in the signing queue for many hours.
IMA
All this probably (hopefully) sounds great, but as with all things there are a few problems. Well, really just one big problem. Back in the Fedora 37 days, we enabled IMA signing for RPMs. The short version of how that works is that each file inside an RPM gets signed. And, unfortunately, the PKCS#11 interface is synchronous. This means that rpmsign requests a signature when it creates an OpenPGP signature on the RPM header. It then requests a signature for each file, waiting for each signature request to finish before starting the next one. Now, when the RPM only has dozens or a couple hundred files that happens fairly quickly. A signature might only take a couple milliseconds and the request/response latency might be a couple dozen more milliseconds (or a lot less depending on how close the server is from the client).
Of course, not all RPMs only have a few hundred files. Some have many, many thousand. For these, a simple OpenGPG signature takes less than a second (even if the RPM is, say, 10GiB). The IMA signing can take many minutes, depending on how many files it has. I’ve been running a client locally, with a server in a nearby datacenter, and the largest time I saw for a signing operation was around 40 minutes. This is really unfortunate, but it is important to remember that we can now do thousands of signing operations at the same time, so even when these slow builds are signed, other builds can be serviced in a timely manner.
There’s a couple ways we can make this better, but the “easiest” involve a custom signing implementation that avoids using the PKCS#11 path.
What’s Next?
In the next few weeks I’m hopeful we’ll have enough time to sit down and deploy Siguldry to Fedora’s staging environment. I want to really kick the tires and ensure it’s rock solid before we move it to production, but I think it’s reasonable to hope for that in the next couple of months.
While that’s happening, I’m going to add support for ML-DSA signing. That will allow us to produce signatures that are safe in a post-quantum cryptography world. After that, I’ll investigate OpenPGP’s hybrid signature schemes, although given recent developments I wonder if we want to bother with those. I’ll defer to what the professional cryptographers have to say about that.
Sometime in the next few Fedora releases, though, you should expect to see new signature algorithms in use (pending FESCo approval etc etc).
Comments and Feedback
Thoughts, comments, or feedback greatly welcomed on Mastodon
Another wonderfull flock is in the books. This post is likely
to be kind of long, and was written a few days after I got back
home, so it's likely I forgot some things or misremembered them
somehow. If so, it's not intentional.
TLDR version: Another great flock. Lots of good conversations,
lots of good talks, good food, good friends. I'd like to give kudos
to all the people who put things on. It's not easy planning a large
event like this and at least from my perspective everything went
very smoothly.
Day -3 / -2
My journey started a few days before flock on Thursday the 11th.
I had actually planned to come in a day early to recover from
travel, but it turns out there was a meeting scheduled then, so
I got to recover in the meeting instead (more on that in a minute).
Two hour drive to portland airport (PDX) turned into about 2.5 due
to traffic, but I had planned buffer so it was fine. This time
I was taking icelandair via Keflavík (KEF) instead of my usual
jump via Amsterdam. The transfer time was quite low (around an hour),
but I read that it was easy to transfer there.
The flight was fine, the transfer was a bit fun: They deplaned us out
on a tarmack and we had to take a bus to the terminal. That took a while
as they wanted the bus fully loaded. Then, they said the customs area
was understaffed today and would take longer than normal. So, I rushed
as best I could and ended up at my gate only to hear that the next flight
was waiting for crew and hadn't boarded yet. :)
The hotel in prague was the same one as last years flock. Last year I had
stayed at a nearby hotel, but this year I just stayed at the conference
one. I have to say the rooms were nicer there and it was pretty nice
to not have to walk over and back a lot.
I got in friday afternoon and took a short nap, then met up with Tomas
and Adam for dinner. We picked a nice sounding place nearby and it
was a bunch of nice conversation and food.
Day -1
Saturday I had planned to recover from travel, but instead I went to
a face to face meeting we had with a bunch of the Red Hatters that were
coming to flock. There wasn't anything secret here, it was mostly just
discussing what various groups were working on and how we could all
help each other out and get things landed/working.
There was a lot of technical discussion and planning type things
along with lots of things that were further discussed at flock.
I was able to chime in on some hardware questions and some
cloud resources along with some policy questions.
Saturday evening was the 'sponsors dinner' with a bunch of folks from
companies sponsoring flock along with leeds of various parts of the project.
Amusingly, it was in the same resturant we were at the previous night!
It was all good though. I sat near Jermey Cline, David Duncan and Jef Speleta
and we had a bunch of conversations on tons of topics.
Day 1
The first day this year was workshops, the keynote and other regular talks
would be the next day. I can understand why you might want to do things this
way as it allows you to get people excited and working on something and then
leverage that work for their talk in the later days. On the other hand it means
that the workshops had a lot of talk/presentation before being able to get
to the actual work part of the workshop.
The first slot I ended up in the 'hallway track' (as I would many times in
the coming days). Talked to too many people to even list. :)
Next I went to the forgejo workshop. It was fine and there were a few questions,
but either everyone had burned out their discussions on it, or the folks with
those questions/discussions weren't there as it seemed to go very quietly.
I think lots of contributors are getting used to forge.fedoraproject.org now
and can imgaine how a migrated src.fedoraproject.org might look.
Lunch was up next. This year the mentor summit was doing a 'lunch and learn'
thing each day, and I went each day. I sat at the 'operations' table on Sunday
and had a great conversation with several new to fedora folks. We talked about
matrix a fair bit and software we all use.
After lunch I went to the Fedora Data & Analytics Workshop with Justin and
Michael. I think things were pretty interesting, but we spent a lot of
time getting everyone up to speed on the background and only had a short
time at the end to work on workshoppy things. Still, very interesting stuff
here. We have lots of data, but we dont really analize it very much, and
I am hopefull this system will allow us to do so! Right after this workshop
I got pulled into a hallway track discussion and didn't get a chance to
say Hi to Michael in person. :(
After that was more hallway discussions and then our team had a team dinner.
Always great to see people I work with day to day over the internet in person.
Some of us then rushed back from dinner for...
The flock 2026 candy swap. This year was even bigger I think that last year.
We barely fit in the bar where this was happening. Tons of good stuff, lots of
good stories. A few people said to me "Do you do this every year? Wow, this is
great! I would have brought something if I knew". After the swap, beers in the
bar and I managed to have a nice discussion on Books with MattH and Aoife
along with some music discussions with many others.
Day 2
Monday started out with the keynote state of Fedora from Jef. There was even
a nice bit of stats out of the data workshop that was interesting.
Then up in the same room was the Council round table, followed by the FESCo
one. Interestingly, the Council didn't get any AI related questions, but
FESCo did. Do watch those recordings if you are interested in any of those
questions/answers.
There was a talk on hummingbird after that, which was great. I'm excited
to see hummingbird and to see what it's going to grow into.
Mentor summit lunch and learn monday was the 'infrastructure' table for me.
We had folks from Azure linux and Amazon linux (both now based on Fedora)
asking a bunch of infrastructure questions. I also asked them a bunch about
their infrastructure too. It was very encouraging this year to not only see
groups like this at flock, but asking how they can be more involved and help
Fedora out and thus help themselves as downstreams. I'm really hopefull we
can help each other and all end up better for it. This was probibly the most
encouraging thing I saw at flock this year. :)
After lunch I went to Lenkas Forgejo runners on fedora forge talk. This was
largely stuff I knew about, but it was great to see all in one place. There
were some nice questions. We are definitely seeing some good use from these
runners and it's good to see work on them continue.
I had a bunch of hallway conversations after that, then off to...
Post-quantum cryptography for Fedora infrastructure with Alexander and Jakob.
The timelines here are really scary to me. There's a lot thats just starting
to land now, but the dates for moving things are coming up super fast.
I hope it will all work out, but it's a lot of things and a lot of work. :(
Dinner Monday was the flock reception. It was in a nearby outdoor food court,
which I had actually had lunch at last year. It was a nice setup, they blocked
off a large area of tables for us and gave us vouchers to get a meal at any of
the... many food places. They also brought us all a bunch of free appitizers
to the point where it almost wasn't worth getting a real meal. :)
Had a lot of conversations on a lot of topics here. I managed to find Michael
Winters and we started to talk, but then they called the group photo
and we were seperated, then after that we both got pulled into other conversations.
(This was a theme)
A group of us then went to a belgian beer place and stayed talking until
the wee hours.
Day 3
The last day already!
I started with the "RHEL11 and what it means for you" talk. Nice to see dates
listed there and an attempt to get fedora things landed in time for RHEL11.
Next up was the state of the fedora kernel from Justin. Nothing too much that
I didn't know, but was good to clarify things and hear questions from folks.
In the next slot I really wanted to go to Kashyap's riscv talk, but unfortunately
that was when _my_ talk was scheduled also. ;( So, I gave my talk about scrapers.
It was a pretty nice crowd, and there were lots of good questions from people.
I did have AV problems however, they were unable to capture from the projector
for the live stream, so they could only use the camera to capture it. I have
no idea why that was happening. Somehow also, I wasn't able to read my notes
while giving the talk, so I missed saying a few things I meant to mention.
Things like: "robots.txt was orig called RobotsNotWanted.txt", and asking if
anyone remembered the /. effect. ;) Overall I think it went well.
After a bit of hallway conversation, next up with the talk about Microsoft
contributions to Fedora. It's great for this work to get highlighted. I am
very happy with all the work Jeremy has been doing on our signing infrastructure,
along with all the other places they contribute.
I then went to the "Upgrading Fedora Infrastructure from Nagios to Zabbix" talk.
This was given by Michal, because Greg was unable to attend. He did a fine job
going over things, and Greg was in the matrix room for the talk answering questions.
I'm super happy about this finally getting over the finish line (at least the
initial one) I think we are in a much better place.
Last day of Mentor summit lunch and learn I sat at the Release Engineering table.
We had a mix of folks this time. Some Amazon linux folks, someone asking how to
get involved that was just starting out, and someone who was involved in server/docs.
We had a pretty wide ranging conversation.
After lunch were the lightning talks. There were a bunch of them, and I was
amazed at how many folks had slides. I guess they were ready to talk about their
thing at a minutes notice. Lots of interesting stuff in there. Worth a video
re-watch.
Finally, the last session of the conference: The Fedora’s Contributor Recognition
Program. I won this award last year, and was involved a small amount with choosing
winners this year. All the proposed candidates were great! Fedora is nothing without
it's contibutors and the folks awarded were all super well deserving folks.
Make sure you say 'thanks' to those who help you.
With the conference officially over, I was very tired, so I went to take a nap.
Michael Winters was down in the bar, and I said I would try and meet up after
I got up. So, after my nap I wandered down and... I swear I saw him walking off
to dinner with a group of folks. Missed again. (I'm not avoiding you Michael!
Honest!)
I ended up at dinner with a small group and we actually talked non computer nerd
things ( music, podcasts, and even politics ).
Day 4
The next day was the travel home. Due to timezones I would leave the Prague airport
at 2pm and get into portland at like 5:30pm. (it is NOT 3.5 hours of flights).
I again had fun with the transfer in iceland. Our plane from Pague was a few minutes
late, then we needed to take the bus to the terminal, then passport control was
even more backed up than it was last time. I finally got to my gate and their
was an attendet there who asked me my name. I then took a bus with only 3 other people
on it to the plane. They held it for us, but I made it.
The rest of the trip back was uneventfull, but will take a while to recover.
Luckily, there's a holiday this friday so I do have a 3 day weekend.
A few Downsides
Overall I think things went super nicely and smoothly, but:
The weird AV issues for my talk. It's likely something off about my aarch64 laptop.
So, perhaps I should take a boring x86_64 one next time. Or spend some time poking
at it before time for my talk.
There were a lot of folks that I usually look forward to talking with in person that were
not able to make it this time. Due to various reasons, but it was sad to not see them.
You all know who you are, you were missed.
Hallway conversations
There were a ton of hallway conversations, and I am sure I don't remember most of them
but the few I do:
Alexander showed me a new rust based IDP setup he has been working on. Super cool looking
and very on point for Fedora.
There was a number of AI conversations, but more of the 'what do you think is going
to happen when the bubble bursts' than anything else. There was a foundations whiteboard
where people could add things around the 4 foundations and under friends was "more people,
less AI".
Lots of good talks with Amazon Linux and Azure Linux folks. I hope to see them chime in
on matrix with questions/comments/contributions.
Jeremy and I had a number of talks about the new signing stuff.
Nice to see Toshio again and talk with him on lots of things.
Had some good talks about 2fa and otp and such with Gotmax23.
A conversation I had a number of times was "where should we do flock next year?".
There were a lot of ideas, no telling what will win out. We might not do too badly
to just do it in Prague again, IMHO.
Release Candidate versions are available in the testing repository for Fedora and Enterprise Linux (RHEL / CentOS / Alma / Rocky and other clones) to allow more people to test them. They are available as Software Collections, for parallel installation, the perfect solution for such tests, and as base packages.
RPMs of PHP version 8.5.8RC1 are available
as base packages in the remi-modular-test for Fedora 42-44 and Enterprise Linux≥ 8
as SCL in remi-test repository
RPMs of PHP version 8.4.23RC1 are available
as base packages in the remi-modular-test for Fedora 42-44 and Enterprise Linux≥ 8
as SCL in remi-test repository
ℹ️ The packages are available for x86_64 and aarch64.
ℹ️ PHP version 8.3 is now in security mode only, so no more RC will be released.
Did your pip install fail with longintrepr.h: No such file or directory? The file likely is on your system, but it sometime or another it was moved, from /usr/include/python3.xx/longintrepr.hto /usr/include/python3.xx/cpython/longintrepr.h. The proper fix is to update the package in question with the new path, but if you’re installing an old version of something or a package that’s no longer maintained you can work around it like this:
This post is tough to write because Flock to Fedora is my
favorite conference, and last year’s Flock might have been the
best conference I’ve ever been to. I love the Fedora community, Prague is
beautiful, the venue is nice, we always have so many interesting talks and
workshops, the organizers do an amazing job preparing this event for us, so I
feel really guilty saying that I did not have much fun this year. That is 100%
on me, though. Flock bears no blame.
It wasn’t exactly the wisest decision ever to run
my first half-marathon the very evening before the conference,
and then waking up early to commute to Prague. It must have hit me much more
than I was willing to admit. I feel fine physically, but I can’t pay attention
to anything because nothing feels exciting. That is the most lifeless I’ve felt
in a long time.
So, in case anyone is wondering … it was me, not you.
Highlights
Despite all that, these were the highlights of the event for me.
Great job on the migration from pagure.io to
forge.fedoraproject.org. It was
handled exceptionally well. Next, we look forward to the migration of
src.fedoraproject.org to Forgejo, which can’t come
soon enough. Tomáš Hrčka teased us that this migration may take
considerably less time than the previous one because they already know how to do
things (e.g., deploy Forgejo, etc).
Ellis Low showed us how to run an LLM on our laptops through Llama
and how to interact with it. I really appreciated him saying that, except for
this one small text-in, text-out magic black box, everything else is standard
software engineering as we know it.
Most of the discussion revolved around a decline of new contributors and us
failing to connect with the next generation. “They talk about changing
wallpaper, we talk about burnout” – Aleksandra Fedorova.
As presentations go, Kevin Fenzi’s talk about AI scrapers was
the best. It started with minor A/V dificulities, which is a nice reminder
to not feel bad, the next time it happens to you, because it happens to
everybody. Even the main infra guy. Semi-joking aside, I enjoyed the brief
history of web scraping and Kevin’s taxonomy of scrapers, clearly showing that
not all scrapers are the same (e.g.,
web.archive.org). However, the current AI scrapers
are the plague of the internet, and I am so glad the Fedora Infra team fights
them as best as they can to keep our services running.
Chat with Adam
I ran into my former Copr team mate Adam Šamalík in the hallway,
and we had a nice chat about life. When we still worked together (10 years ago,
oh how the time flies), I randomly asked him what he did on a weekend. “We had
no plans, so we bought plane tickets and went on a date to Amsterdam with my
girlfriend”. Fucking what?! That was the coolest response ever. Since that
day, I remember it every time my creaking, lazy bones struggle to do something
spontaneous. Why am I telling it now? Adam is now happily living in the
Netherlands full-time. Funny how one impromptu decision can completely change
your life.
We use gitlab to post internal releases of archives. I want to be able to automate the process of downloading these artifacts. Here it is using the glab CLI.
Note that each of these commands can be done with the glab api subcommand instead, but the path needs to be escaped.
export REPO=YourCompany%2Fsw-eng%2Fproduct-manifest
glab api "$REPO/releases"
glab api "$REPO/releases/item%2F5.4.3.2"
This last one gives you the URLs for direct download. You want to save them to a file via redirection.
glab api "https://gitlab.com/api/v4/projects/99999999/packages/generic/sw_release/1.2.3.4/eng_1.2.3.4-product-name-version-version.tar.xz" > eng_1.2.3.4-product-name-version-version.tar.xz
Ce week-end marque la sortie de deux versions majeures de mes projets SeedboxSync et SeedboxSyncFrontend. Au programme : support FTP, optimisations de performances, suivi en temps réel des téléchargements, administration depuis l'interface web et plusieurs améliorations de l'expérience utilisateur.
Some time has passed since I've tightened TLS settings
on my home server. Let's move it a notch higher, this time including home k3s cluster.
Use ECC certificates
In 2026, using elliptic curves cryptography certificates should be the norm. Fortunately,
automatically obtaining them is easy. I'm using cert-manager for kubernetes ingresses.
Switching to ECC is a just a matter of adding an algorithm annotation on Ingress:
and removing the secret containing old cert and key.
Small caveat: FreeIPA still lags. While it supports ACME protocol,
ECC through it is not possible, yet. I've left my internal domains with RSA certificates.
For the main server, I refresh certificates using small Ruby script. I had the change RSA.new(3072) to
an EC key generation, rest happened automatically:
Last time I've limited support of Transport Layer Security to versions 1.2 and 1.3. Today, let's allow the latest only.
I don't care about supporting Windows 7-era clients (years out of support).
Ingress on k3s is handled by Traefik. Simplest way to influence its config is by creating a global (named default) TLS
configuration option:
This project does not allow contributions generated by large languages models (LLMs) and chatbots. This ban includes, but is not limited to, tools like ChatGPT, Claude, Copilot, DeepSeek, and Devin AI. We are taking these steps as precaution due to the potential negative influence of AI generated content on quality, as well as likely copyright violations.
This ban of AI generated content applies to all parts of the projects, including, but not limited to, code, documentation, issues, and artworks. An exception applies for purely translating texts for issues and comments to English.
AI tools can be used to answer questions and find information. However, we encourage contributors to avoid them in favor of using existing documentation and our chats and forums. Since AI generated information is frequently misleading or false, we cannot supply support on anything referencing AI output.
I won’t attempt to argue that you should allow use of AI for writing code. If you wish to ban LLM-generated code, fine. That’s probably inadvisable, but I am not going to object.
But this policy is far stricter than that. Notably, it strictly prohibits AI-generated content in issue reports (except to translate text). Don’t do this! Prohibiting bug reports is stupid and just makes your software worse. Please make sure your project’s AI policy allows for at least AI-generated static analysis results and AI-generated vulnerability reports. Otherwise, you prohibit entirely unobjectionable problem reports.
It’s hard to imagine what could possibly be the value of prohibiting valid bug reports. AI-generated static analysis works well: the AI is able to think about your code, follow execution paths, and automatically discard most false positives to avoid bothering you with them, and the quality of reports is generally pretty high. They are far from perfect, but the same is true of humans.
2. Resource leak in update_credentials_cb on gnutls_credentials_set failure
File: tls/gnutls/gtlsconnection-gnutls.c:169-172
When gnutls_credentials_set() fails, the function returns without calling g_gnutls_certificate_credentials_unref(credentials). The credentials was either freshly allocated or ref-bumped, so it leaks.
Pasting this into an issue report clearly violates the ban on AI-generated content. And yet, why would you not want to receive a clear and concrete bug report for memory leak?
I understand not all maintainers are fond of AI, but is your dislike really so extreme that you would choose to ignore valid problems and intentionally make your software worse? If not, then your AI policy should thoughtfully consider how to handle AI-generated content in issue reports. Certainly do not adopt a policy that outright bans all AI-generated content in issue reports.
As an issue reporter, you could theoretically take the problem found by the AI and rephrase all the words, then claim that it is no longer AI-generated content because it is rewritten. This is a waste of time and usually results in a lower-quality, less-detailed result, but you could plausibly do that. Or, if you want to go above and beyond, you could just jump ahead to creating a merge request. But realistically, if your project does not allow any use of AI in issue reports, it’s more likely that either (a) you won’t receive the issue report in the first place, or (b) you won’t receive such issue reports from experienced developers who read and respect your policy, while users who do not read your policy will continue to submit them.
What about security vulnerability reports? Since the start of this year, I have reviewed well over 100 vulnerability reports that I strongly suspect were generated by AI. To reach the “over 100” claim, I sadly only considered vulnerability reports submitted during a particularly heavy four week period, so this is an extremely loose lower bound. Suffice to say, I have seen a lot of them. The quality varies dramatically. Vulnerability reports are now often better or worse than before: better because an experienced human working with a good AI is able to find vulnerabilities that would have surely gone unnoticed without AI, and worse because an inexperienced human with a bad AI might create some pretty terrible issue reports, a significant proportion of which are just outright spam. Low-quality reports remain a problem, but nowadays most AI-generated issue reports are quite good.
Maintainers do not need to tolerate spammy vulnerability reports. If an issue report is bad, of course go ahead and close it. If it’s really bad, then I sometimes don’t even bother replying. But banning good vulnerability reports solely because some portion of the report was generated by AI is unacceptable. AI-assisted vulnerability reports are the new industry standard, and this is not likely to change. Prohibiting issue reports reduces the quality and safety of your software, punishing your users. This is too extreme.
Unny was using a font based on his handwriting style for the cartoons, designed by K.H. Hussain of Rachana. Recently, a new font designed by Varshini KVSS & ‘Kandam Collective’ is developed by Rachana Institute of Typography to use in the cartoons, and it is released as open source — see the specimen and download links.
The character set of the font is Latin only. There are plenty of alternate glyphs (for upper case and lower cases of i, j, l, g, etc. — for instance check the double ‘l’ in ‘Intelligence’ on the specimen above). Such characters are rendered alternately to give a feel of the randomness that handwriting evokes.
The source (and issue tracking) are available at RIT fonts repository.
I have a bunch of different ARM SBCs, some Raspberry Pis, some Rockchip based, and some others. Some of them I have in 1U rack mount cases. Some in cases that I have 3d printed. They have all had a common issue. When something goes wrong, I need to unplug them, move them to my desk, and connect them to my desktop via a USB to tty adaptor. Aside from being inconvenient, it also meant I had to be home to debug what was happening.
I figured there had to be a better way. In the end, I used Claude to help me write a solution. What I came up with is a project to make an ESP32 Web Terminal. Using an ESP32 wired to the UART on the SBCs, I can securely access the serial console of my devices. I have used a couple of different ESP32 devices, from a USD$3 ESP32 C3 Mini to a USD$8 XIAO ESP32S3. all of which work well. For the Raspberry Pis, I purchased some premade JST SH1.0 mm 3 Pin Wire connector cables to plug into the uart port, and for other devices, I made some custom cables with 2.54 mm Dupont crimp pin connectors.
I have most of my SBC’s powered by PoE. In the pictured example, I soldered some headers onto the PCB for the PoE hat to use to provide a 5V power source to the ESP32, and it all runs self-contained in the rack-mount unit. I do want to work on making the connections a little neater and the housing better. But for now, this is functional. While the software on the ESP allows resetting the SBC and controlling the power, I do not currently have that wired up.
As for provisioning the ESPs, I run FreeIPA at home for authentication, and I have dogtag set up as a CA. When I have wired up and flashed a new board, it initially runs as an access point I connect to in order to set up my home network. Once it is connected to my network, I run an Ansible playbook to create and upload SSL certificates signed by my CA and change the admin password. That way, I can connect without any SSL warnings and use a known non-default password to log in. So far, I am quite happy with how they have performed. I still have some things I want to make better, and I also want to finish testing a setup where I connect to an SBC that exposes its serial port over USB. In theory, it should work with ESP32S3, I just need to test.
While it has added quite a few more devices to my network, it is useful to be able to access and debug what is happening without having to move devices and plug them into another computer. I have considered options for externally powering the ESPs and the best ways to connect the ESPs to the SBCs. I would like to at least be more robust with a 3d printed enclosure for the ESP. Each unit has cost me between USD$5 and USD$12, and each has acceptable performance. I have intentionally stuck to using small ESP’s, though it would work well with bigger devices also. Powering each through a shared power source would require additional circuitry to isolate them and prevent voltage leaks.
This post discusses tools reluctantly written with AI assistance. If you don’t entertain
using them under any circumstance, and think even reading about them legally compromise
your ability to reimplement them yourselves, stop reading now
Happy Friday! Of course it’s not Friday anymore in Asia, and if I finish this post in time, it’s still the work day in the Western Hemisphere.
When you work in a large distributed project, it’s not dissimilar to working in a large multinational company - there are too many people to know everyone (or at least not at once!) and you might not know where someone is and if you’re pinging them in the middle of the night.
Software Engineering Radio is a podcast for people in IT/development with over 700 episodes across many topics over 20 years. They haven't touched on the Linux kernel much. I was invited on as part of my role at Red Hat as a Distinguished Engineer, but the podcast is really an insight into kernel maintenance, in graphics and beyond, touching on the scope and scale of the project.
It was my first time to record something that wasn't just me talking at a conference/meetup, and it was all very professional, with sound checks and brainstorming before hand.
The content is at a pretty broad and introductory level. We talked about kernel development processes, maintenance processes, and we touch on rust in the kernel a bit. It's mostly about the sheer size and scale of the project and how Linus releases things, how trees get to Linus and how the GPU work is done.
Using a system with 80 AArch64 cores can be a pleasure. Or a pain…
Multicore heaven?
Having 80 cores sounds nice, doesn’t it? But not so much during actual use…
You see, building Fedora packages was flying by. With all cores in use, ccache
buffers filling up (in case of rebuilds), and 128 GB of RAM in constant use, etc.
But at the same time, 100% load on all cores means you cannot listen to music
on Spotify or watch online videos, etc. All that because the CPU cores are
occupied by the build processes.
I tried to use cgroups to limit cpu.max for each fedpkg mockbuild call. It
did not help much: the audio was still jerky.
To compare: I wrote this post on a system powered by a Ryzen 5 3600 CPU while a
package build was running in the background. All twelve CPU threads were 100%
busy, yet the music did not skip.
All of this shows that cores-heavy CPUs are perhaps not a good choice for a
desktop machine. Latencies, the scheduler and context switching — all of this
introduces enough noise to make a desktop user suffer.
The lack of single-thread speed
Arm processors are good in many cases, as long as you do not need pure,
single-thread, CPU power.
It is very noticeable in a web browser. For example, Bitwarden unlocks with a
noticeable delay, while on a Ryzen 5 3600, it is nearly instant. And it feels even
worse when you watch some YouTube videos like “who will make faster PC on a €100
budget”, and then you run the same browser benchmark and get worse results…
Many software builds also highlight this problem. I have a feeling that
developers have grown used to a small number of fast CPU cores, which is the
norm on the x86-64 architecture, and their code is written to take it for granted.
And then you look at your machine, where 70 cores do nothing, waiting for some
code to finally compile or link. I have seen one software package where the
bootstrap was composed of TWO source files. Both were over two megabytes in
size and full of machine-generated C code. Two cores were kept busy for quite a
while, while the other 78 had to wait.
Of course, there are also packages which will take all cores, whole memory and
as much swap as possible, and do magic in nearly no time. When I started build of
the PrusaSlicer package, I had to add some swap because Firefox was gone due
to OOM. Having less than 2 GB of RAM per CPU core really sucks ;D
Summary
To use a desktop system you do not need many cores. As long as they are fast.
This is an independent, censorship-resistant site run by volunteers. This site and the blogs of individual volunteers are not officially affiliated with or endorsed by the Fedora Project.
Poland.
“AI accelerated learning. Mentorship directed it. Real systems grounded it.” Kaja Prokopova @ “From Sysadmin to Software Engineer: A Non-Traditional Path into Tech”
Petr Kaška spoke about detecting malicious prompts. You can attack your model using 
Anežka Müller speaks about how she organizes a small one-day, one-track conference, Python Pizza. You can use her recipe for free 
James Freeman with his theory about technology dopamine culture.
Half of the batch. Fresh from print.
