~helmut/crossqa.git
12 days agosimplify common functions master
Helmut Grohne [Fri, 19 Feb 2021 21:05:54 +0000 (22:05 +0100)]
simplify common functions

4 months agobuild.py: fix invalid column name
Helmut Grohne [Sun, 1 Nov 2020 11:26:54 +0000 (12:26 +0100)]
build.py: fix invalid column name

Fixes: f6d219e1a4fd ("add build architecture to schema of depcheck table")

4 months agodepcheck.py: avoid variable type change
Helmut Grohne [Sat, 17 Oct 2020 05:22:14 +0000 (07:22 +0200)]
depcheck.py: avoid variable type change

mypy trips over bits changing the type from str to int.

4 months agobuild.py: fix build/native confusion in function name
Helmut Grohne [Sun, 11 Oct 2020 17:39:45 +0000 (19:39 +0200)]
build.py: fix build/native confusion in function name

4 months agowebapp.py: improve parameter validation for /schedule
Helmut Grohne [Sat, 10 Oct 2020 07:32:27 +0000 (09:32 +0200)]
webapp.py: improve parameter validation for /schedule

Allow scheduling any_arm64. Disallow scheduling garbage_any.

4 months agodepcheck.py: delete unused functions
Helmut Grohne [Fri, 9 Oct 2020 20:48:39 +0000 (22:48 +0200)]
depcheck.py: delete unused functions

4 months agoadd build architecture to schema of depcheck table
Helmut Grohne [Thu, 8 Oct 2020 20:13:57 +0000 (22:13 +0200)]
add build architecture to schema of depcheck table

4 months agowebapp.py: really fix schedule response
Helmut Grohne [Thu, 8 Oct 2020 19:54:15 +0000 (21:54 +0200)]
webapp.py: really fix schedule response

Also handle invalid form values with a BadRequest.

Fixes: bab219106d1b ("webapp.py: fix schedule response")
Fixes: a52d12012b1b ("webapp: add concept of "build architecture" to frontend")

4 months agoadd build architecture to schema of depstate table
Helmut Grohne [Thu, 8 Oct 2020 15:16:06 +0000 (17:16 +0200)]
add build architecture to schema of depstate table

4 months agowebapp.py: fix schedule response
Helmut Grohne [Thu, 8 Oct 2020 14:45:45 +0000 (16:45 +0200)]
webapp.py: fix schedule response

Fixes: a52d12012b1b ("webapp: add concept of "build architecture" to frontend")

4 months agobuild.py: add option --buildarch
Helmut Grohne [Thu, 8 Oct 2020 11:40:46 +0000 (13:40 +0200)]
build.py: add option --buildarch

This allows selecting the architecture of the build server defaulting to
the local native architecture.

4 months agoadd build architecture to schema of buildrequests table
Helmut Grohne [Thu, 8 Oct 2020 11:11:15 +0000 (13:11 +0200)]
add build architecture to schema of buildrequests table

4 months agobuild.py: fix up sql statement after refactoring
Helmut Grohne [Thu, 8 Oct 2020 05:17:57 +0000 (07:17 +0200)]
build.py: fix up sql statement after refactoring

Fixes: d8a9b7aa8c98 ("add build architecture to schema of building table")

4 months agobuild.py: fix up sql statement after refactoring
Helmut Grohne [Wed, 7 Oct 2020 20:04:28 +0000 (22:04 +0200)]
build.py: fix up sql statement after refactoring

Fixes: 7b5da3c5b65e ("add build architecture to schema of builds table")

4 months agoadd build architecture to schema of building table
Helmut Grohne [Wed, 7 Oct 2020 16:02:42 +0000 (18:02 +0200)]
add build architecture to schema of building table

4 months agoadd build architecture to schema of builds table
Helmut Grohne [Wed, 7 Oct 2020 14:48:12 +0000 (16:48 +0200)]
add build architecture to schema of builds table

4 months agowebapp: add concept of "build architecture" to frontend
Helmut Grohne [Wed, 7 Oct 2020 14:41:14 +0000 (16:41 +0200)]
webapp: add concept of "build architecture" to frontend

The frontend now pretends that we had a concept of a "build
architecture" in its visual appearance. Whenever dealing with, hard code
"amd64" though. That allows refactoring the underlying database without
changing how the frontend looks.

4 months agodepcheck.py: reduce use of BUILD_ARCH
Helmut Grohne [Mon, 5 Oct 2020 21:36:02 +0000 (23:36 +0200)]
depcheck.py: reduce use of BUILD_ARCH

4 months agosupport concurrent builds
Helmut Grohne [Mon, 5 Oct 2020 19:55:42 +0000 (21:55 +0200)]
support concurrent builds

Add new table "building" for recording builds in progress. When a
concurrent build is issued, it will avoid building a source package that
is presently built. It will also avoid building for an architecture that
is presently built for. The latter condition helps reduce the number of
satisfiability issues.

4 months agobuild.sh: use sbuild/lintian integration
Helmut Grohne [Mon, 5 Oct 2020 17:45:41 +0000 (19:45 +0200)]
build.sh: use sbuild/lintian integration

Since upgrading from stretch to buster, the lintian integration actually
works. Rely on it.

4 months agodrop mips as a release architecture
Helmut Grohne [Sat, 3 Oct 2020 22:07:05 +0000 (00:07 +0200)]
drop mips as a release architecture

4 months agodepcheck: add python3.9-minimal to bad_foreign_packages
Helmut Grohne [Sat, 3 Oct 2020 22:06:51 +0000 (00:06 +0200)]
depcheck: add python3.9-minimal to bad_foreign_packages

12 months agodepcheck: add python3.8-minimal to bad_foreign_packages
Helmut Grohne [Sun, 16 Feb 2020 08:50:18 +0000 (09:50 +0100)]
depcheck: add python3.8-minimal to bad_foreign_packages

13 months agofetch and display bugs
Helmut Grohne [Mon, 13 Jan 2020 19:07:00 +0000 (20:07 +0100)]
fetch and display bugs

Retrieve bugs from udd-mirror by tags/usertags and display relevant bug
numbers in the webapp.

13 months agoschema.sql: index builds on source
Helmut Grohne [Mon, 13 Jan 2020 19:00:39 +0000 (20:00 +0100)]
schema.sql: index builds on source

The per-source page rendered by webapp.py incurs a lot of query that can
be sped up using this index.

15 months agodepcheck: ignore givebacks
Helmut Grohne [Tue, 5 Nov 2019 05:04:21 +0000 (06:04 +0100)]
depcheck: ignore givebacks

When an architecture encounters an installation error, the chance that
dose and apt disagree is very high. Thus skip the relevant architecture
until the next mirror push entirely.

17 months agodepcheck: don't litter temporary files on error
Helmut Grohne [Sun, 8 Sep 2019 09:06:51 +0000 (11:06 +0200)]
depcheck: don't litter temporary files on error

When a multiprocessing.Pool is garbage collected or __exit__ed, it
calls the terminate method. In that case, temporary files are not
cleaned up. This happens for example, when an exception is raised in one
of the workers. Thus we close and join the pool explicitly.

19 months agobuild.py: handle errors from ssh in a better way
Helmut Grohne [Thu, 1 Aug 2019 08:12:37 +0000 (10:12 +0200)]
build.py: handle errors from ssh in a better way

Previously, it would retry immediately and produce empty log files
marked as failures. When ssh fails, we no longer record a build nor log.
We also wait a little before retrying.

21 months agowebapp: use a package-specific debcheck link
Helmut Grohne [Fri, 10 May 2019 15:10:08 +0000 (17:10 +0200)]
webapp: use a package-specific debcheck link

Reported-by: Paul Wise <pabs@debian.org>
21 months agowebapp: add an index listing recently failed builds
Helmut Grohne [Thu, 9 May 2019 19:54:42 +0000 (21:54 +0200)]
webapp: add an index listing recently failed builds

23 months agowebapp.py: sort the architecture drop down
Helmut Grohne [Tue, 26 Mar 2019 20:35:12 +0000 (21:35 +0100)]
webapp.py: sort the architecture drop down

23 months agotable buildreqeusts should have an explicit id
Helmut Grohne [Sun, 24 Mar 2019 06:25:38 +0000 (07:25 +0100)]
table buildreqeusts should have an explicit id

We rely on the id being constant. However, sqlite does not guarantee
that the implicit rowid stays constant across vacuum. Making the rowid
explicit fixes that.

23 months agobuild.py: add back random ordering
Helmut Grohne [Sat, 23 Mar 2019 12:22:47 +0000 (13:22 +0100)]
build.py: add back random ordering

Otherwise "any" builds will always schedule for arm64.

23 months agobuild.py: prefer older buildrequests
Helmut Grohne [Sat, 23 Mar 2019 11:48:09 +0000 (12:48 +0100)]
build.py: prefer older buildrequests

23 months agoallow scheduling builds via webapp
Helmut Grohne [Sat, 23 Mar 2019 09:06:16 +0000 (10:06 +0100)]
allow scheduling builds via webapp

23 months agowebapp.py: fix html structure
Helmut Grohne [Tue, 19 Mar 2019 20:09:46 +0000 (21:09 +0100)]
webapp.py: fix html structure

<footer> must be inside <body>

23 months agowebapp.py: add a footer pointing to the source
Helmut Grohne [Sun, 17 Mar 2019 21:01:52 +0000 (22:01 +0100)]
webapp.py: add a footer pointing to the source

23 months agolicense the beast as gpl-2+
Helmut Grohne [Sun, 17 Mar 2019 21:00:59 +0000 (22:00 +0100)]
license the beast as gpl-2+

23 months agowebapp.py: make it prettier
Helmut Grohne [Wed, 13 Mar 2019 20:26:05 +0000 (21:26 +0100)]
webapp.py: make it prettier

Thanks to Paul Wise, Guillem Jover and itd@oftc for their feedback.

23 months agowebapp: avoid raising an unhandled FileNotFoundError
Helmut Grohne [Tue, 12 Mar 2019 05:17:02 +0000 (06:17 +0100)]
webapp: avoid raising an unhandled FileNotFoundError

23 months agowebapp: parse rounded timestamps
Helmut Grohne [Wed, 6 Mar 2019 07:18:38 +0000 (08:18 +0100)]
webapp: parse rounded timestamps

23 months agowebapp: get rid of the annoying sqlalchemy warning
Helmut Grohne [Tue, 5 Mar 2019 20:45:45 +0000 (21:45 +0100)]
webapp: get rid of the annoying sqlalchemy warning

The feature is irrelevant here as the ORM isn't being used.

23 months agowebapp: jinja 2.8 doesn't support dotted assignments
Helmut Grohne [Tue, 5 Mar 2019 20:27:03 +0000 (21:27 +0100)]
webapp: jinja 2.8 doesn't support dotted assignments

23 months agowebapp: only show the bootstrap.d.n link when there is a problem
Helmut Grohne [Tue, 5 Mar 2019 20:18:34 +0000 (21:18 +0100)]
webapp: only show the bootstrap.d.n link when there is a problem

Thanks to Johannes Schauer for the suggestion.

23 months agowebapp: remove the architecture compaction
Helmut Grohne [Tue, 5 Mar 2019 19:48:39 +0000 (20:48 +0100)]
webapp: remove the architecture compaction

It confused Guillem Jover at least.

23 months agoadd initial web presentation
Helmut Grohne [Tue, 5 Mar 2019 19:15:19 +0000 (20:15 +0100)]
add initial web presentation

23 months agoschema.sql: improve unique index
Helmut Grohne [Tue, 5 Mar 2019 19:01:31 +0000 (20:01 +0100)]
schema.sql: improve unique index

Most of the time, we'll query depstate by source. Thus it should come
first in the uniqueness constraint.

23 months agodepcheck.py: abort early when nothing needs checking
Helmut Grohne [Tue, 5 Mar 2019 18:58:07 +0000 (19:58 +0100)]
depcheck.py: abort early when nothing needs checking

23 months agoreduce timestamps to 1s resolution
Helmut Grohne [Tue, 5 Mar 2019 18:47:32 +0000 (19:47 +0100)]
reduce timestamps to 1s resolution

We don't need more resolution. It is a distraction and removing it saves
around 1% of db size.

2 years agodepcheck: support parallel operation
Helmut Grohne [Sat, 2 Mar 2019 15:10:13 +0000 (16:10 +0100)]
depcheck: support parallel operation

2 years agodepcheck: remove the 6h condition
Helmut Grohne [Sat, 2 Mar 2019 14:41:15 +0000 (15:41 +0100)]
depcheck: remove the 6h condition

If the release timestamp doesn't change, we should assume that the
content didn't either.

2 years agodepcheck.py: move the filter into sql
Helmut Grohne [Sat, 2 Mar 2019 12:55:43 +0000 (13:55 +0100)]
depcheck.py: move the filter into sql

2 years agodepcheck: elide redundant element from tuple
Helmut Grohne [Sat, 2 Mar 2019 09:36:45 +0000 (10:36 +0100)]
depcheck: elide redundant element from tuple

2 years agobuild.py: make the build server configurable
Helmut Grohne [Sat, 2 Mar 2019 07:27:12 +0000 (08:27 +0100)]
build.py: make the build server configurable

2 years agodepcheck.py: make the mirror configurable
Helmut Grohne [Tue, 26 Feb 2019 06:48:03 +0000 (07:48 +0100)]
depcheck.py: make the mirror configurable

2 years agoadd a basic .gitignore
Helmut Grohne [Sat, 23 Feb 2019 18:51:05 +0000 (19:51 +0100)]
add a basic .gitignore

2 years agoadd type hints to common.py
Helmut Grohne [Sat, 23 Feb 2019 18:50:47 +0000 (19:50 +0100)]
add type hints to common.py

2 years agobuild.sh: add a lintian tag
Helmut Grohne [Mon, 11 Feb 2019 04:34:23 +0000 (05:34 +0100)]
build.sh: add a lintian tag

The triplet-dir-and-architecture-mismatch detects installing files to
${DEB_BUILD_MULTIARCH}. It is very relevant to cross building and rarely
happens elsewhere.

2 years agobuild.py: handle the case of no satisfiable packages
Helmut Grohne [Sun, 10 Feb 2019 13:49:23 +0000 (14:49 +0100)]
build.py: handle the case of no satisfiable packages

2 years agomake depcheck.py compatible with stretch
Helmut Grohne [Fri, 8 Feb 2019 18:21:27 +0000 (19:21 +0100)]
make depcheck.py compatible with stretch

2 years agomove yield_chunks to common.py
Helmut Grohne [Tue, 5 Feb 2019 16:19:49 +0000 (17:19 +0100)]
move yield_chunks to common.py

2 years agoinitial checkin
Helmut Grohne [Sun, 3 Feb 2019 11:42:23 +0000 (12:42 +0100)]
initial checkin