~helmut/crossqa.git
2 weeks agobuild.py: fix build/native confusion in function name master
Helmut Grohne [Sun, 11 Oct 2020 17:39:45 +0000 (19:39 +0200)]
build.py: fix build/native confusion in function name

2 weeks agowebapp.py: improve parameter validation for /schedule
Helmut Grohne [Sat, 10 Oct 2020 07:32:27 +0000 (09:32 +0200)]
webapp.py: improve parameter validation for /schedule

Allow scheduling any_arm64. Disallow scheduling garbage_any.

2 weeks agodepcheck.py: delete unused functions
Helmut Grohne [Fri, 9 Oct 2020 20:48:39 +0000 (22:48 +0200)]
depcheck.py: delete unused functions

2 weeks agoadd build architecture to schema of depcheck table
Helmut Grohne [Thu, 8 Oct 2020 20:13:57 +0000 (22:13 +0200)]
add build architecture to schema of depcheck table

2 weeks agowebapp.py: really fix schedule response
Helmut Grohne [Thu, 8 Oct 2020 19:54:15 +0000 (21:54 +0200)]
webapp.py: really fix schedule response

Also handle invalid form values with a BadRequest.

Fixes: bab219106d1b ("webapp.py: fix schedule response")
Fixes: a52d12012b1b ("webapp: add concept of "build architecture" to frontend")

2 weeks agoadd build architecture to schema of depstate table
Helmut Grohne [Thu, 8 Oct 2020 15:16:06 +0000 (17:16 +0200)]
add build architecture to schema of depstate table

2 weeks agowebapp.py: fix schedule response
Helmut Grohne [Thu, 8 Oct 2020 14:45:45 +0000 (16:45 +0200)]
webapp.py: fix schedule response

Fixes: a52d12012b1b ("webapp: add concept of "build architecture" to frontend")

2 weeks agobuild.py: add option --buildarch
Helmut Grohne [Thu, 8 Oct 2020 11:40:46 +0000 (13:40 +0200)]
build.py: add option --buildarch

This allows selecting the architecture of the build server defaulting to
the local native architecture.

2 weeks agoadd build architecture to schema of buildrequests table
Helmut Grohne [Thu, 8 Oct 2020 11:11:15 +0000 (13:11 +0200)]
add build architecture to schema of buildrequests table

2 weeks agobuild.py: fix up sql statement after refactoring
Helmut Grohne [Thu, 8 Oct 2020 05:17:57 +0000 (07:17 +0200)]
build.py: fix up sql statement after refactoring

Fixes: d8a9b7aa8c98 ("add build architecture to schema of building table")

2 weeks agobuild.py: fix up sql statement after refactoring
Helmut Grohne [Wed, 7 Oct 2020 20:04:28 +0000 (22:04 +0200)]
build.py: fix up sql statement after refactoring

Fixes: 7b5da3c5b65e ("add build architecture to schema of builds table")

2 weeks agoadd build architecture to schema of building table
Helmut Grohne [Wed, 7 Oct 2020 16:02:42 +0000 (18:02 +0200)]
add build architecture to schema of building table

2 weeks agoadd build architecture to schema of builds table
Helmut Grohne [Wed, 7 Oct 2020 14:48:12 +0000 (16:48 +0200)]
add build architecture to schema of builds table

2 weeks agowebapp: add concept of "build architecture" to frontend
Helmut Grohne [Wed, 7 Oct 2020 14:41:14 +0000 (16:41 +0200)]
webapp: add concept of "build architecture" to frontend

The frontend now pretends that we had a concept of a "build
architecture" in its visual appearance. Whenever dealing with, hard code
"amd64" though. That allows refactoring the underlying database without
changing how the frontend looks.

3 weeks agodepcheck.py: reduce use of BUILD_ARCH
Helmut Grohne [Mon, 5 Oct 2020 21:36:02 +0000 (23:36 +0200)]
depcheck.py: reduce use of BUILD_ARCH

3 weeks agosupport concurrent builds
Helmut Grohne [Mon, 5 Oct 2020 19:55:42 +0000 (21:55 +0200)]
support concurrent builds

Add new table "building" for recording builds in progress. When a
concurrent build is issued, it will avoid building a source package that
is presently built. It will also avoid building for an architecture that
is presently built for. The latter condition helps reduce the number of
satisfiability issues.

3 weeks agobuild.sh: use sbuild/lintian integration
Helmut Grohne [Mon, 5 Oct 2020 17:45:41 +0000 (19:45 +0200)]
build.sh: use sbuild/lintian integration

Since upgrading from stretch to buster, the lintian integration actually
works. Rely on it.

3 weeks agodrop mips as a release architecture
Helmut Grohne [Sat, 3 Oct 2020 22:07:05 +0000 (00:07 +0200)]
drop mips as a release architecture

3 weeks agodepcheck: add python3.9-minimal to bad_foreign_packages
Helmut Grohne [Sat, 3 Oct 2020 22:06:51 +0000 (00:06 +0200)]
depcheck: add python3.9-minimal to bad_foreign_packages

8 months agodepcheck: add python3.8-minimal to bad_foreign_packages
Helmut Grohne [Sun, 16 Feb 2020 08:50:18 +0000 (09:50 +0100)]
depcheck: add python3.8-minimal to bad_foreign_packages

9 months agofetch and display bugs
Helmut Grohne [Mon, 13 Jan 2020 19:07:00 +0000 (20:07 +0100)]
fetch and display bugs

Retrieve bugs from udd-mirror by tags/usertags and display relevant bug
numbers in the webapp.

9 months agoschema.sql: index builds on source
Helmut Grohne [Mon, 13 Jan 2020 19:00:39 +0000 (20:00 +0100)]
schema.sql: index builds on source

The per-source page rendered by webapp.py incurs a lot of query that can
be sped up using this index.

11 months agodepcheck: ignore givebacks
Helmut Grohne [Tue, 5 Nov 2019 05:04:21 +0000 (06:04 +0100)]
depcheck: ignore givebacks

When an architecture encounters an installation error, the chance that
dose and apt disagree is very high. Thus skip the relevant architecture
until the next mirror push entirely.

13 months agodepcheck: don't litter temporary files on error
Helmut Grohne [Sun, 8 Sep 2019 09:06:51 +0000 (11:06 +0200)]
depcheck: don't litter temporary files on error

When a multiprocessing.Pool is garbage collected or __exit__ed, it
calls the terminate method. In that case, temporary files are not
cleaned up. This happens for example, when an exception is raised in one
of the workers. Thus we close and join the pool explicitly.

14 months agobuild.py: handle errors from ssh in a better way
Helmut Grohne [Thu, 1 Aug 2019 08:12:37 +0000 (10:12 +0200)]
build.py: handle errors from ssh in a better way

Previously, it would retry immediately and produce empty log files
marked as failures. When ssh fails, we no longer record a build nor log.
We also wait a little before retrying.

17 months agowebapp: use a package-specific debcheck link
Helmut Grohne [Fri, 10 May 2019 15:10:08 +0000 (17:10 +0200)]
webapp: use a package-specific debcheck link

Reported-by: Paul Wise <pabs@debian.org>
17 months agowebapp: add an index listing recently failed builds
Helmut Grohne [Thu, 9 May 2019 19:54:42 +0000 (21:54 +0200)]
webapp: add an index listing recently failed builds

19 months agowebapp.py: sort the architecture drop down
Helmut Grohne [Tue, 26 Mar 2019 20:35:12 +0000 (21:35 +0100)]
webapp.py: sort the architecture drop down

19 months agotable buildreqeusts should have an explicit id
Helmut Grohne [Sun, 24 Mar 2019 06:25:38 +0000 (07:25 +0100)]
table buildreqeusts should have an explicit id

We rely on the id being constant. However, sqlite does not guarantee
that the implicit rowid stays constant across vacuum. Making the rowid
explicit fixes that.

19 months agobuild.py: add back random ordering
Helmut Grohne [Sat, 23 Mar 2019 12:22:47 +0000 (13:22 +0100)]
build.py: add back random ordering

Otherwise "any" builds will always schedule for arm64.

19 months agobuild.py: prefer older buildrequests
Helmut Grohne [Sat, 23 Mar 2019 11:48:09 +0000 (12:48 +0100)]
build.py: prefer older buildrequests

19 months agoallow scheduling builds via webapp
Helmut Grohne [Sat, 23 Mar 2019 09:06:16 +0000 (10:06 +0100)]
allow scheduling builds via webapp

19 months agowebapp.py: fix html structure
Helmut Grohne [Tue, 19 Mar 2019 20:09:46 +0000 (21:09 +0100)]
webapp.py: fix html structure

<footer> must be inside <body>

19 months agowebapp.py: add a footer pointing to the source
Helmut Grohne [Sun, 17 Mar 2019 21:01:52 +0000 (22:01 +0100)]
webapp.py: add a footer pointing to the source

19 months agolicense the beast as gpl-2+
Helmut Grohne [Sun, 17 Mar 2019 21:00:59 +0000 (22:00 +0100)]
license the beast as gpl-2+

19 months agowebapp.py: make it prettier
Helmut Grohne [Wed, 13 Mar 2019 20:26:05 +0000 (21:26 +0100)]
webapp.py: make it prettier

Thanks to Paul Wise, Guillem Jover and itd@oftc for their feedback.

19 months agowebapp: avoid raising an unhandled FileNotFoundError
Helmut Grohne [Tue, 12 Mar 2019 05:17:02 +0000 (06:17 +0100)]
webapp: avoid raising an unhandled FileNotFoundError

19 months agowebapp: parse rounded timestamps
Helmut Grohne [Wed, 6 Mar 2019 07:18:38 +0000 (08:18 +0100)]
webapp: parse rounded timestamps

19 months agowebapp: get rid of the annoying sqlalchemy warning
Helmut Grohne [Tue, 5 Mar 2019 20:45:45 +0000 (21:45 +0100)]
webapp: get rid of the annoying sqlalchemy warning

The feature is irrelevant here as the ORM isn't being used.

19 months agowebapp: jinja 2.8 doesn't support dotted assignments
Helmut Grohne [Tue, 5 Mar 2019 20:27:03 +0000 (21:27 +0100)]
webapp: jinja 2.8 doesn't support dotted assignments

19 months agowebapp: only show the bootstrap.d.n link when there is a problem
Helmut Grohne [Tue, 5 Mar 2019 20:18:34 +0000 (21:18 +0100)]
webapp: only show the bootstrap.d.n link when there is a problem

Thanks to Johannes Schauer for the suggestion.

19 months agowebapp: remove the architecture compaction
Helmut Grohne [Tue, 5 Mar 2019 19:48:39 +0000 (20:48 +0100)]
webapp: remove the architecture compaction

It confused Guillem Jover at least.

19 months agoadd initial web presentation
Helmut Grohne [Tue, 5 Mar 2019 19:15:19 +0000 (20:15 +0100)]
add initial web presentation

19 months agoschema.sql: improve unique index
Helmut Grohne [Tue, 5 Mar 2019 19:01:31 +0000 (20:01 +0100)]
schema.sql: improve unique index

Most of the time, we'll query depstate by source. Thus it should come
first in the uniqueness constraint.

19 months agodepcheck.py: abort early when nothing needs checking
Helmut Grohne [Tue, 5 Mar 2019 18:58:07 +0000 (19:58 +0100)]
depcheck.py: abort early when nothing needs checking

19 months agoreduce timestamps to 1s resolution
Helmut Grohne [Tue, 5 Mar 2019 18:47:32 +0000 (19:47 +0100)]
reduce timestamps to 1s resolution

We don't need more resolution. It is a distraction and removing it saves
around 1% of db size.

19 months agodepcheck: support parallel operation
Helmut Grohne [Sat, 2 Mar 2019 15:10:13 +0000 (16:10 +0100)]
depcheck: support parallel operation

19 months agodepcheck: remove the 6h condition
Helmut Grohne [Sat, 2 Mar 2019 14:41:15 +0000 (15:41 +0100)]
depcheck: remove the 6h condition

If the release timestamp doesn't change, we should assume that the
content didn't either.

19 months agodepcheck.py: move the filter into sql
Helmut Grohne [Sat, 2 Mar 2019 12:55:43 +0000 (13:55 +0100)]
depcheck.py: move the filter into sql

19 months agodepcheck: elide redundant element from tuple
Helmut Grohne [Sat, 2 Mar 2019 09:36:45 +0000 (10:36 +0100)]
depcheck: elide redundant element from tuple

19 months agobuild.py: make the build server configurable
Helmut Grohne [Sat, 2 Mar 2019 07:27:12 +0000 (08:27 +0100)]
build.py: make the build server configurable

20 months agodepcheck.py: make the mirror configurable
Helmut Grohne [Tue, 26 Feb 2019 06:48:03 +0000 (07:48 +0100)]
depcheck.py: make the mirror configurable

20 months agoadd a basic .gitignore
Helmut Grohne [Sat, 23 Feb 2019 18:51:05 +0000 (19:51 +0100)]
add a basic .gitignore

20 months agoadd type hints to common.py
Helmut Grohne [Sat, 23 Feb 2019 18:50:47 +0000 (19:50 +0100)]
add type hints to common.py

20 months agobuild.sh: add a lintian tag
Helmut Grohne [Mon, 11 Feb 2019 04:34:23 +0000 (05:34 +0100)]
build.sh: add a lintian tag

The triplet-dir-and-architecture-mismatch detects installing files to
${DEB_BUILD_MULTIARCH}. It is very relevant to cross building and rarely
happens elsewhere.

20 months agobuild.py: handle the case of no satisfiable packages
Helmut Grohne [Sun, 10 Feb 2019 13:49:23 +0000 (14:49 +0100)]
build.py: handle the case of no satisfiable packages

20 months agomake depcheck.py compatible with stretch
Helmut Grohne [Fri, 8 Feb 2019 18:21:27 +0000 (19:21 +0100)]
make depcheck.py compatible with stretch

20 months agomove yield_chunks to common.py
Helmut Grohne [Tue, 5 Feb 2019 16:19:49 +0000 (17:19 +0100)]
move yield_chunks to common.py

20 months agoinitial checkin
Helmut Grohne [Sun, 3 Feb 2019 11:42:23 +0000 (12:42 +0100)]
initial checkin