summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2013-07-20use sqlalchemy.textHelmut Grohne
Without using this wrapper the sql statements are not munged by sqlalchemy. Specifically paramstyle is not translated. For sqlite3 this did not matter, because it allows the changed paramstyle, but for postgres it fails without sqlalchemy.text wrappers.
2013-07-17Merge branch master into sqlalchemyHelmut Grohne
This basically pulls the packageid branch into sqlalchemy. The merge was complex, because many sql statements diverged. The merge brings us one step closer to supporting postgres, because an "INSERT OR REPLACE" was removed from readyaml.py in the packageid branch. Conflicts: update_sharing.py webapp.py
2013-07-15Merge branch 'packageid'Helmut Grohne
2013-07-12importpkg: simplify state logicHelmut Grohne
2013-07-12importpkg: split process_package to process_controlHelmut Grohne
2013-07-10use sqlalchemy paramstyleHelmut Grohne
By using the :name syntax inside sql statements, sqlalchemy will replace the contents with whatever paramstyle the underlying dbapi2 module needs. In case of psycopg2 the paramstyle is not qmark for instance.
2013-07-10webapp: fix handling of total_sizeHelmut Grohne
The expression "total_size and 0" masks any positive integer to 0.
2013-07-10schema: reference package table by integer keyHelmut Grohne
One approach to improve performance is to reduce the database size. A package name takes up 15 bytes in average. A number of a package takes up two bytes. Multiply that difference with the number of references and it should be noticeably. A small test set show a reduction by 10%.
2013-07-10schema.sql: drop unused indexHelmut Grohne
sharing_package_index is a sub-index of sharing_insert_index and therefore unnecessary.
2013-07-03README: explain update_sharing.pyHelmut Grohne
2013-06-23update_sharing: postgres does not support "INSERT OR IGNORE"Helmut Grohne
2013-06-23dedup.utils: add enbale_sqlite_foreign_keys helperHelmut Grohne
Makes usage of sqlalchemy easier, cause I can invoke it once and it works for all connections.
2013-06-23Merge master into sqlalchemyHelmut Grohne
This is necessary to avoid severe merge conflicts when converting importpkg.py to sqlalchemy. The actual sql invocation has moved to a different file in master. Conflicts: README (diverged set of dependencies)
2013-06-23port update_sharing.py to sqlalchemyHelmut Grohne
2013-06-23Merge branch yamlimportHelmut Grohne
+ Way faster on multiple cores. + More reliable, cause http connections do not time out when the db blocks. - Way slower on single core with contended io path. No clue why. Still update_sharing.py makes up the bulk of processing time.
2013-06-19webapp: fix hash example link after git uploadHelmut Grohne
The git binary changed and so did its hash. Choosing a more stable example now: The GPL-3.
2013-06-13webapp: use sqlalchemyHelmut Grohne
* Arguably the interface is nicer. * Actually closes connections. => wal files get deleted. * Permits switching from sqlite to anything.
2013-06-11autoimport: don't fork for readyamlHelmut Grohne
This appears to be a huge performance boost.
2013-06-11autoimport: support processing individual filesHelmut Grohne
This gets back the original functionality of importpkg.py.
2013-06-10split the import phase to a yaml streamHelmut Grohne
importpkg.py now emits a yaml stream instead of updating the database. The acutual updating now happens in readyaml.py. In this process autoimport.py was significantly reworked to import packages in parallel.
2013-05-27dedup.image: img.convert can also raise that crazy stuffHelmut Grohne
2013-05-09webapp: declare html5 and utf-8Helmut Grohne
2013-05-09webapp: enrich comparison page with version infoHelmut Grohne
2013-05-08fix attribution of logoHelmut Grohne
I remembered the wrong name. The logo was made by Sune Vuorela.
2013-05-05webapp: markup error in /source templateHelmut Grohne
2013-05-05webapp: validator complained about <link> with sizesHelmut Grohne
2013-05-05webapp: reference favicon from base.htmlHelmut Grohne
2013-05-05added favicon.icoHelmut Grohne
Authored: Cyril Brulebois
2013-05-02webapp: use jinja's filesizeformatHelmut Grohne
Except it doesn't work, so replace it with our version. At least we might be able to drop this code in a future update.
2013-05-02webapp: reduce size of comparison outputHelmut Grohne
Only add rowspan when it carries a meaning.
2013-04-27webapp: add a css class binary-packageHelmut Grohne
2013-04-25webapp: total_size is None if num_files is 0Helmut Grohne
2013-04-25webapp: color filenames when hovering themHelmut Grohne
2013-04-25webapp: turn the <br> after filename into a styleHelmut Grohne
2013-04-25move css to /style.cssHelmut Grohne
2013-04-25webapp: make filenames css styleableHelmut Grohne
2013-04-25webapp: top-align fields in /compare pagesHelmut Grohne
Suggested by Paul Wise.
2013-04-25fix markup in base.htmlHelmut Grohne
2013-04-24implement the /compare/pkg1/pkg2 page differentlyHelmut Grohne
The original version had two major drawbacks: 1) The SQL query used would cause a btree sort, so the time waiting for the first output was rather long. 2) For packages with many equal files, the output would grow with O(n^2). Thanks to the suggestions by Christine Grohne and Klaus Aehlig. The approach now groups files in package1 by their main hash value (sha512). It also does some work SQL was designed to solve manually now. To speed up page generation a new caching table was added identifying which files have corresponding shared files.
2013-04-14webapp: added some useful notesHelmut Grohne
2013-04-13base.html: add link to wiki.debian.orgHelmut Grohne
2013-04-08README: improve query after schemachangeHelmut Grohne
2013-03-26webapp: fix problem from the previous mergeHelmut Grohne
2013-03-26Merge branch schemachangeHelmut Grohne
2013-03-20webapp: report correct sizesHelmut Grohne
2013-03-20webapp: remove broken assertHelmut Grohne
Fails on long inputs.
2013-03-18dedup.image: mask errors from PILHelmut Grohne
2013-03-12dedup.arreader: missing bytes markerHelmut Grohne
2013-03-12move ArReader from importpkg to dedup.arreaderHelmut Grohne
Also document it.
2013-03-10README: update queries to match content table splitHelmut Grohne
generated by cgit v1.2.3 (git 2.39.1) at 2025-05-17 20:58:33 +0000