summaryrefslogtreecommitdiff
AgeCommit message (Collapse)Author
2013-08-16make debian version_compare available in sqlHelmut Grohne
2013-08-16webapp templates: add an anchor for file issuesHelmut Grohne
2013-08-02model comparability as an equivalence relationHelmut Grohne
webapp has had a relation hash_functions, that modeled "comparable functions". Images should not be compares to other files, since it makes no sense to store them as the RGBA stream, that is being hashed. This comparability property resembles an equivalence relation. So the function table gains a column eqclass. Each class is represented by a number and functions are statically assigned to these classes. Now the filtering happens in SQL instead of Python.
2013-08-01support hashing gif imagesHelmut Grohne
* Rename "image_sha512" to "png_sha512". * dedup.image.ImageHash is now a base class for image hashes such as PNGHash and GIFHash. * Enable both hashes in importpkg. * Fix README. * Add new hash combinations to webapp. * Add "gif file not named *.gif" to issues in update_sharing. * Add redirect for "image_sha512" to webapp for backwards compatibility.
2013-07-30templates/binary: space between package and compareHelmut Grohne
2013-07-30templates: wiki.d.o redirects to https nowHelmut Grohne
2013-07-30fix update_sharing to work after functionid mergeHelmut Grohne
2013-07-29importpkg.py: support uncompressed data.tarHelmut Grohne
2013-07-27also move the static directory into the dedup packageHelmut Grohne
2013-07-27move templates to dedup packageHelmut Grohne
They cluttered webapp.py and now vim can give proper highlighting for the templates.
2013-07-26verify package hashes when importing via httpHelmut Grohne
2013-07-26Merge branch functionidHelmut Grohne
Actual savings on the full data set are around 7%. Conflicts: README
2013-07-25display "issues" with files in package viewHelmut Grohne
Currently this is invalid .gz files and png files not named .png.
2013-07-25README: foo.PNG is also a valid png nameHelmut Grohne
2013-07-24readyaml: cache the whole function tableHelmut Grohne
This should reduce the query bandwidth to the rdbms.
2013-07-23webapp: make html for index validHelmut Grohne
2013-07-23README: fix typo in queryHelmut Grohne
2013-07-23webapp: remove unused functionHelmut Grohne
2013-07-23adapt queries in README to new schemaHelmut Grohne
2013-07-23schema: reference hash functions by integer keyHelmut Grohne
This already worked quite well for package.id. On a test data set of 5% size this transformation reduces the database size by about 4%.
2013-07-22schema: extend content_package_indexHelmut Grohne
We can avoid a b-tree sort in the package comparison of the web app, if the package index, also provides a size.
2013-07-15Merge branch 'packageid'Helmut Grohne
2013-07-12importpkg: simplify state logicHelmut Grohne
2013-07-12importpkg: split process_package to process_controlHelmut Grohne
2013-07-10schema: reference package table by integer keyHelmut Grohne
One approach to improve performance is to reduce the database size. A package name takes up 15 bytes in average. A number of a package takes up two bytes. Multiply that difference with the number of references and it should be noticeably. A small test set show a reduction by 10%.
2013-07-10schema.sql: drop unused indexHelmut Grohne
sharing_package_index is a sub-index of sharing_insert_index and therefore unnecessary.
2013-07-03README: explain update_sharing.pyHelmut Grohne
2013-06-23Merge branch yamlimportHelmut Grohne
+ Way faster on multiple cores. + More reliable, cause http connections do not time out when the db blocks. - Way slower on single core with contended io path. No clue why. Still update_sharing.py makes up the bulk of processing time.
2013-06-19webapp: fix hash example link after git uploadHelmut Grohne
The git binary changed and so did its hash. Choosing a more stable example now: The GPL-3.
2013-06-11autoimport: don't fork for readyamlHelmut Grohne
This appears to be a huge performance boost.
2013-06-11autoimport: support processing individual filesHelmut Grohne
This gets back the original functionality of importpkg.py.
2013-06-10split the import phase to a yaml streamHelmut Grohne
importpkg.py now emits a yaml stream instead of updating the database. The acutual updating now happens in readyaml.py. In this process autoimport.py was significantly reworked to import packages in parallel.
2013-05-27dedup.image: img.convert can also raise that crazy stuffHelmut Grohne
2013-05-09webapp: declare html5 and utf-8Helmut Grohne
2013-05-09webapp: enrich comparison page with version infoHelmut Grohne
2013-05-08fix attribution of logoHelmut Grohne
I remembered the wrong name. The logo was made by Sune Vuorela.
2013-05-05webapp: markup error in /source templateHelmut Grohne
2013-05-05webapp: validator complained about <link> with sizesHelmut Grohne
2013-05-05webapp: reference favicon from base.htmlHelmut Grohne
2013-05-05added favicon.icoHelmut Grohne
Authored: Cyril Brulebois
2013-05-02webapp: use jinja's filesizeformatHelmut Grohne
Except it doesn't work, so replace it with our version. At least we might be able to drop this code in a future update.
2013-05-02webapp: reduce size of comparison outputHelmut Grohne
Only add rowspan when it carries a meaning.
2013-04-27webapp: add a css class binary-packageHelmut Grohne
2013-04-25webapp: total_size is None if num_files is 0Helmut Grohne
2013-04-25webapp: color filenames when hovering themHelmut Grohne
2013-04-25webapp: turn the <br> after filename into a styleHelmut Grohne
2013-04-25move css to /style.cssHelmut Grohne
2013-04-25webapp: make filenames css styleableHelmut Grohne
2013-04-25webapp: top-align fields in /compare pagesHelmut Grohne
Suggested by Paul Wise.
2013-04-25fix markup in base.htmlHelmut Grohne
generated by cgit v1.2.3 (git 2.39.1) at 2025-05-17 10:57:14 +0000