~helmut/debian-dedup.git - Unnamed repository; edit this file 'description' to name the repository.

Age	Commit message (Collapse)	Author
2014-03-08	enable result buffering for postgres	Helmut Grohne

2014-03-08	Merge branch 'master' into sqlalchemy	Helmut Grohne
	In the mean time, the master branch evolved quite a bit and the schema changed again (eqclass added to function table). The main reason for the merge is to resolve the large amounts of conflicts once, so development of the sqlalchemy branch can continue and still benefit from changes in the master branch such as schema compatibility, adapting the indent level in web app due to the use of contextlib.closing which resembles sqlalchemy's "with db.begin() as conn:". Conflicts: autoimport.py dedup/utils.py readyaml.py update_sharing.py webapp.py
2014-02-21	update_sharing: weaken assumptions about db layout	Helmut Grohne
	Hash functions are partitioned into equivalence classes. We are generally only interested in sharing among hash functions with the same equivalence class, but the algorithm would compute any sharing. While the current layout never produces the same hashes for functions in difference equivalence classes (for different output length), that may change in future. Also allow hash functions, that belong to no equivalence class at all (eqclass = NULL) as a means to add additional metadata to content without computing any sharing for it.
2013-09-02	add option -d --database for db path to all scripts	Helmut Grohne

2013-08-02	Merge branch master into sqlalchemy	Helmut Grohne
	This makes the sqlalchemy branch schema-compatible with master again. The biggest change on master was the introduction of the function table. It caused most of the conflicts. Note that webapp had one conflict not detected by git: The selecting of issues in show_package needed sqlalchemy conversion. Conflicts: README update_sharing.py webapp.py
2013-08-01	support hashing gif images	Helmut Grohne
	* Rename "image_sha512" to "png_sha512". * dedup.image.ImageHash is now a base class for image hashes such as PNGHash and GIFHash. * Enable both hashes in importpkg. * Fix README. * Add new hash combinations to webapp. * Add "gif file not named .gif" to issues in update_sharing. Add redirect for "image_sha512" to webapp for backwards compatibility.
2013-07-30	fix update_sharing to work after functionid merge	Helmut Grohne

2013-07-26	Merge branch functionid	Helmut Grohne
	Actual savings on the full data set are around 7%. Conflicts: README
2013-07-25	display "issues" with files in package view	Helmut Grohne
	Currently this is invalid .gz files and png files not named .png.
2013-07-23	schema: reference hash functions by integer key	Helmut Grohne
	This already worked quite well for package.id. On a test data set of 5% size this transformation reduces the database size by about 4%.
2013-07-20	another missing sqlalchemy.text wrapper	Helmut Grohne

2013-07-20	use sqlalchemy.text	Helmut Grohne
	Without using this wrapper the sql statements are not munged by sqlalchemy. Specifically paramstyle is not translated. For sqlite3 this did not matter, because it allows the changed paramstyle, but for postgres it fails without sqlalchemy.text wrappers.
2013-07-17	Merge branch master into sqlalchemy	Helmut Grohne
	This basically pulls the packageid branch into sqlalchemy. The merge was complex, because many sql statements diverged. The merge brings us one step closer to supporting postgres, because an "INSERT OR REPLACE" was removed from readyaml.py in the packageid branch. Conflicts: update_sharing.py webapp.py
2013-07-10	use sqlalchemy paramstyle	Helmut Grohne
	By using the :name syntax inside sql statements, sqlalchemy will replace the contents with whatever paramstyle the underlying dbapi2 module needs. In case of psycopg2 the paramstyle is not qmark for instance.
2013-07-10	schema: reference package table by integer key	Helmut Grohne
	One approach to improve performance is to reduce the database size. A package name takes up 15 bytes in average. A number of a package takes up two bytes. Multiply that difference with the number of references and it should be noticeably. A small test set show a reduction by 10%.
2013-06-23	update_sharing: postgres does not support "INSERT OR IGNORE"	Helmut Grohne

2013-06-23	dedup.utils: add enbale_sqlite_foreign_keys helper	Helmut Grohne
	Makes usage of sqlalchemy easier, cause I can invoke it once and it works for all connections.
2013-06-23	port update_sharing.py to sqlalchemy	Helmut Grohne

2013-04-24	implement the /compare/pkg1/pkg2 page differently	Helmut Grohne
	The original version had two major drawbacks: 1) The SQL query used would cause a btree sort, so the time waiting for the first output was rather long. 2) For packages with many equal files, the output would grow with O(n^2). Thanks to the suggestions by Christine Grohne and Klaus Aehlig. The approach now groups files in package1 by their main hash value (sha512). It also does some work SQL was designed to solve manually now. To speed up page generation a new caching table was added identifying which files have corresponding shared files.
2013-03-09	split content table to a hash table	Helmut Grohne
	In the old content table (package, filename, size) would be the same for multiple hash functions. Now the schema represents that each file has precisely one size, but multiple hashes.
2013-03-07	enable enforcing foreign keys	Helmut Grohne

2013-03-02	update_sharing: wrong database name	Helmut Grohne

2013-03-02	add sharing table	Helmut Grohne
	The sharing table is a cache for the /binary web pages. It essentially contains the numbers presented. This caching table is not automatically populated. It needs to be reconstructed after every (group of) package imports.