Mirror/dhtcrawler2
1
0
Fork 0
mirror of https://github.com/btdig/dhtcrawler2.git synced 2025-01-19 12:41:36 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
67 Commits 2 Branches 0 Tags 3.4 MiB
7b1a435a43
Commit Graph

67 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Kevin Lynx
7b1a435a43 fix some sphinx related bugs, now it can be used to build sphinx index, still in experiment stage, add `giza' library to query sphinx in http_fontend 2013-07-30 22:14:28 +08:00
Kevin Lynx
7ab79b5d2e variable name change 2013-07-29 23:26:12 +08:00
Kevin Lynx
e5011ab75a fix sphinx doc creation failed 2013-07-29 23:14:41 +08:00
Kevin Lynx
f242d4e44f add sphinx support, in expirment status right now 2013-07-29 23:03:39 +08:00
Kevin Lynx
b961dc9c46 add LICENSE.txt, lincensed by MIT 2013-07-24 20:13:12 +08:00
Kevin Lynx
ec456de63d fix daterange issure which not only record today torrents, not it only show the today inserted torrents 2013-07-23 22:15:08 +08:00
Kevin Lynx
4dc05bf2cc adjust http stats display 2013-07-23 21:45:31 +08:00
Kevin Lynx
cb914fe609 system stats adjust, add more stats to http front-end 2013-07-23 21:40:17 +08:00
Kevin Lynx
2a9f99940a add a new force to string log func, add log to httpd, it can log unicode characters to logfiles 2013-07-22 22:59:10 +08:00
Kevin Lynx
3b0e5701c8 complete all http uri to json api 2013-07-22 21:24:56 +08:00
Kevin Lynx
980c6cad57 add query stats for new hash_writer 2013-07-21 22:20:47 +08:00
Kevin Lynx
e46c264056 add `size' function to hash_download_cache, to debug 2013-07-21 21:55:30 +08:00
Kevin Lynx
108a1bfd1b fix hash_download_cache startup bug 2013-07-21 21:29:47 +08:00
Kevin Lynx
dcf0181839 NOTE: rewrite hash_reader, config changed, dht_hash database changed, require to remove existed dht_hash database 2013-07-21 21:18:40 +08:00
Kevin Lynx
75b3d82f4c change default config 2013-07-21 09:23:48 +08:00
Kevin Lynx
060804ae31 fix cache_indexer bug 2013-07-20 19:38:16 +08:00
Kevin Lynx
2658040f3a add simple `get' json api, fix http search space decode 2013-07-20 10:56:41 +08:00
Kevin Lynx
54a30122fa fix hash_date 2013-07-19 21:32:10 +08:00
Kevin Lynx
37ccb19575 change hash_date only record the new inserted torrents 2013-07-19 21:08:32 +08:00
Kevin Lynx
76542be37a config max download task per hash-reader 2013-07-18 22:04:57 +08:00
Kevin Lynx
4882cbf692 fix hash-writer cache-writting issure 2013-07-18 13:59:37 +08:00
Kevin Lynx
cd9ae2ec53 Merge branch 'src' of github.com:kevinlynx/dhtcrawler2 into src 2013-07-18 13:09:12 +08:00
Kevin Lynx
5592b3989b fix hash_reader stop working bug 2013-07-18 12:43:51 +08:00
Kevin Lynx
01451534ee change crawler to cache hashes and merge hashes before inserted into db 2013-07-17 23:32:30 +08:00
Kevin Lynx
1dc4a2c588 fix cache_indexer download error 2013-07-17 17:44:28 +08:00
Kevin Lynx
75841f8237 fix cache_indexer state not saved correctly 2013-07-16 22:48:11 +08:00
Kevin Lynx
91ab9d9321 fix hash_reader message queue increasing memory leak bug, fix cache_indexer not sync totally 2013-07-16 22:25:42 +08:00
Kevin Lynx
eb4f6fa9ea try to fix high cpu usage when no hash and no wait_download 2013-07-15 23:02:41 +08:00
Kevin Lynx
91731f7839 restore `top' cache 2013-07-15 22:15:06 +08:00
Kevin Lynx
a82fa71cf4 to avoid there's no hash and no wait_download, the hash reader may stop working 2013-07-15 22:03:52 +08:00
Kevin Lynx
b5692682e2 integrate cache_index for hash_reader, default is disabled 2013-07-15 21:40:22 +08:00
Kevin Lynx
a0e1be291d add progress displaying for cache_indexer 2013-07-15 13:45:25 +08:00
Kevin Lynx
062130cbd9 add pre-compiled rmmseg windows 64 version 2013-07-15 12:52:43 +08:00
Kevin Lynx
c78e5d2f9c add cache_indexer, not integrated now, see src/cache_indexer/readme.md 2013-07-14 22:57:57 +08:00
Kevin Lynx
539dd9103b hash_reader now use findAndModify to read and delete hash/wait_download 2013-07-14 15:34:52 +08:00
Kevin Lynx
356a1c0a92 fix name_seger tool 2013-07-14 13:57:49 +08:00
Kevin Lynx
d64630950e config text segment method [none, simple, rmmseg], default is simple 2013-07-13 22:26:15 +08:00
Kevin Lynx
753486c16a add rmmseg, with a pre-compiled win32 version 2013-07-13 21:53:03 +08:00
Kevin Lynx
77893f0759 disable numid default, update rebar to support user macro 2013-07-12 10:30:39 +08:00
Kevin Lynx
84ec8fd9dc add torrent download stats in hash reader, to debug something 2013-07-11 22:42:20 +08:00
Kevin Lynx
9d8c60f0da integrate torrent downloader monitor 2013-07-11 22:08:27 +08:00
Kevin Lynx
24b8061577 chang http top query, add a new database to map date to hashes, to support query by date range 2013-07-11 20:44:58 +08:00
Kevin Lynx
fe85e0acf4 add tor_download req monitor, not integrated yet 2013-07-11 17:52:19 +08:00
Kevin Lynx
f776cbbc46 torbuilder(importer) fix badarith bug when there're invalid name torrent 
files
 2013-07-11 09:05:54 +08:00
Kevin Lynx
ae9f2f0c8e change http startup 2013-07-09 22:42:40 +08:00
Kevin Lynx
46338b02c6 rename db_hash_reader to hash_reader, change inc_announce response field 2013-07-09 21:52:47 +08:00
Kevin Lynx
b9130ec759 use safe insert in torrent builder 2013-07-09 17:01:04 +08:00
Kevin Lynx
5d8c198357 fix hash_reader stop working bug 2013-07-09 15:08:01 +08:00
Kevin Lynx
5a1be24b89 update torrent importer 2013-07-08 22:18:16 +08:00
Kevin Lynx
4d31535727 separate embedded css style into page.temp 2013-07-08 12:44:54 +08:00