Mirror/dhtcrawler2
1
0
Fork 0
mirror of https://github.com/btdig/dhtcrawler2.git synced 2025-01-19 20:51:36 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
52 Commits 2 Branches 0 Tags 3.4 MiB
060804ae31
Commit Graph

52 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Kevin Lynx
060804ae31 fix cache_indexer bug 2013-07-20 19:38:16 +08:00
Kevin Lynx
2658040f3a add simple `get' json api, fix http search space decode 2013-07-20 10:56:41 +08:00
Kevin Lynx
54a30122fa fix hash_date 2013-07-19 21:32:10 +08:00
Kevin Lynx
37ccb19575 change hash_date only record the new inserted torrents 2013-07-19 21:08:32 +08:00
Kevin Lynx
76542be37a config max download task per hash-reader 2013-07-18 22:04:57 +08:00
Kevin Lynx
4882cbf692 fix hash-writer cache-writting issure 2013-07-18 13:59:37 +08:00
Kevin Lynx
cd9ae2ec53 Merge branch 'src' of github.com:kevinlynx/dhtcrawler2 into src 2013-07-18 13:09:12 +08:00
Kevin Lynx
5592b3989b fix hash_reader stop working bug 2013-07-18 12:43:51 +08:00
Kevin Lynx
01451534ee change crawler to cache hashes and merge hashes before inserted into db 2013-07-17 23:32:30 +08:00
Kevin Lynx
1dc4a2c588 fix cache_indexer download error 2013-07-17 17:44:28 +08:00
Kevin Lynx
75841f8237 fix cache_indexer state not saved correctly 2013-07-16 22:48:11 +08:00
Kevin Lynx
91ab9d9321 fix hash_reader message queue increasing memory leak bug, fix cache_indexer not sync totally 2013-07-16 22:25:42 +08:00
Kevin Lynx
eb4f6fa9ea try to fix high cpu usage when no hash and no wait_download 2013-07-15 23:02:41 +08:00
Kevin Lynx
91731f7839 restore `top' cache 2013-07-15 22:15:06 +08:00
Kevin Lynx
a82fa71cf4 to avoid there's no hash and no wait_download, the hash reader may stop working 2013-07-15 22:03:52 +08:00
Kevin Lynx
b5692682e2 integrate cache_index for hash_reader, default is disabled 2013-07-15 21:40:22 +08:00
Kevin Lynx
a0e1be291d add progress displaying for cache_indexer 2013-07-15 13:45:25 +08:00
Kevin Lynx
062130cbd9 add pre-compiled rmmseg windows 64 version 2013-07-15 12:52:43 +08:00
Kevin Lynx
c78e5d2f9c add cache_indexer, not integrated now, see src/cache_indexer/readme.md 2013-07-14 22:57:57 +08:00
Kevin Lynx
539dd9103b hash_reader now use findAndModify to read and delete hash/wait_download 2013-07-14 15:34:52 +08:00
Kevin Lynx
356a1c0a92 fix name_seger tool 2013-07-14 13:57:49 +08:00
Kevin Lynx
d64630950e config text segment method [none, simple, rmmseg], default is simple 2013-07-13 22:26:15 +08:00
Kevin Lynx
753486c16a add rmmseg, with a pre-compiled win32 version 2013-07-13 21:53:03 +08:00
Kevin Lynx
77893f0759 disable numid default, update rebar to support user macro 2013-07-12 10:30:39 +08:00
Kevin Lynx
84ec8fd9dc add torrent download stats in hash reader, to debug something 2013-07-11 22:42:20 +08:00
Kevin Lynx
9d8c60f0da integrate torrent downloader monitor 2013-07-11 22:08:27 +08:00
Kevin Lynx
24b8061577 chang http top query, add a new database to map date to hashes, to support query by date range 2013-07-11 20:44:58 +08:00
Kevin Lynx
fe85e0acf4 add tor_download req monitor, not integrated yet 2013-07-11 17:52:19 +08:00
Kevin Lynx
f776cbbc46 torbuilder(importer) fix badarith bug when there're invalid name torrent 
files
 2013-07-11 09:05:54 +08:00
Kevin Lynx
ae9f2f0c8e change http startup 2013-07-09 22:42:40 +08:00
Kevin Lynx
46338b02c6 rename db_hash_reader to hash_reader, change inc_announce response field 2013-07-09 21:52:47 +08:00
Kevin Lynx
b9130ec759 use safe insert in torrent builder 2013-07-09 17:01:04 +08:00
Kevin Lynx
5d8c198357 fix hash_reader stop working bug 2013-07-09 15:08:01 +08:00
Kevin Lynx
5a1be24b89 update torrent importer 2013-07-08 22:18:16 +08:00
Kevin Lynx
4d31535727 separate embedded css style into page.temp 2013-07-08 12:44:54 +08:00
Kevin Lynx
db93cd73f9 add a torrent database builder tool, to append the local torrents to database 2013-07-07 19:50:58 +08:00
Kevin Lynx
8ee2e54e3d fix hash_reader stop working bug 2013-07-06 21:28:50 +08:00
Kevin Lynx
1a9f38df49 see master branch 2013-07-06 09:42:32 +08:00
Kevin Lynx
8f04efed5d IMPORTANT: add number id to torrent database 2013-07-06 01:18:25 +08:00
Kevin Lynx
2944019ca1 see master branch for logs 2013-07-06 00:18:56 +08:00
Kevin Lynx
5b344c4fe2 add torrent local cache, not tested now 2013-07-03 22:56:06 +08:00
Kevin Lynx
0dcf0854a8 fix http cache bug 2013-07-03 20:08:33 +08:00
Kevin Lynx
bb2cbfcc2d async update cache 2013-07-03 17:53:13 +08:00
Kevin Lynx
530d1ce8c5 update http cache in a process 2013-07-03 17:45:35 +08:00
Kevin Lynx
720a680743 add http cache 2013-07-03 17:20:07 +08:00
Kevin Lynx
bdaf17fec1 update readme 2013-07-02 20:40:55 +08:00
Kevin Lynx
c82f00671a update readme 2013-07-02 20:35:13 +08:00
Kevin Lynx
950a12169d add repl-set scripts 2013-07-02 17:43:12 +08:00
Kevin Lynx
0a04f437cf change http to read data from mongodb slave db in repla set, can still 
read from master
 2013-07-02 13:56:44 +08:00
Kevin Lynx
3505116585 change hash reader waiting time to 1 min 2013-07-02 12:27:53 +08:00