Mirror/dhtcrawler2
1
0
Fork 0
mirror of https://github.com/btdig/dhtcrawler2.git synced 2025-01-18 20:21:37 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
73 Commits 2 Branches 0 Tags 3.4 MiB
92826bf848
Commit Graph

73 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Kevin Lynx
92826bf848 turn off the damn debug log 2013-08-01 23:20:02 +08:00
Kevin Lynx
79291ab4e9 fix sphinx xml utf8 related issure, filter these unicode control characters, only backup delta file if the operation failed 2013-08-01 23:17:52 +08:00
Kevin Lynx
1d870e2e42 add sphinx search stats 2013-07-31 22:06:18 +08:00
Kevin Lynx
1d27f2416b add a simple page navigation for sphinx_search 2013-07-31 20:56:48 +08:00
Kevin Lynx
e1c905b0a7 fix sphinx_build memory leak bug, caused by mongo_cursor 2013-07-31 12:16:14 +08:00
Kevin Lynx
46c99cabd8 sphinx worker call infinity 2013-07-30 22:43:32 +08:00
Kevin Lynx
7b1a435a43 fix some sphinx related bugs, now it can be used to build sphinx index, still in experiment stage, add `giza' library to query sphinx in http_fontend 2013-07-30 22:14:28 +08:00
Kevin Lynx
7ab79b5d2e variable name change 2013-07-29 23:26:12 +08:00
Kevin Lynx
e5011ab75a fix sphinx doc creation failed 2013-07-29 23:14:41 +08:00
Kevin Lynx
f242d4e44f add sphinx support, in expirment status right now 2013-07-29 23:03:39 +08:00
Kevin Lynx
b961dc9c46 add LICENSE.txt, lincensed by MIT 2013-07-24 20:13:12 +08:00
Kevin Lynx
ec456de63d fix daterange issure which not only record today torrents, not it only show the today inserted torrents 2013-07-23 22:15:08 +08:00
Kevin Lynx
4dc05bf2cc adjust http stats display 2013-07-23 21:45:31 +08:00
Kevin Lynx
cb914fe609 system stats adjust, add more stats to http front-end 2013-07-23 21:40:17 +08:00
Kevin Lynx
2a9f99940a add a new force to string log func, add log to httpd, it can log unicode characters to logfiles 2013-07-22 22:59:10 +08:00
Kevin Lynx
3b0e5701c8 complete all http uri to json api 2013-07-22 21:24:56 +08:00
Kevin Lynx
980c6cad57 add query stats for new hash_writer 2013-07-21 22:20:47 +08:00
Kevin Lynx
e46c264056 add `size' function to hash_download_cache, to debug 2013-07-21 21:55:30 +08:00
Kevin Lynx
108a1bfd1b fix hash_download_cache startup bug 2013-07-21 21:29:47 +08:00
Kevin Lynx
dcf0181839 NOTE: rewrite hash_reader, config changed, dht_hash database changed, require to remove existed dht_hash database 2013-07-21 21:18:40 +08:00
Kevin Lynx
75b3d82f4c change default config 2013-07-21 09:23:48 +08:00
Kevin Lynx
060804ae31 fix cache_indexer bug 2013-07-20 19:38:16 +08:00
Kevin Lynx
2658040f3a add simple `get' json api, fix http search space decode 2013-07-20 10:56:41 +08:00
Kevin Lynx
54a30122fa fix hash_date 2013-07-19 21:32:10 +08:00
Kevin Lynx
37ccb19575 change hash_date only record the new inserted torrents 2013-07-19 21:08:32 +08:00
Kevin Lynx
76542be37a config max download task per hash-reader 2013-07-18 22:04:57 +08:00
Kevin Lynx
4882cbf692 fix hash-writer cache-writting issure 2013-07-18 13:59:37 +08:00
Kevin Lynx
cd9ae2ec53 Merge branch 'src' of github.com:kevinlynx/dhtcrawler2 into src 2013-07-18 13:09:12 +08:00
Kevin Lynx
5592b3989b fix hash_reader stop working bug 2013-07-18 12:43:51 +08:00
Kevin Lynx
01451534ee change crawler to cache hashes and merge hashes before inserted into db 2013-07-17 23:32:30 +08:00
Kevin Lynx
1dc4a2c588 fix cache_indexer download error 2013-07-17 17:44:28 +08:00
Kevin Lynx
75841f8237 fix cache_indexer state not saved correctly 2013-07-16 22:48:11 +08:00
Kevin Lynx
91ab9d9321 fix hash_reader message queue increasing memory leak bug, fix cache_indexer not sync totally 2013-07-16 22:25:42 +08:00
Kevin Lynx
eb4f6fa9ea try to fix high cpu usage when no hash and no wait_download 2013-07-15 23:02:41 +08:00
Kevin Lynx
91731f7839 restore `top' cache 2013-07-15 22:15:06 +08:00
Kevin Lynx
a82fa71cf4 to avoid there's no hash and no wait_download, the hash reader may stop working 2013-07-15 22:03:52 +08:00
Kevin Lynx
b5692682e2 integrate cache_index for hash_reader, default is disabled 2013-07-15 21:40:22 +08:00
Kevin Lynx
a0e1be291d add progress displaying for cache_indexer 2013-07-15 13:45:25 +08:00
Kevin Lynx
062130cbd9 add pre-compiled rmmseg windows 64 version 2013-07-15 12:52:43 +08:00
Kevin Lynx
c78e5d2f9c add cache_indexer, not integrated now, see src/cache_indexer/readme.md 2013-07-14 22:57:57 +08:00
Kevin Lynx
539dd9103b hash_reader now use findAndModify to read and delete hash/wait_download 2013-07-14 15:34:52 +08:00
Kevin Lynx
356a1c0a92 fix name_seger tool 2013-07-14 13:57:49 +08:00
Kevin Lynx
d64630950e config text segment method [none, simple, rmmseg], default is simple 2013-07-13 22:26:15 +08:00
Kevin Lynx
753486c16a add rmmseg, with a pre-compiled win32 version 2013-07-13 21:53:03 +08:00
Kevin Lynx
77893f0759 disable numid default, update rebar to support user macro 2013-07-12 10:30:39 +08:00
Kevin Lynx
84ec8fd9dc add torrent download stats in hash reader, to debug something 2013-07-11 22:42:20 +08:00
Kevin Lynx
9d8c60f0da integrate torrent downloader monitor 2013-07-11 22:08:27 +08:00
Kevin Lynx
24b8061577 chang http top query, add a new database to map date to hashes, to support query by date range 2013-07-11 20:44:58 +08:00
Kevin Lynx
fe85e0acf4 add tor_download req monitor, not integrated yet 2013-07-11 17:52:19 +08:00
Kevin Lynx
f776cbbc46 torbuilder(importer) fix badarith bug when there're invalid name torrent 
files
 2013-07-11 09:05:54 +08:00