Mirror/dhtcrawler2
1
0
Fork 0
mirror of https://github.com/btdig/dhtcrawler2.git synced 2025-01-18 12:11:39 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity
91 Commits 2 Branches 0 Tags 3.4 MiB
82f14c458c
Commit Graph

91 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Kevin Lynx
82f14c458c optimize highlight for http 2013-08-24 17:39:09 +08:00
Kevin Lynx
c0b383a7b7 add http search result highlight 2013-08-24 16:42:51 +08:00
Kevin Lynx
2f8842a18d fix stats 1970 bug 2013-08-22 21:04:15 +08:00
Kevin Lynx
53e3036f4f add sphinx config doc 2013-08-14 20:48:35 +08:00
Kevin Lynx
fa0b80908a update readme 2013-08-12 20:58:56 +08:00
Kevin Lynx
4a9b85c973 fix sphinx builder query range bug; improve sphinx builder db query performance, to avoid `getmore' command and the 4M useless response 2013-08-07 21:50:06 +08:00
Kevin Lynx
f887208cd5 log bug 2013-08-06 21:13:24 +08:00
Kevin Lynx
ba64278d12 build date index at startup 2013-08-06 20:50:49 +08:00
Kevin Lynx
648fe7f20d daman! fix crawler initiali id generation bug (not set random seed) 2013-08-05 22:03:54 +08:00
Kevin Lynx
377912982d add some debug log to sphinx_builder, test sphinx_builder when there's no hashes there and got new hashes 2013-08-05 21:31:16 +08:00
Kevin Lynx
7af23fcc49 add log level config for sphinx_builder 2013-08-04 21:41:46 +08:00
Kevin Lynx
db83eecfd5 change sphinx_builder, query from mongodb by `skip' really don't work well, build a date index to query by date range 2013-08-04 21:37:50 +08:00
Kevin Lynx
6f6aac3b35 adjust crawler log directory 2013-08-03 21:39:14 +08:00
Kevin Lynx
f16d25dae7 http ui stuff 2013-08-03 17:20:52 +08:00
Kevin Lynx
60bb12538e http ui adjust 2013-08-03 17:10:42 +08:00
Kevin Lynx
aed757f2a8 modify `giza' library so that i can get sphinx search stats, and because of this, i can add a more detailed page navigation 2013-08-03 16:58:04 +08:00
Kevin Lynx
237d90f81a add a new config search_method', if set to sphinx', hash reader will not create name_array, also add config for httpd, to config the search method 2013-08-03 15:49:29 +08:00
Kevin Lynx
486c354ba0 change sphinx torrent loading using an existing cursor 2013-08-02 22:19:31 +08:00
Kevin Lynx
92826bf848 turn off the damn debug log 2013-08-01 23:20:02 +08:00
Kevin Lynx
79291ab4e9 fix sphinx xml utf8 related issure, filter these unicode control characters, only backup delta file if the operation failed 2013-08-01 23:17:52 +08:00
Kevin Lynx
1d870e2e42 add sphinx search stats 2013-07-31 22:06:18 +08:00
Kevin Lynx
1d27f2416b add a simple page navigation for sphinx_search 2013-07-31 20:56:48 +08:00
Kevin Lynx
e1c905b0a7 fix sphinx_build memory leak bug, caused by mongo_cursor 2013-07-31 12:16:14 +08:00
Kevin Lynx
46c99cabd8 sphinx worker call infinity 2013-07-30 22:43:32 +08:00
Kevin Lynx
7b1a435a43 fix some sphinx related bugs, now it can be used to build sphinx index, still in experiment stage, add `giza' library to query sphinx in http_fontend 2013-07-30 22:14:28 +08:00
Kevin Lynx
7ab79b5d2e variable name change 2013-07-29 23:26:12 +08:00
Kevin Lynx
e5011ab75a fix sphinx doc creation failed 2013-07-29 23:14:41 +08:00
Kevin Lynx
f242d4e44f add sphinx support, in expirment status right now 2013-07-29 23:03:39 +08:00
Kevin Lynx
b961dc9c46 add LICENSE.txt, lincensed by MIT 2013-07-24 20:13:12 +08:00
Kevin Lynx
ec456de63d fix daterange issure which not only record today torrents, not it only show the today inserted torrents 2013-07-23 22:15:08 +08:00
Kevin Lynx
4dc05bf2cc adjust http stats display 2013-07-23 21:45:31 +08:00
Kevin Lynx
cb914fe609 system stats adjust, add more stats to http front-end 2013-07-23 21:40:17 +08:00
Kevin Lynx
2a9f99940a add a new force to string log func, add log to httpd, it can log unicode characters to logfiles 2013-07-22 22:59:10 +08:00
Kevin Lynx
3b0e5701c8 complete all http uri to json api 2013-07-22 21:24:56 +08:00
Kevin Lynx
980c6cad57 add query stats for new hash_writer 2013-07-21 22:20:47 +08:00
Kevin Lynx
e46c264056 add `size' function to hash_download_cache, to debug 2013-07-21 21:55:30 +08:00
Kevin Lynx
108a1bfd1b fix hash_download_cache startup bug 2013-07-21 21:29:47 +08:00
Kevin Lynx
dcf0181839 NOTE: rewrite hash_reader, config changed, dht_hash database changed, require to remove existed dht_hash database 2013-07-21 21:18:40 +08:00
Kevin Lynx
75b3d82f4c change default config 2013-07-21 09:23:48 +08:00
Kevin Lynx
060804ae31 fix cache_indexer bug 2013-07-20 19:38:16 +08:00
Kevin Lynx
2658040f3a add simple `get' json api, fix http search space decode 2013-07-20 10:56:41 +08:00
Kevin Lynx
54a30122fa fix hash_date 2013-07-19 21:32:10 +08:00
Kevin Lynx
37ccb19575 change hash_date only record the new inserted torrents 2013-07-19 21:08:32 +08:00
Kevin Lynx
76542be37a config max download task per hash-reader 2013-07-18 22:04:57 +08:00
Kevin Lynx
4882cbf692 fix hash-writer cache-writting issure 2013-07-18 13:59:37 +08:00
Kevin Lynx
cd9ae2ec53 Merge branch 'src' of github.com:kevinlynx/dhtcrawler2 into src 2013-07-18 13:09:12 +08:00
Kevin Lynx
5592b3989b fix hash_reader stop working bug 2013-07-18 12:43:51 +08:00
Kevin Lynx
01451534ee change crawler to cache hashes and merge hashes before inserted into db 2013-07-17 23:32:30 +08:00
Kevin Lynx
1dc4a2c588 fix cache_indexer download error 2013-07-17 17:44:28 +08:00
Kevin Lynx
75841f8237 fix cache_indexer state not saved correctly 2013-07-16 22:48:11 +08:00