Viktor Lofgren
fb673de370
(crawler) Change the header 'User-agent' to 'User-Agent'
2025-01-28 15:34:16 +01:00
Viktor Lofgren
eee73ab16c
(crawler) Be more lenient when performing a domain probe
2025-01-28 15:24:30 +01:00
Viktor Lofgren
a2b076f9be
(converter) Add progress tracking for big domains in converter
2025-01-26 18:03:59 +01:00
Viktor Lofgren
c8b0a32c0f
(crawler) Reduce long retention of CrawlDataReference objects and their associated SerializableCrawlDataStreams
2025-01-26 15:40:17 +01:00
Viktor Lofgren
f0d74aa3bb
(converter) Fix close() ordering to prevent converter crash
2025-01-26 14:47:36 +01:00
Viktor Lofgren
74a1f100f4
(converter) Refactor to remove CrawledDomainReader and move its functionality into SerializableCrawlDataStream
2025-01-26 14:46:50 +01:00
Viktor Lofgren
eb049658e4
(converter) Add truncation att the parser step to prevent the converter from spending too much time on excessively large documents
...
Refactor to do this without introducing additional copies
2025-01-26 14:28:53 +01:00
Viktor Lofgren
db138b2a6f
(converter) Add truncation att the parser step to prevent the converter from spending too much time on exessively large documents
2025-01-26 14:25:57 +01:00
Viktor Lofgren
1673fc284c
(converter) Reduce lock contention in converter by separating the processing of full and simple-track domains
2025-01-26 13:21:46 +01:00
Viktor Lofgren
503ea57d5b
(converter) Reduce lock contention in converter by separating the processing of full and simple-track domains
2025-01-26 13:18:14 +01:00
Viktor Lofgren
db99242db2
(converter) Adding some logging around the simple processing track to investigate an issue with the converter stalling
2025-01-26 12:02:00 +01:00
Viktor Lofgren
3d68be64da
(crawler) Add default CT when it's missing for icons
2025-01-22 13:55:47 +01:00
Viktor Lofgren
98a340a0d1
(crawler) Add favicon data to domain state db in its own table
2025-01-22 11:41:20 +01:00
Viktor Lofgren
8862100f7e
(crawler) Improve logging and error handling
2025-01-21 21:44:21 +01:00
Viktor Lofgren
274941f6de
(crawler) Smarter parquet->slop crawl data migration
2025-01-21 21:26:12 +01:00
Viktor Lofgren
abec83582d
Fix refactoring gore
2025-01-21 15:08:04 +01:00
Viktor Lofgren
088310e998
(converter) Improve simple processing performance
...
There was a regression introduced in the recent slop migration changes in the performance of the simple conversion track. This reverts the issue.
2025-01-21 14:13:33 +01:00
Viktor Lofgren
4c74e280d3
(crawler) Fix urlencoding in sitemap fetcher
2025-01-21 13:33:35 +01:00
Viktor Lofgren
5b347e17ac
(crawler) Automatically migrate to slop from parquet when crawling
2025-01-21 13:33:14 +01:00
Viktor Lofgren
55d6ab933f
Merge branch 'master' into slop-crawl-data-spike
2025-01-21 13:32:58 +01:00
Viktor Lofgren
43b74e9706
(crawler) Fix exception handler and resource leak in WarcRecorder
2025-01-20 23:45:28 +01:00
Viktor Lofgren
579a115243
(crawler) Reduce log spam from error handling in new sitemap fetcher
2025-01-20 23:17:13 +01:00
Viktor Lofgren
78a958e2b0
(crawler) Fix broken test that started failing after the search engine moved to a new domain
2025-01-20 18:52:14 +01:00
Viktor Lofgren
4e939389b2
(crawler) New Jsoup based sitemap parser
2025-01-20 14:37:44 +01:00
Viktor Lofgren
e67a9bdb91
(crawler) Migrate away from using OkHttp in the crawler, use Java's HttpClient instead.
2025-01-19 15:07:11 +01:00
Viktor Lofgren
567e4e1237
(crawler) Fast detection and bail-out for crawler traps
...
Improve logging and exclude robots.txt from this logic.
2025-01-18 15:28:54 +01:00
Viktor Lofgren
4342e42722
(crawler) Fast detection and bail-out for crawler traps
...
Nephentes has been doing the rounds in social media, adding an easy detection and mitigation mechanism for this type of trap, as sadly not all webmasters set up their robots.txt correctly. Out of the box crawl limits will also deal with this type of attack, but this fix is faster.
2025-01-17 13:02:57 +01:00
Viktor Lofgren
59e2dd4c26
(specialization) Soften length requirements for wiki-specialized documents (incl. cppreference)
2025-01-07 15:41:30 +01:00
Viktor Lofgren
ca1807caae
(specialization) Add new specialization for cppreference.com
...
Give this reference website some synthetically generated tokens to improve the likelihood of a good match.
2025-01-07 15:41:05 +01:00
Viktor Lofgren
26c20e18ac
(keyword-extraction) Soften constraints on keyword patterns, allowing for longer segmented words
2025-01-07 15:20:50 +01:00
Viktor Lofgren
bae44497fe
(crawler) Add a new system property crawler.maxFetchSize
...
This gives the same upper limit to the live crawler and the big boy crawler, though the live crawler will reject items too large, and the big crawler will truncate at that point.
2024-12-30 15:10:11 +01:00
Viktor Lofgren
0d59202aca
(crawler) Do not remove W/-prefix on weak e-tags
...
The server expects to get them back prefixed, as we received them.
2024-12-27 20:56:42 +01:00
Viktor Lofgren
0ca43f0c9c
(live-crawler) Improve live crawler short-circuit logic
...
We should not wait until we've fetched robots.txt to decide whether we have any data to fetch! This makes the live crawler very slow and leads to unnecessary requests.
2024-12-27 20:54:42 +01:00
Viktor Lofgren
927bc0b63c
(live-crawler) Add Accept-Encoding: gzip to outbound requests
...
This change adds `Accept-Encoding: gzip` to all outbound requests from the live crawler and feed fetcher, and the corresponding decoding logic for the compressed response data.
The change addresses issue #136 , save for making the fetcher's requests conditional.
2024-12-27 03:59:34 +01:00
Viktor Lofgren
d968801dc1
(converter) Drop feed data from SlopDomainRecord
...
Also remove feed extraction from converter. This is the crawler's responsibility now.
2024-12-26 17:57:08 +01:00
Viktor Lofgren
89db69d360
(crawler) Correct feed URLs in domain state db
...
Discovered feed URLs were given a double slash after their domain name in the DB. This will go away in the URL normalizer, so the URLs are still viable, but the commit fixes the issue regardless.
2024-12-26 15:18:31 +01:00
Viktor Lofgren
895cee7004
(crawler) Improved feed discovery, new domain state db per crawlset
...
Feed discover is improved with by probing a few likely endpoints when no feed link tag is provided. To store the feed URLs, a sqlite database is added to each crawlset that stores a simple summary of the crawl job, including any feed URLs that have been discovered.
Solves issue #135
2024-12-26 15:05:52 +01:00
Viktor Lofgren
4bb71b8439
(crawler) Correct content type probing to only run on URLs that are suspected to be binary
2024-12-26 14:26:23 +01:00
Viktor Lofgren
e4a41f7dd1
(crawler) Correct content type probing to only run on URLs that are suspected to be binary
2024-12-26 14:13:17 +01:00
Viktor Lofgren
a557c7ae7f
(live-crawler) Limit concurrent accesses per domain using DomainLocks from main crawler
2024-12-23 23:31:03 +01:00
Viktor Lofgren
47e58a21c6
Refactor documentBody method and ContentType charset handling
...
Updated the `documentBody` method to improve parsing retries and error handling. Refactored `ContentType` charset processing with cleaner logic, removing redundant handling for unsupported charsets. Also, updated the version of the `slop` library in dependency settings.
2024-12-17 17:11:37 +01:00
Viktor Lofgren
3714104976
Add loader for slop data in converter.
...
Also alter CrawledDocument to not require String parsing of the underlying byte[] data. This should reduce the number of large memory allocations quite significantly, hopefully reducing the GC churn a bit.
2024-12-17 15:40:24 +01:00
Viktor Lofgren
f6f036b9b1
Switch to new Slop format for crawl data storage and processing.
...
Replaces Parquet output and processing with the new Slop-based format. Includes data migration functionality, updates to handling and writing of crawl data, and introduces support for SLOP in domain readers and converters.
2024-12-15 19:34:03 +01:00
Viktor Lofgren
b510b7feb8
Spike for storing crawl data in slop instead of parquet
...
This seems to reduce RAM overhead to 100s of MB (from ~2 GB), as well as roughly double the read speeds. On disk size is virtually identical.
2024-12-15 15:49:47 +01:00
Viktor Lofgren
eb2fe18867
(sideload) Add LSH generation for sideloaded StackExchange data
...
Previously, the sideloader did not generate a locality-sensitive hashCode for document details. This caused all documents from the same domain to be considered duplicates by the deduplication logic.
2024-12-13 02:10:52 +01:00
Viktor Lofgren
a7468c8d23
(converter) Ensure paths are created for converter batch writer
2024-12-13 01:35:07 +01:00
Viktor Lofgren
fb2beb1eac
(converter) Fix data-loss bug where the converter writer would remove all but the last batch of processed data
2024-12-13 01:19:30 +01:00
Viktor Lofgren
0fb03e3d62
(export) Add logging to AtagExporter for error handling
2024-12-12 22:54:32 +01:00
Viktor Lofgren
0a53ac68a0
Add specialization for steam store and GOG
2024-12-11 18:32:45 +01:00
Viktor Lofgren
e65d75a0f9
(crawler) Reintroduce content type probing and clean out bad content type data from the existing crawl sets
2024-12-11 17:01:52 +01:00
