d84a2c183f
The crawl spec abstraction was used to upload lists of domains into the system for future crawling. This was fairly clunky, and it was difficult to understand what was going to be crawled. Since a while back, a new domains listing view has been added to the control view that allows direct access to the domains table. This is much preferred and means the operator can directly manage domains without specs. This commit removes the crawl spec abstraction from the code, and changes the GUI to direct to the domains list instead. |
||
|---|---|---|
| .. | ||
| java | ||
| test/nu/marginalia/crawling | ||
| build.gradle | ||
| readme.md | ||
Crawling Models
Contains crawl data models shared by the crawling-process and converting-process.
To ensure backward compatibility with older versions of the data, the serialization is abstracted away from the model classes.
The new way of serializing the data is to use parquet files.
The old way was to use zstd-compressed JSON. The old way is still supported for now, but the new way is preferred as it's not only more succinct, but also significantly faster to read and much more portable. The JSON support will be removed in the future.
Central Classes
Serialization
These serialization classes automatically negotiate the serialization format based on the file extension.
Data is accessed through a SerializableCrawlDataStream, which is a somewhat enhanced Iterator that can be used to read data.
Parquet Serialization
The parquet serialization is done using the CrawledDocumentParquetRecordFileReader and CrawledDocumentParquetRecordFileWriter classes, which read and write parquet files respectively.
The model classes are serialized to parquet using the CrawledDocumentParquetRecord
The record has the following fields:
domain- The domain of the documenturl- The URL of the documentip- The IP address of the documentcookies- Whether the document has cookieshttpStatus- The HTTP status code of the documenttimestamp- The timestamp of the documentcontentType- The content type of the documentbody- The body of the documentetagHeader- The ETag header of the documentlastModifiedHeader- The Last-Modified header of the document
The easiest way to interact with parquet files is to use DuckDB, which lets you run SQL queries on parquet files (and almost anything else).
e.g.
$ select httpStatus, count(*) as cnt
from 'my-file.parquet'
group by httpStatus;
┌────────────┬───────┐
│ httpStatus │ cnt │
│ int32 │ int64 │
├────────────┼───────┤
│ 200 │ 43 │
│ 304 │ 4 │
│ 500 │ 1 │
└────────────┴───────┘