Commit Graph

155 Commits

Author SHA1 Message Date
Bobby Wibowo
0ebefe083a
refactor: removed clamscan passthrough support
unfortunately it simply was not reliable enough

and maintaining it is simply adding more complexity to the codes

moreover it was only possible to passthrough regular non-chunked uploads
2022-08-02 16:19:57 +07:00
Bobby Wibowo
285e79c5a7
feat: configurable uploads/albums/users per page
please check sample.config.js for new options

if missing from config, defaults to 25 per page (old defaults)
2022-07-31 15:51:32 +07:00
Bobby
4591b8bb42
refactor: generateUniqueToken -> getUniqueToken
this now matches lifecycle with similar functions in upload and album
controllers

also added a new util function .mask() for basic string masking
2022-07-30 08:37:57 +07:00
Bobby Wibowo
fae28f9aa2
feat: deprecate uploads.cacheFileIdentifiers conf
maintaining it is an unnecessary complexity
it's a feature that doesn't scale too well anyways

also renamed "queryDbForFileCollisions" to
"queryDatabaseForIdentifierMatch"
and updated config description accordingly

this should also now properly free the internal onHold Set
2022-07-29 09:14:55 +07:00
Bobby Wibowo
235a1c56e1
refactor: move devmode flag to utils 2022-07-25 07:39:35 +07:00
Bobby Wibowo
1b109e0dc0
feat: req content-type json -> application/json 2022-07-25 06:12:55 +07:00
Bobby
776ab8ab37
feat: assert request content-type in post apis 2022-07-22 08:40:40 +07:00
Bobby Wibowo
ac63f8b76d
perf: no try-catch block on cloudflare cache purge 2022-07-14 14:41:55 +07:00
Bobby Wibowo
38d86779ae
refactor: HUGE REFACTOR for hyper-express 2022-07-10 19:46:25 +07:00
Bobby Wibowo
3a415165b4
feat: utilsController pass errors 2022-07-10 14:15:14 +07:00
Bobby
2ca2fef301
feat: use SimpleDataStore for album pages cache
this should have better lifecycle and use less memory over time, since
we can define max items in cache
at the moment hard-coded to 10 cached pages (inclusive of nojs version
if ever generated)
2022-07-06 17:51:34 +07:00
Bobby Wibowo
8a1ff434d9
feat: in-memory caching of content-disposition
please read config.sample.js
ignore if not serving files with node or not having the option turned on
2022-07-03 10:35:36 +07:00
Bobby Wibowo
ab96bd5d99
feat: queue cloudflare purge cache with fastq 2022-06-29 17:52:16 +07:00
Bobby Wibowo
9d38c431dc
fix: statistics failing to respond on errors 2022-06-29 14:29:47 +07:00
Bobby Wibowo
9d4c08e460
feat: some env vars support 2022-06-22 13:27:38 +07:00
Bobby
8799189a04
feat: initiate db connection only once
db is now initiated via utilsController, and gets re-used by main
lolisafe.js script and the other controllers

this should now make sure we only have 1 active db connection pool ever

this is a pretty big refactor due to how we used to reference db,
but it should functionally be identical
2022-06-04 04:21:56 +07:00
Bobby
2d147e748b
feat: usergroup-based file retention periods
this supersedes the old temporaryUploadAges, while maintaining full
backwards-compatibility.

please consult config.sample.js if you want to start using this
2022-05-07 02:17:31 +07:00
Bobby
4ecec788d0
feat: console logs of temp uploads check ups 2022-05-07 02:01:33 +07:00
Bobby
395361d5fc
fix: count of album zips generated in stats
change logic to list physical files instead, since the zipGeneratedAt
attribute may still exist despite site owners having already done
physical clean-ups
2022-05-05 14:54:21 +07:00
Bobby
7dd4f50e9e
feat: allow markdown in album description 2022-05-05 12:55:21 +07:00
Bobby
86c26cb50c
feat: some bypass support to passthrough scanning
only usergroup and file extension bypass

real file size can't be determined before passthrough scan,
so there's no bypass by max file size
please read the comments in sample config file

refactored utils.clamscan into utils.scan
2022-04-23 04:44:01 +07:00
Bobby
1c0fd26496
refactor: pass less data into user object 2022-04-16 21:36:34 +07:00
Bobby
db254c602b
feat: experimental clamscan passthrough support
when enabled, passthrough scanning will be used for non-chunked uploads

upload processing will be significantly faster if scanning is required
2022-04-15 16:41:05 +07:00
Bobby
6788dc2094
fix: align with clamscan v2 2022-04-15 15:36:50 +07:00
Bobby
ffc82f6a2a
feat: logger.debug() outputs only on dev mode 2022-04-15 15:30:32 +07:00
Bobby
e31af2d267
fix: round down system uptime
invalidate system info cache quicker (1s to 0.5s)
2021-09-08 00:46:08 +07:00
Bobby Wibowo
ae6d7936c7
feat: blacklist extensions for strip tags
GIFs are known to not work without custom globally-installed libvips
with ImageMagick or GraphicsMagick support.
https://sharp.pixelplumbing.com/api-output#gif
https://sharp.pixelplumbing.com/install#custom-libvips

It's highly recommended to update your config following the changes to
the sample config file.

This also addressed a bug where images would still get recorded to DB
despite them not existing physically due to strip tags errors.
2021-02-01 06:13:37 +07:00
Bobby Wibowo
5a74776978
fix: thumbnails not generated with mixed case
e.g. .jpG, .JPG, .PnG, etc.

Also simplified some codes.
2021-02-01 05:23:53 +07:00
Bobby Wibowo
c921db7ee2
docs: better extension names parsing 2021-01-29 23:17:56 +07:00
Bobby Wibowo
631d14d887
feat: better extension names parsing
It will now preserve upper/lower/mixed case.

It will now properly preserve all tarballs (#370).
2021-01-29 23:15:24 +07:00
Bobby Wibowo
a752bb89e8
fix: added failsafe to available disk stats 2021-01-29 22:44:58 +07:00
Bobby Wibowo
968494bb37
fix: systeminformation v5 breaking changes
Made the codes for stats generation a bit more readable.

Usage percentage for file systems will now properly reflect "non-root"
usage percentage in ext2/3/4 file systems.
2021-01-27 23:50:45 +07:00
Bobby Wibowo
81cf940160
feat: added audios count to uploads stats
As always, this uses hardcoded whitelisted audio extensions (the ones
that are also internally used for "is:audio" filter).
2021-01-08 11:22:31 +07:00
Bobby Wibowo
991d743ef0
feat: add "files in albums" to albums stats 2021-01-08 11:18:50 +07:00
Bobby Wibowo
defa3f2a8c
perf: skip checking zips on disk for album stats
Closes #353.
2021-01-08 11:03:26 +07:00
Bobby Wibowo
c1562e11f4
refactor: ClientError on utilsController
Improved utils.authorize().

Various fixes to thumbnailer and tags stripper.
2021-01-08 10:48:08 +07:00
Bobby Wibowo
1142b64e3c
refactor: no throw literal in utilsController 2021-01-08 09:56:01 +07:00
Bobby Wibowo
0a19b025a0
removed utils.parallelLimit
well, that was a pointless endeavor.
i wasn't thinking clearly.

also updated all scripts that previously used them, to use a combo of
for-loop and setInterval to print progress.
2020-12-27 19:44:10 +07:00
Bobby Wibowo
18db55ba24
Added scripts/rebuild-hashes.js
Closes #345
2020-12-27 18:16:01 +07:00
Bobby Wibowo
d7b11e7e56
try cf purge up to 3 times
with preset delay in between (60 secs if rate limited, 5 secs for
unexpected errors)

an alternative to global queue mechanism in #342
not perfect, but easier to implement

closes #342
2020-12-26 16:52:07 +07:00
Bobby Wibowo
771a3f65d2
query amount of temporary uploads for statistics 2020-12-25 21:18:45 +07:00
Bobby Wibowo
346d9864f3
improved codes for statistics
much more expandable, and should be easier to understand overall.

make more statistics operations run concurrently to speed them up.

make linuxDiskStats config key obsolete by using systeminformation
package to also query for any mounted file systems.
2020-12-25 21:06:21 +07:00
Bobby Wibowo
9c7241d145
Allow filtering audio files with is:audio
For now only support FLAC, MP3, WAV and WMA
More extensions will come at a later date
2020-11-03 22:51:29 +07:00
Bobby Wibowo
db2897fbe6
statistics: always refetch ClamAV version
also better display when not using ClamAV
2020-11-02 18:23:23 +07:00
Bobby Wibowo
afdbc7e9a1
Added ClamAV version to Statistics 2020-11-01 07:10:43 +07:00
Bobby Wibowo
88f852584c
!! REPLACED ClamAV BACKEND: clamdjs -> clamscan !!
Update your config file!
2020-11-01 06:35:56 +07:00
Bobby Wibowo
47dd512910
Removed custom ESLint curly rule
Sigh, why did you do this, past me..?

Also fixed "Delete uploads by names".
2020-10-31 01:12:09 +07:00
Bobby Wibowo
22e6c2c3c6
Updated
Replaced all Array.concat() with spread operator and/or push().
Fixed some faulty search logics when using exclusion filters.
Fixed capitalization of mac font in _variables.scss. Stylelint somehow
fricked up again in my previous commit.
2020-10-11 17:32:22 +07:00
Bobby Wibowo
c113184385
Improved disk usage entry in statistics
Now will calculate usage as (total - avail).
In Linux, ext filesystems by default reserves 5% of the space to be
usable by root, making them essentially already "used" space.
Originally we didn't take that into account.

Disk usage percentage will now round down: Math.round() -> Math.floor().
The general behavior in other tools such as "df".
2020-09-08 19:04:12 +07:00
Bobby Wibowo
65847232c4
Added "time taken" info for Statistics menu 2020-08-24 08:08:21 +07:00