Commit Graph

500 Commits

Author SHA1 Message Date
Bobby Wibowo
6ba30a23c6
feat: improved chunked uploads lifecycle
added checks when there's an attempt to uploads chunks to same file in
parallel

improved final file size checks
2022-07-25 07:09:28 +07:00
Bobby Wibowo
ee8f1914ca
feat: req content-type for upload api 2022-07-25 06:13:12 +07:00
Bobby Wibowo
1b109e0dc0
feat: req content-type json -> application/json 2022-07-25 06:12:55 +07:00
Bobby
4ca64b141f
refactor: do not await assertRequestType
it's not an async function, lmao
2022-07-22 08:50:26 +07:00
Bobby
951737d7d0
feat: handle upload post api based on request type
it'd previously always try to parse as multipart first
now it'll immediately assume the upload post api is for url uploads if
the request type is json
2022-07-22 08:42:11 +07:00
Bobby
776ab8ab37
feat: assert request content-type in post apis 2022-07-22 08:40:40 +07:00
Bobby Wibowo
aa85d04d34
fix: url uploads failing 2022-07-22 04:20:37 +07:00
Bobby Wibowo
96d276b396
fix: check if req.path_parameters is set 2022-07-22 02:03:59 +07:00
Bobby Wibowo
07d0237031
refactor: res.query -> .query_parameters
direct hyper-express prop get
2022-07-22 02:02:59 +07:00
Bobby Wibowo
25f87b3a49
chore: albumsController.js 2022-07-22 01:44:53 +07:00
Bobby Wibowo
7b9fca0bc3
refactor: req.params -> .path_parameters
direct hyper-express prop get
2022-07-22 01:44:15 +07:00
Bobby Wibowo
76a73b7e83
refactor: ServeStatic.middleware -> .handler
also moved it from middlewares to handlers directory

reasoning is that this class is better suited to handle routes directly
instead of being a global middleware
since IO stat to check if request path matches a physical file in the
disk every single time is not very performant
2022-07-22 01:12:35 +07:00
Bobby Wibowo
30e9227a78
feat: custom pages use ServeLiveDirectory
they now have conditional GET suppor too
2022-07-22 01:09:17 +07:00
Bobby Wibowo
51e12e13c0
refactor: rateLimiter.js 2022-07-22 00:57:57 +07:00
Bobby Wibowo
d6020d81ae
feat: serveStatic with accept-ranges support
for streaming support

and with conditional GET support
2022-07-22 00:01:25 +07:00
Bobby Wibowo
d9fd98f7de
feat: improved serveLiveDirectory
allow disabling etag and lastModified headers if required
2022-07-21 23:56:57 +07:00
Bobby Wibowo
1b4b73b67c
feat: improved errorsController.js
mainly handling generic errors
2022-07-21 23:56:08 +07:00
Bobby Wibowo
c0e91e205c
chore: rateLimiter.js 2022-07-21 23:55:48 +07:00
Bobby Wibowo
97bd8f9e5a
chore: serveLiveDirectory.js 2022-07-21 21:14:56 +07:00
Bobby Wibowo
e7a15ecc47
feat: custom livedirectory middleware
with conditional gets support
2022-07-21 21:13:46 +07:00
Bobby Wibowo
ad22285661
refactor: res.set -> res.header
res.set() is an expressjs-compat function with unnecessary checks for
our use case
2022-07-21 20:28:10 +07:00
Bobby
e6753ab15d
fix: handle connection drop on multiform upload 2022-07-15 01:40:57 +07:00
Bobby
27f3bc3119
perf: don't wrap multipart handler in try-catch
fixed in https://github.com/kartikk221/hyper-express/releases/tag/6.3.0
2022-07-15 01:06:28 +07:00
Bobby Wibowo
f40c9e0287
chore: serveStatic.js 2022-07-14 18:34:25 +07:00
Bobby Wibowo
7f6c29b136
fix: properly clean rejected empty files
should also now properly clean temp files from other unexpected errors
2022-07-14 18:18:39 +07:00
Bobby Wibowo
5ee82ce680
perf: don't wait for unlink promise in fail upload
just let it run in the background and respond to client immediately
2022-07-14 18:17:46 +07:00
Bobby Wibowo
7710e63d70
fix: prevent hashStream.update() after .dispose() 2022-07-14 17:01:59 +07:00
Bobby Wibowo
29b16edc04
perf: improve uploads flow
lessen temporary objects/variables creation,
and refactor some variable names to be more obvious
2022-07-14 16:35:06 +07:00
Bobby Wibowo
ac63f8b76d
perf: no try-catch block on cloudflare cache purge 2022-07-14 14:41:55 +07:00
Bobby Wibowo
c32f18a697
fix: uploads mimetype not properly set 2022-07-12 15:26:53 +07:00
Bobby Wibowo
0f6409132a
feat: busboy limits and multipart errors handler
unfortunately to capture multipart errors, we have to wrap the entire
callback function with try-catch block

but it appears overall processing is still slightly faster than
expressjs + multer
2022-07-12 14:39:16 +07:00
Bobby Wibowo
7f9d05da26
feat: multer -> hyper-express multipartfield
get outta here multer, lmao
2022-07-12 13:07:13 +07:00
Bobby Wibowo
e9736f436c
feat: have res.render return the compiled html 2022-07-12 10:41:58 +07:00
Bobby Wibowo
80d59ff2f0
feat: front-end pages middleware
custom pages may now override any built-in pages on the fly as lolisafe
is running

also added internal persistent cache feature into NunjucksRenderer
front-end pages will now be persistently cached during production
2022-07-12 10:31:59 +07:00
Bobby Wibowo
9852dec3d1
chore: RateLimiter clean ups 2022-07-12 10:30:36 +07:00
Bobby Wibowo
a356ce5ad3
refactor: standardize paths in require()'s 2022-07-12 08:51:22 +07:00
Bobby Wibowo
79631ce624
feat: RateLimiter custom middleware class
this adds new production dependency rate-limiter-flexible

this deprecates old rateLimits option in config

to use the new rate limiters, the new option is named rateLimiters and
rateLimitersWhitelist
please consult config.sample.js

rate limiters will also be now processed before any other middlewares,
as only makes sense
2022-07-12 08:48:09 +07:00
Bobby Wibowo
26ae853362
fix: errorsController
not properly printing errors when headers already sent
2022-07-12 08:05:45 +07:00
Bobby Wibowo
eb4057d10c
chore: don't call next on async middleware
you're not supposed to do that, lmao
2022-07-12 08:04:50 +07:00
Bobby Wibowo
b0913eaf59
refactor: ServeStatic custom middleware (WIP)
currently when enabled will force-close lolisafe

i still need to find a decent backend library to make life easier
2022-07-12 06:29:21 +07:00
Bobby Wibowo
af754d7d71
feat: NunjucksRenderer custom middleware class 2022-07-12 06:24:04 +07:00
Bobby Wibowo
7733967624
refactor: errorsController func names 2022-07-12 06:21:21 +07:00
Bobby Wibowo
38d86779ae
refactor: HUGE REFACTOR for hyper-express 2022-07-10 19:46:25 +07:00
Bobby Wibowo
b89945d693
chore: remove controllers/handlers 2022-07-10 19:43:46 +07:00
Bobby Wibowo
59c023588e
refactor: authController pass errors 2022-07-10 14:24:18 +07:00
Bobby Wibowo
b3a304729f
refactor: uploadController pass errors 2022-07-10 14:20:49 +07:00
Bobby Wibowo
0f47ed76b0
feat: tokenController pass errors 2022-07-10 14:17:59 +07:00
Bobby Wibowo
3a415165b4
feat: utilsController pass errors 2022-07-10 14:15:14 +07:00
Bobby Wibowo
99a7a2a677
refactor: albumsController pass errors
motivation: less try-catch
2022-07-10 14:11:29 +07:00
Bobby Wibowo
d970f1d0a5
feat: errorsController
errors should instead be passed along to express via next(), to let
express call the errorsController
2022-07-10 14:10:59 +07:00
Bobby
2ca2fef301
feat: use SimpleDataStore for album pages cache
this should have better lifecycle and use less memory over time, since
we can define max items in cache
at the moment hard-coded to 10 cached pages (inclusive of nojs version
if ever generated)
2022-07-06 17:51:34 +07:00
Bobby
b9d0f787d7
feat: improved SimpleDataStore.hold() again
this time uses an internal Set()

this should hopefully be the final iteration of this helper class until
a new "strategy" needs to be added
2022-07-06 17:49:13 +07:00
Bobby
17c863f724
feat: SimpleDataStore.hold() no longer use up size
this means internal Map-store may actually contain more keys than limit,
but only if the extraenous keys are held-keys (i.e. has null value)

this expects that you don't manually set keys with null values into the
store however
2022-07-06 17:14:00 +07:00
Bobby
ecb30cd159
feat: SimpleDataStore allow pre-holding key 2022-07-06 16:37:54 +07:00
Bobby Wibowo
d31181b4be
refactor: SimpleDataStore.STRATEGIES 2022-07-03 11:08:00 +07:00
Bobby Wibowo
39aaa25f3f
perf: SimpleDataStore 2022-07-03 11:00:48 +07:00
Bobby Wibowo
8a1ff434d9
feat: in-memory caching of content-disposition
please read config.sample.js
ignore if not serving files with node or not having the option turned on
2022-07-03 10:35:36 +07:00
Bobby Wibowo
1404cf9328
fix: utf-8 filename breaks
https://github.com/expressjs/multer/issues/1104
2022-07-03 10:18:04 +07:00
Bobby Wibowo
ab96bd5d99
feat: queue cloudflare purge cache with fastq 2022-06-29 17:52:16 +07:00
Bobby Wibowo
38e673226f
fix: inconsistent size field in DB !! yarn migrate
we used to store number directly into the string size field, and
during the conversion it seemed to always add ".0" at the final string,
probably because the driver or sqlite3 itself assumes float

please run yarn migrate after pulling this commit
if you skip converting the DB, file duplicates check will fail to
function

and in the future im planning to do size statistics in bigint, which
will also fail if not converted
2022-06-29 14:35:00 +07:00
Bobby Wibowo
9d38c431dc
fix: statistics failing to respond on errors 2022-06-29 14:29:47 +07:00
Bobby Wibowo
b117fa9ad1
feat: expose utils to nunjucks templates
also fix nunjucks templates not adhering to env vars overrides for
domains config options, because they attempted to read config directly
2022-06-29 13:58:09 +07:00
Bobby
b17b24b159
feat: new page /file/:identifier
this will display all information recorded from the specified file, but
only to the users that own them (it requires token)

this page also has a delete file button, allowing us to provide link to
this page for sharex deletion url option

once again, this is only for authenticated users, and will only show
file that the users own, unless said user is a moderator or higher
2022-06-28 12:03:49 +07:00
Bobby
5d1bea39ef
feat: new api /api/upload/get/:identifier
this api only returns file that the user owns (thus token must be set)
2022-06-28 11:57:56 +07:00
Bobby Wibowo
f37313a84c
fix: verify database dir 2022-06-22 14:23:54 +07:00
Bobby Wibowo
409d0c4ffd
refactor: relocate db scripts 2022-06-22 13:53:01 +07:00
Bobby Wibowo
9d4c08e460
feat: some env vars support 2022-06-22 13:27:38 +07:00
Bobby Wibowo
954df13aa2
fix: better null domain fallback logic 2022-06-22 11:22:35 +07:00
Bobby
8799189a04
feat: initiate db connection only once
db is now initiated via utilsController, and gets re-used by main
lolisafe.js script and the other controllers

this should now make sure we only have 1 active db connection pool ever

this is a pretty big refactor due to how we used to reference db,
but it should functionally be identical
2022-06-04 04:21:56 +07:00
Bobby
110aab097b
chore: versioned git urls in comments
future-proofing
2022-05-24 07:02:06 +07:00
Bobby
9cc4832717
feat: same error for invalid user or pass on login
every now and then i got curious of what people add to their forks,
and sometimes i'd be like, "hey, that makes sense!"

i remember that back in the days, i switched to nunjucks because of
someone's fork too
we used to use handlebars, which i've had some misgivings with for
awhile back then

either way all i'm saying is that it's the case this time around with
this commit
thanks for reading

49d12004e8
2022-05-24 06:56:58 +07:00
Bobby
d2b8656800
fix: url uploads stricter error handling
this should properly dispose of unfinished write & hasher streams on
any errors if they have been initiated

also do content-length header check a bit more early
2022-05-08 15:35:28 +07:00
Bobby
ce71a9e8d6
feat: better size check for url uploads
first layer is via sending HEAD request to the url to determine its size
via content-length header
however not all hosts properly set the header, so we ignore it if
it isn't a valid number

next via size option in fetch(), which supposedly limits response body
size during the request itself (?)

lastly via checking actual bytes written to physical file as reported by
fs.createWriteStream()
2022-05-08 15:19:29 +07:00
Bobby
71a6adc3d3
fix: url uploads failing with missing content-type 2022-05-08 15:15:10 +07:00
Bobby
eeb1611b2a
fix: failsafes for chunks timeout 2022-05-08 12:01:18 +07:00
Bobby
f3b7d5e56d
fix: empty string for age header failing 2022-05-08 12:00:27 +07:00
Bobby
2d147e748b
feat: usergroup-based file retention periods
this supersedes the old temporaryUploadAges, while maintaining full
backwards-compatibility.

please consult config.sample.js if you want to start using this
2022-05-07 02:17:31 +07:00
Bobby
4ecec788d0
feat: console logs of temp uploads check ups 2022-05-07 02:01:33 +07:00
Bobby
5d3ef6e566
feat: perms.group() to get user's group name 2022-05-07 01:36:38 +07:00
Bobby
b35f4ae6eb
fix: remove token from local storage if invalid
this required expanding our custom error classes with support for
arbitrary internal api error codes

however it'll only be used for invalid token errors for now (10001)

no plan to assign codes to other existing api errors
at that point it's probably better to redo the whole api infrastructure
2022-05-06 21:58:23 +07:00
Bobby
395361d5fc
fix: count of album zips generated in stats
change logic to list physical files instead, since the zipGeneratedAt
attribute may still exist despite site owners having already done
physical clean-ups
2022-05-05 14:54:21 +07:00
Bobby
ecdeaa8b8a
chore: logger.inspect 2022-05-05 14:33:16 +07:00
Bobby
30b8f0040c
fix: delete album only worked for own albums 2022-05-05 14:30:41 +07:00
Bobby
450bf72e7a
feat: allow mods to actually delete albums 2022-05-05 13:58:54 +07:00
Bobby
39b7d96b8f
feat: static preview of album description markdown
in dashboard's edit album prompt
2022-05-05 13:17:32 +07:00
Bobby
7dd4f50e9e
feat: allow markdown in album description 2022-05-05 12:55:21 +07:00
Bobby
3d94ae599e
feat: debug log for clean files 2022-04-23 05:01:56 +07:00
Bobby
88d32a6dcf
refactor: uploadController.js 2022-04-23 04:48:59 +07:00
Bobby
86c26cb50c
feat: some bypass support to passthrough scanning
only usergroup and file extension bypass

real file size can't be determined before passthrough scan,
so there's no bypass by max file size
please read the comments in sample config file

refactored utils.clamscan into utils.scan
2022-04-23 04:44:01 +07:00
Bobby
9af52e068d
docs: config and uploadController 2022-04-16 23:06:25 +07:00
Bobby
1c0fd26496
refactor: pass less data into user object 2022-04-16 21:36:34 +07:00
Bobby
4b9740cf5c
docs: uploadController.js 2022-04-16 20:44:11 +07:00
Bobby
b5aa8d0758
fix: passthrough scan 2022-04-16 13:33:11 +07:00
Bobby
dd6f225461
fix: missing default vars in multerStorage 2022-04-15 20:31:12 +07:00
Bobby
db254c602b
feat: experimental clamscan passthrough support
when enabled, passthrough scanning will be used for non-chunked uploads

upload processing will be significantly faster if scanning is required
2022-04-15 16:41:05 +07:00
Bobby
6788dc2094
fix: align with clamscan v2 2022-04-15 15:36:50 +07:00
Bobby
ffc82f6a2a
feat: logger.debug() outputs only on dev mode 2022-04-15 15:30:32 +07:00
Bobby
e31af2d267
fix: round down system uptime
invalidate system info cache quicker (1s to 0.5s)
2021-09-08 00:46:08 +07:00
Bobby
b86aa8dc08
fix: private upload custom response config option 2021-06-08 08:01:11 +07:00
Bobby Wibowo
48488667c8
feat: limit upload to specific usergroups 2021-05-22 20:59:00 +07:00