Commit Graph

289 Commits

Author SHA1 Message Date
Bobby Wibowo
46c84db64f
Merge branch 'patch-3' of https://github.com/camjac251/lolisafe into camjac251-patch-3 2020-06-06 19:34:46 +07:00
Bobby Wibowo
28c64d531a
Potentially much better video thumbnails
Notable changes, use fluent-ffmpeg's functions instead of passing raw
ffmpeg switches with inputOptions() or outputOptions().

It seems fluent-ffmpeg may be doing some janks with outputOptions().
I can't get select="eq(pict_type\\,I)" to work with it, but it works
with videoFilters(). Weird stuff.
2020-06-06 19:28:42 +07:00
camjac251
cf2593465a Use top variable over config 2020-06-06 02:39:53 -05:00
Bobby Wibowo
8b4b0e79c5
Improved albums public page cache and more
Removed its dependency towards albums' editedAt property.
Editing album's metas (name, description, etc) will no longer update
its editedAt property.
Instead it will now ONLY be updated when adding/removing files to/from
it. Just like how it was meant to be, which was to be used to check
whether it's necessary to re-generate their downloadable ZIPs.

Albums public page cache will still be properly invalidated when
adding/removing files to/from it, as well as after editing their metas.

Added views/album-notice.njk to be used to render okay-ish notice when
an album's public page is still being generated.
I was originally thinking of using it for disabled albums as well, but
I refrained from it to reduce the possibility of disabled album IDs from
being easily scanned (as it just returns 404 now).

Removed invalidatedAt property from stats cache. Instead their caches
will immediately be nullified as they should (thus frees up memory
slightly as well).

Stats cache for albums will now only be cleared when truly necessary.
As in, adding/removing files to/from albums will no longer clear them.

Updated Nunjucks files to properly use h1, h2, h3 tags in actual
hierarchical orders.
Elements that don't need to use hX tags will now use P instead.
Nothing changes visually, only structurally.

Fixed some elements in Nunjucks using single quotes instead of
double quotes. They'd have worked the same, but consistency.

Added h1 title in FAQ page.

Make text for no JS warning a bit bigger, and improved the phrasing
a little bit.
2020-06-03 10:44:24 +07:00
camjac251
9a4c0d5cea Update utilsController.js 2020-06-01 04:27:57 -05:00
camjac251
776414814d add user config option for thumbnail size 2020-06-01 04:17:23 -05:00
Bobby Wibowo
7f58d80cff
Fixed URL uploads
I forgot that getting rid of the old hashing system would break this
due to the lack of hashing.
So, fixed that.
2020-06-01 12:23:15 +07:00
Bobby Wibowo
5e5d5c5647
Manage albums admin page, and more!
Resolves #194.

Added pagination for Manage your albums page.

Albums sidebar will now only list 9 albums at most.
Use Manage your albums page to view the rest.
Albums in the list will now have View uploads button after all.

Delete album button for albums renamed to Disable album.
Since techincally the server would've always been disabling the albums
instead of deleting them.
It was something upstream dev's decided, and I haven't bothered changing
its behavior.

I'll work on actual Delete album feature some other days.

As the title says, added Manage albums admin page.

Viewing uploads of an album will hook into albumid: filter key.

I'll work on filter and bulk operations some other days.

Updated styling for disabled albums and users.
Instead of havine a line through them, they will be greyed out.
Disable public page of albums will still use line through however.

Links to album's disabled public page are now clickable.

Added a new button styling is-dangerish.
It'll be orange.

Renamed /api/albums/delete to /api/albums/disable.
For backwards compatibility, /api/albums/delete will still work
but automatically re-routed to /api/albums/disable.

/api/uploads/list will no longer print SQLite errors for moderators
or higher when encountering them.
It was originally used to inform moderators of non-existing colum names
when used for sorting.
But on one of the recent commits, I had added a check for allowed colum
names.

Improved some caching in dashboard page.

Added new entries to cookie policy.

Some other small things.

Bumped v1 version string and rebuilt client assets.
2020-06-01 11:44:16 +07:00
Bobby Wibowo
5f8bad907c
Code clean ups 2020-06-01 08:44:48 +07:00
camjac251
de70b93cc6
thumbnail generation fix and scale
ffmpeg can sometimes generate broken thumbnails (sometimes a full grey image) when it seeks in the input instead of the output. When it's added after the input, it is a bit slower but it is more stable and will fixes the issue for various formats like HEVC or Prores.

The resolution could benefit from being increased 3x for users who scale up the webpage in their browser or tablet views. It becomes scaled up 200% or 300% when viewed on a smaller screen.
2020-05-29 05:51:38 -05:00
Bobby Wibowo
867d005b34
Updated uploadController.js
Removed unnecessary check.
2020-05-29 02:56:11 +07:00
Bobby Wibowo
62a977542e
Switched to BLAKE3 for file hashing [MORE]
UPDATE YOUR CONFIG FILE IF YOU USE CHUNKED UPLOADS!
Read more about this at the end.

Added new dependency: blake3

Hashes will be created as the uploads are being written to disk.
With exception for chunked uploads!
For them specifically, their hashes will be created as they're being
rebuilt into a single file.
Should still be a lot better than the previous case where it had to
re-read the already written files.

To support that feature, added a new file
controllers/multerStorageController.js.
It's just a custom storage engine for Multer.

chunkSize option now allows setting max chunk size from config file.
Previously it was hardcoded to 95MB, but assuming you have paid
Cloudflare plans, you can actually have up to 500MB.

Also moved the option to be after maxSize and before urlMaxSize.
Made a lot more sense to me this way, as chunked uploads only work on
regular uploads.

Updated v1 version string and rebuilt client assets.
2020-05-29 02:52:58 +07:00
Bobby Wibowo
df11fb12ce
Fixed racing condition in initial dirs creation
Fixed #190
2020-05-28 23:10:56 +07:00
camjac251
1efa8c4ebc
more video extensions for thumbs and reorder
added additional extensions to the video list for more support in thumbnails on files (all are common filetypes so ffmpeg should support the various codecs needed)
Also reordered the image and video list alphabetically
2020-05-25 22:47:24 -05:00
Bobby Wibowo
98b2f32ec5
Updated
Fixed browser's timezone offset not being sent with the proper header
name ("minOffset" -> "minoffset").

Server will now actually think client is on UTC timezone if "minoffset"
header is unset.

Stricter "all" header check for uploads list API endpoint.
It will now only accept the header if set to "1" (string).

Fixed server error when either date: or expiry: keys get parsed as
empty strings.

"minoffset" header when listing uploads will now only be sent if
date: or expiry: keys are properly used (i.e. match expected patterns).

Updated filters help message about wrapping with double quotes when
specifying both date and time (since there will be a space in between).

Filters input will now have some automatic clean ups other than trims.
- Replace all tabs with whitespaces.
- Remove whitespaces after "<key>:".

Bumped v1 version string and rebuilt client assets.
2020-05-24 08:28:54 +07:00
Bobby Wibowo
93ee4dae05
Fixed expiry filter breaking with only 1 date 2020-05-17 23:43:08 +07:00
Bobby Wibowo
ebf0037df5
Disallow albumid keys on albums' uploads lists 2020-05-16 22:42:08 +07:00
Bobby Wibowo
51c5a81b18
!!! RUN "yarn migrate" !!!
Added "yarn migrate" as alias for "node ./database/migration.js".
Updated README.md about it.

Added a new column to users database: registration.
It will be used to store user's registration timestamp.
Registration date will be displayed in Dashboard's Manage Users.
Since this is a new column,
existing users will not have registration dates.

Last token change date will now be displayed in Dashboard as well.

<code> elements will now properly have relative font size.

User ID will now be displayed in Edit user dialog for reference purpose.

Bumped v1 version string and rebuilt client assets.
2020-05-16 22:32:32 +07:00
Bobby Wibowo
93c6031bb1
Accept orderby: as alternative for sort: 2020-05-16 22:07:15 +07:00
Bobby Wibowo
968fd74676
Updated
No longer initiate expired uploads check if
config.uploads.temporaryUploadsInterval is not set.

"yarn delete-expired" will now list names of the expired uploads too.
2020-05-16 21:45:14 +07:00
Bobby Wibowo
417adbde93
Fixed type-is and keyword exclusion logic 2020-05-03 04:51:22 +07:00
Bobby Wibowo
f189d86abc
Updated uploadController.js
Fixed filtering null albumid
2020-05-03 04:40:04 +07:00
Bobby Wibowo
126cfe0e15
Added type-is filter keys (is:image and is:video)
Fixed text queries threshold being applied to moderators

Bumped v1 version string and rebuilt client assets
2020-05-03 04:32:45 +07:00
Bobby Wibowo
d201b03f59
Added filtering uploads by albumid
This works when listing all uploads as well, but Album column will only
be shown when albumid key is used in the filters.
I plan to someday add Manage Albums menu, which will use "View uploads"
buttons, just like the ones in Manage Users.
2020-05-03 03:30:50 +07:00
Bobby Wibowo
1980d536db
Initialize upload filters for regular users (WIP)
Updated ESLint's ECMA version to 9 (2018).
I'll need to use some lookbehind regex directives from now on.
It's supported since Node 10, which is the oldest version I'll support.

Refactored "can not" -> "cannot".

Filtering for regular users is still work in progress.
Some features aren't working as expected yet.
2020-05-03 02:39:24 +07:00
Bobby Wibowo
41d52d947b
Added support for negative page num
e.g. -1 means last page, -2 means the 2nd from last, and so on
will only accept up to -N where N is the amount of pages
anything lower will alwasy return the first page
this works for both list uploads and list users APIs

fixed some Object.assign in dashboard.js

added bottom control buttons in manage users
2020-05-02 22:42:23 +07:00
Bobby Wibowo
19b4a5e217
Updated Linux-only disk stats
Restore disk usage stats even when config.linuxDiskStats is off

The said config will now only toggle the 'extended' stats which are
disk usage of each directories within the uploads directory
2020-05-02 19:28:13 +07:00
Bobby Wibowo
ff54f088bd
Added client assets update detection
/api/check and /api/token/verify will now pass along v1 version string
if used (when config.cacheControl is used) for both home.js and
dashboard.js to use to compare version strings.
This should help notify users using browsers with a bit more overbearing
caching (generally mobile browsers).

Rebuilt client assets and bumped v1 version string

Slight update to yarn.lock (a small syntax change for node-gyp, idk why)
2020-04-30 05:56:28 +07:00
Bobby Wibowo
922269181c
More improvements to uploads filtering!
Renamed "orderby" key to "sort" (for sorting uploads).

Fixed non-keyed keyword exclusions not working as expected when
more than one are used at the same time.

Support not specifying "from" date when filtering with range keys
(date and expiry).

Proper logic for NULL values inclusion/exclusion when filtering with
user and/or ip keys.

Improved Help? prompt again!!
Also clarify about timezone differences.

Added logger.debug() function.
Basically a shorthand for console.log(require('util').inspect()).

Rebuilt client asssets and bumped v1 version string.
2020-04-20 01:19:20 +07:00
Bobby Wibowo
dc59476592
Updated utilsController.js
Suppress more ffmpeg/ffprobe error messages
2020-04-19 21:52:34 +07:00
Bobby Wibowo
a6156da798
Added .m4v and .m2ts video extensions
Their thumbs can be made with ffmpeg
Run "yarn thumbs 2" to create thumbs for existing uploads with those
extensions

Rebuilt client assets and bumped v1 version string
2020-04-19 21:41:17 +07:00
Bobby Wibowo
3e3878b93c
Massively overhauled uploads filtering endpoint
Please consult the Help? button again to learn all the syntax changes!
The prompt will now also have its width expanded!

Updated dependency, knex: 0.20.13 -> 0.20.15.

Added new dependency: search-query-parser.

Updated all sub-dependencies.

Critical? Admins-only API /users/edit will no longer return NEW password
salt of the user when randomizing their password.

Added page.escape() function to js/misc/utils.js.
This will be used to escape input in upload filters input box.
The same function used in utilsController.js.

Pretty dates will now use / instead of - for date separator.
This is due to the fact that date range key for filtering uploads
can not accepts dates with - separator.
To avoid inconsistency, we will now use / separator.

Caching system of album public pages will now be disabled during
development (yarn develop).

Cleaned up domClick() function in js/dashboard.js.

If using date or expiry range keys when filtering uploads, attach
client's timezone offset to the API requets.
This will be used by the server to calculate timezone differences.

Success prompt when changing token will now auto-close.

Removed ID column from Manage Users.

Improved success prompt when editing users.
This will properly list all of the edited fields at once,
excluding user group change.
Success message for user group change will require a bit more changes
on the API endpoint, which is a bit annoying.

Rebuilt client-side assets and bumped v1 version string.
2020-04-19 02:52:11 +07:00
Bobby Wibowo
1c260c87b0
Added feature to create new user from Manage Users
New admins-only API route: /api/users/create

Restored checkboxes and bulk buttons in Manage Users
Currently useless, as bulk operators are still WIP

Added filter input in Manage Users, currently WIP

Rebuilt client-side assets and bumped v1 version string
2020-04-17 14:25:18 +07:00
Bobby Wibowo
7389ac6c07
Init support for filtering with keywords (WIP) 2020-04-17 13:36:57 +07:00
Bobby Wibowo
940be1ad99
Updated uploadController.js
Filters input in Manage Uploads will now display error if it's filled
with invalid filter/sort keys.
SQLITE_ERROR will also now be forwarded to users if they're moderators.
The expectation is to only display it when used through Manage Uploads.
2020-04-12 16:30:33 +07:00
Bobby Wibowo
325ccfac81
Added config option to enable linux disk stats
The option is disabled by default.
Meaning you need to explicitly set it to true to restore
the old behavior.

The one that uses both "df" and "du" binaries.
I found them to be very slow with 100k+ uploads
on my ancient potato server.
2020-04-08 07:08:18 +07:00
Bobby Wibowo
8f50ba49c0
Added sorting feature for Manage Uploads
This adds support for a new key named 'orderby'.
The key needs to be used with internal column names used in the db.
The key can be used more than once.
Help message in Dashboard's Manage Uploads have been updated to include
examples using the said key.
2020-04-04 23:36:43 +07:00
Bobby Wibowo
24b9a8ac78
Updated pathsController.js 2020-04-04 21:20:17 +07:00
Bobby Wibowo
46fa9677ff
Extended configuration for virus scanning
Added whitelist extensions
Added max size limit
2020-04-04 21:20:01 +07:00
Bobby Wibowo
edce59243b
Better cache control when not using CDN 2020-02-11 17:18:04 +07:00
Bobby Wibowo
d9ddfe8e9a
Implemented stripping tags from images
... and optionally videos using ffmpeg (still experimental).

Users can choose whether to strip tags of their uploads or not from
the home uploader's Config tab (safe.fiery.me will have it disabled
by default).

The behavior will also be applied to the downloadable ShareX config.

Server owners can choose to force either behavior.

Make sure to add the new config from config.sample.js.

---

Fixed all instances of "e.i." to "e.g.".
My English sucks okay.

Bumped v1 version string.
2019-11-29 20:42:53 +07:00
Bobby Wibowo
337a0a61ff
Implemented parallel URL uploads
This doesn't use the server's built-in ability to accept multiple URLs
per API request.
It behaves the same as regular uploads, in that it executes one API call
per file, simultaneously.

I figured this is a better implementation to shift queues faster.

---

Fetch error from URL uploads due to exceeding size limit will no longer
be logged in server's console.

Clients will also see better formatted error message for URL uploads'
file size limit errors.

---

Bumped dependencies:
knex: 0.20.2 -> 0.20.3
systeminformation: 4.15.3 -> 4.16.0

Bumped v1 version string
2019-11-29 17:42:29 +07:00
Bobby Wibowo
ee920f942f
Cache system & disk stats longer
System stats will be cached for only 1000 ms (1 s)

Disk stats will be cached for 60000 ms (60 s)
2019-11-14 15:08:56 +07:00
Bobby Wibowo
2a81e899b4
Fixed stats not re-generating after errors 2019-11-14 14:53:33 +07:00
Bobby Wibowo
7aaf913166
Actually parse error messages of du & df to String 2019-11-14 14:46:49 +07:00
Bobby Wibowo
98d38d3402
Fixed broken disk stats in linux 2019-11-14 14:42:09 +07:00
Bobby Wibowo
3d09df501d
Updated
Better 'df' handling (check the TODO entry for more details).

Simplified a few lines in dashboard.js.

Bumped v1 version string.
2019-11-14 05:06:59 +07:00
Bobby Wibowo
b646a4a82f
Updated uploadController.js
String updates.
2019-11-10 03:41:54 +07:00
Bobby Wibowo
a28d862c14
Added group bypass to virus scanning
Also better-ish scan results handling again, I guess

Updated dependency knex: 0.20.0 -> 0.20.1
2019-11-06 03:35:04 +07:00
Bobby Wibowo
36763c2a77
Security fix
Replaced all instances of DB .whereRaw with their much safer equivalent
methods.

All previous usages of .whereRaw were vulnerable to SQL injections,
cause we were passing the data directly.

Fortunately, they were only used in API routes that required staff
(moderators included) accounts.

---

Updated dependency:
helmet: 3.21.1 -> 3.21.2
2019-10-22 10:52:52 +07:00