Improved performance of /api/users/:id (admin's manage users).
Promisify fs.writeFile.
Improved performance of /api/stats.
By a lot in Linux, cause uploads size will be deferred to "du" binary.
In addition, total usage of whichever disk uploads path resides on will
also be queried using "df" binary.
Non-Linux will have to rely on manual calculation by querying DB
for each upload's size.
But logics related to uploads stats were also improved to be almost
twice as fast as before.
Improved parsing of /api/stats results on dashboard.js.
This allows ease of extending server's response by not having to update
dashboard.js by much, if at all.
Improved codes relating to item menus in dashboard's sidebar.
Finally much cleaner now 👍
No longer use /api/upload/delete API route from dashboard.
Single file deletion and bulk files deletion, both from uploads list or
by names, will now properly use a single function that will use
/api/upload/bulkdelete API route.
/api/upload/delete will still be kept indefinitely for backward support.
Fixed oddities with Select all checkbox.
Replaced all instances of modifying HTML element's style attribute with
adding/removing is-hidden CSS helper class.
Rephrased all instances of "files" to "uploads" in any display strings.
Fixed notice message when server is on private mode.
A few other improvements.
As the title says, this commit is a massive overhaul.
I've rewritten/restrucuted almost everything in the controller scripts.
Because of that, there's a considerable possibility that I've broken
something somewhere.
Notable changes:
Added temporary uploads.
Removed file name length changer from dashboard,
in favor of an equivalent in homepage config tab.
This allows non-registered users to also set file name length.
A bunch of other undocmented stuff.
I don't know, I'm too tired to remember them all.
Added logger.js to format console logs (adding timestamps).
Re-ordered modules loading in lolisafe.js, and a few other minor edits.
Updated dev dependencies.
A few other minor edits.
On video thumbnails generation, skip video files that do not have any
video streams/channels.
This will prevent unnecessary spawning of ffmpeg for those files.
Also suppress error logging when video streams/channels are not found.
* Updated path resolving for upload folder option.
This SHOULD now allow using an absolute path, even those outside of
the lolisafe installation.
* Added a config option at uploads -> generateThumbs, named placeholder.
It's a string option that lets you set path of the placeholder image
for files whose thumbnail could not be generated.
It defaults to the old hard-coded path when set to falsy value.
* Fixed thumbs script not using upload folder from config file.
* Added filters for thumb generation error handling.
This is used to ignore some common error messages,
such as file formats not being supported.
* Replaced all instances of getElementById and getElementsByClassName
with querySelector or querySelectorAll.
* Updated utilsController.js to stop disabling
no-async-promise-executor eslint rule.
* Removed unused lines in dashboard.njk.
* Refactored maxFileSize to maxSize in home.{css,js,njk}.
* Updated ClamAV codes in lolisafe.js. No more pinging.
Since querying version will also check connection anyway.
* Option "Upload to album" in homepage is now selectable.
Selecting this option will restore the uploader to not associate files
with an album.
* Fixed uploader to properly respect server's max file size.
Also updated error message of file size to use MB instead of MiB.
* Creating an album from homepage will automatically select the album.
* Updated Dropzone.js to v5.5.0.
* Bumped v1 & v3 version strings.
* Various other small fixes.
Updated deps.
Re-enabled eslint rule no-async-promise-executor.
Updated some codes to follow the said rule.
Though I had to disable the rule in a line at utilsController.js
due to complexity.
I'll give it more thoughts in the future.
Bumped v1 version string.
* Simplified some logics.
* Try to unlink thumbnail when failing to generate video thumbnail,
since ffmpeg may have already created an incomplete thumbnail.
utilsController.js:
+ /api/stats: Cache invalidation requests will now only store timestamp
of the request instead of purging the cache entirely.
Subsequent requests to the API will no longer attempt to generate stats
if a previous request before it is still generating, since it can
take a while in big sites.
So instead, the cached stats will be returned to them, regardless of
whether it's still valid.
This should avoid "race condition" in sites with multiple admins.
* dashboard.js:
+ Added an appendix into "Delete by names" feature in dashboard to
inform staffs that they can use the feature to delete files by other
users as well.
+ Added comments explaning the need of page.domClick() function.
+ Addition to the change regarding /api/stats route: If there is no
cached stats, but a previous request is still generating, inform
the user about it.
config.sample.js + lolisafe.js:
+ We now support configuring rate limits through config file.
No more hard-coded rate limits.
Don't forget to update your config.js accordingly.
package.json + yarn.lock:
+ Updated dependencies.
* Added Statistics menu to Administration items in dashboard.
* Added /api/stats route.
Imo, my implementation of this whole thing is rather dirty-ish, but hey
as long as it works.
I'll be using lolisafe2 for future devs tbh.
Updated utilsController.bulkDeleteFiles() to chunk opeartion by a max of
999 values (which is SQLite's default var limit).
However, there's a risk of hitting SQLITE_BUSY since we attempt to
delete chunks all at once (as can be seen by how we use Promise.all
instead of patiently waiting the chunks one by one).
However, I'm not really sure why, but the operations will still
be finished eventually, so I'll let that be for now (?).
Fixed a silly concat in albumsController.js.
Updated Cloudfalre's cache purge utility.
It will now split URLs into chunks of 30 URLs which then will be purged
one chunk at a time.
I just found out Cloudflare's API have a limit of 30 URLs for the API.
Rewritten function to generate video thumbnails with ffmpeg.
This should be much faster than ever.
This should also solve an issue where potrait videos could have their
thumbnails be taller than 200px, since it was only forcing width to be
no larger than 200px.
Updated dashboard's styling to make sure potrait thumbnails (only matter
for video thumbnails) not going out of its container.
Updated thumbs.js (yarn thumbs) script to display elapsed time for each
operation in seconds.
Bumped v1 version string (for dashboard.css).
Removed .bmp from list of allowed image extensions for thumbs gen
Sharp doesn't support BMP format
Increased seek timestamp for video thumbs gen from 1% to 20%
NoJS uploader will now display the original file names in the results.
lolisafe.js will now automaticaly purge Cloudflare's cache of frontend
pages everytime it launches.
Of course this only applies when cacheControl is on in config file.
This sorta makes scripts/cfpurge.js script obsolete.
Added cfpurge.js to scripts directory.
This can be used to purge cache of frontend pages and uploads.
Do "node scripts/cfpurge.js --help" for usage.
Removed "randver" from package.js/scripts.
I've installed randomstring globally instead and just simply do:
randomstring n
Back then I didn't know it could be used that way.
Images smaller than 200x200 will no longer be resized upwards, instead
they will be padded with transparent pixels.
This was the old behavior before we switched from GM to sharp.
With GM, its resize() function would do exactly that, but I couldn't
figure out how to do the same with only sharp's resize() function, so I
had to make do with a combination of resize() and extend().
Also updated error messages in dashboard when trying to load
out-of-index page in uploads/users lists.
Updated v1 version string due to dashboard.js being modified.
Added extended support for URL uploads.
Namely URL proxy support and separate extensions filter (as in separate
from the primary extensions filter).
There's also a new option to set a disclaimer message that will be
printed underneath the URL uploads form.
Trust proxy is now toggleable from the configuration file.
I think they should only be enabled when you're behind proxy such as
Cloudflare or Incapsula.
I'm not sure how it behaves with only a bare nginx reverse proxy though.
Empty files can now be filtered.
Sorted preset extensions filter in config.sample.js.
Rephrased some options in config.sample.js as well.
maxTries now default to 3 in config.sample.js.
Various other small changes.
Some small fixes
Removed GET route of /upload/delete.
I first wanted to implement a ShareX-compatible deletion URL,
but then I figured I'd need to setup delete token system,
and I was like, "screw that, I don't even use ShareX",
so yeah.
No more enforced curly for if/else/for/while/do blocks w/ one statement.
With that said, auto-fixed all JS files to follow the rule.
I'd also like to apologize for the inconveniences this commit cause,
after all it was me who intentionally enforced curly rule back then.
Why the change of heart?
After doing some more non-JS codes recently, I realized it was
pretty stupid of me to enforce that.
Added description column into albums.
So yeah, now albums can have description.
It'll only be shown in the album's edit popup and public link.
HTML chars will now be escaped from album's name and description.
Removed message warning about CDN cache from album's public link.
A shortened version will be shown as the download button's tooltip.
Darkened color of textarea's placeholder.
Bumped v1 version string.
Reworked unique name generator to prevent the same unique identifier from being used if it was already used with a different extension (e.i. If a file named aBcD.jpg already exists, then files such as aBcD.png or aBcD.txt may not exist).
This is mainly to deal with the fact that thumbnails are only being saved as PNG, so if the same unique name is being used by multiple image/video extensions, then only one of them will have the proper thumbnail.
If you already have existing files with matching unique name but varying extensions, unfortunately you can only deal with them manually for now (either allocating new unique names or deleting them altogether).
Added a new config option to filter files with no extension.
Files with no extensions will no longer have their original name appended to the allocated random name (e.i. A file named "textfile" used to become something like "aBcDtextfile", where "aBcD" was the allocated random name. Now it will only just become "aBcD").
In relation to that, utils.extname() function will now always return blank string if the file name does not seem to have any extension.
Though files such as '.DS_Store' (basically anything that starts with a dot) will still be accepted.
Examples:
.hiddenfile => .hiddenfile
.hiddenfile.sh => .sh
.hiddenfile.001 => .hiddenfile.001
.hiddenfile.sh.001 => .sh.001
Simplified error messages of /api/upload/finishchunks.
Most, if not all, of the error responses for /api/upload* will now have HTTP status code 400 (bad request) instead of 200 (ok).
I plan to generalize this for the other API routes in the future.
Updated home.js to properly handle formatted error message when the response's status code is not 200 (ok).
Bumped v1 version string (due to home.js).
Moved utils.getPrettyBytes() and utils.getPrettySize() to client's dashboard.js.
Thus, server will no longer return prettified size and date (it'll be prettified by the client instead).
To be honest, I don't even know why I had them in server-side, it's obviously better this way.
Removed "path-complete-extname" module in favor of an in-house solution, utilsController.extname().
For now the function will attempt to preserve multi-archive extensions (.001, .002, and so on), as well as some known tarballs (.tar.gz and the likes).
The function will always return lower case extension. It should be fine, but do keep it in mind.
* Updated utilsController.js: Unlink thumb of type symlink whenever generateThumbs() is called.
* Updated thumbs.js: Add stats (success/error/skipped).
* Added an experimental virus scanning feature using ClamAV. This has only been tested with an Ubuntu machine.
* File extensions will now be parsed with path-complete-extname module. This will ensure extensions such as .tar.gz are properly parsed.
Notice: It may take a minute or so to start the safe with virus scanning, as apparently the module takes a while to create the engine. I'm guessing since it'll be loaded to memory? Either way, once the engine is created, everything should work fine. Virus scanning should also not have that much of an impact to the upload time.
* Faster upload response. Back-end will no longer wait for album timestamps to be updated before sending out response.
* Added a simple thumbnail generation script at scripts/thumbs.js. You can use this to generate thumbnails for existing files before enabling the option in config.js.
* Various other code improvements.
* Added upload by URLs. It has its own max size (config.uploads.urlMaxSize), make sure your config matches config.sample.js.
Here's a brief video showing it in action: https://i.fiery.me/CUhQ.mp4.
* /api/upload now supports uploading by URLs. Devs will only need to POST a JSON request containing a key named "urls", which is an array of the urls to upload.
* Added file extension filter to /api/upload/finishchunks.
* Added proper total chunks size check to /api/upload/finishchunks.
* Various code improvements.
* Possible performance improvement. Some bulk db queries will now be executed in a single query instead of spawning multiple async task for each query. This is sorta experimental though, use it at your own risk (though I'll use it right away at safe.fiery.me).
* It's now possible for root user to add files to other users' albums through the API route. I don't plan on allowing root user to list other users' album list from the dashboard, I just thought that there'd be no harm in extending the API a little bit.
* Kinda better error logging for uncaught exception and unhandled rejection. Their stack trace should be logged now.
* Added Cloudflare purge cache support. Check configuration sample at config.sample.js.
When it's enabled, whenever files are being deleted, it will send a POST request to Cloudflare's API to purge cache of the deleted files.
This adds a new dependency called "snekfetch". It's lightweight though.
* uploadsController.delete() will now wrap uploadsController.bulkDelete() instead.
NOTICE: Please update your config.js. Use config.sample.js as the template.
There were a couple of renames and restructures.
* Album zipper API route will now internally save its state when it's generating zip files, and any subsequent requests will silently be "postponed" until the first spawned task is finished. This will guarantee that there are no multiple zipping tasks for the same album. The method may seem a bit hackish though.
* All instances of console.log(error) were replaced with console.error(error). This will guarantee that any error goes to stderr instead of stdout.
* Deleting file by names will now properly remove successful files from the textarea. There was a logic flaw.
* Failure to generate thumbnails will no longer print the full stack, but instead only the error message. It will also then symlink a template image from /public/images/unavailable.png (it's only a simple image that says that it failed to generate thumbnail).
This haven't been tested in Windows machines, but it'll probably work fine.
I thought of adding a new column to files table which will store information whether the thumbnail generation is sucessful or not, but oh well, I'll go with this method for now.
* Updated API route: /upload/bulkdelete.
It now accepts an additional property named "field". In it you can now enter either "id" or "name", which will set whether it will bulk delete by ids or names respectively. It also no longer accepts property named "ids", instead it has to be named "values" (which of course is an array of either ids or names). So yeah, now the API route can be used to bulk delete by ids and names.
In the future this will be expanded to bulk deleting files by username (only accessible by root of course).
* Added a form to bulk delete files by names for the hardcore user, like me (https://i.fiery.me/AHph.png).
* Some design update. Mainly forms restructuring aimed at tight screens.
* Changing file name length, requesting new token and setting new password will no longer reload the dashboard page on success. Instead it will simply silently reload the form.
* utils.bulkDeleteFilesByIds() replaced by utils.bulkDeleteFiles() which now can either by ids or names. This will be the one that will eventually be extended for deleting by username.
* Various other code improvements.
