this means internal Map-store may actually contain more keys than limit,
but only if the extraenous keys are held-keys (i.e. has null value)
this expects that you don't manually set keys with null values into the
store however
we used to store number directly into the string size field, and
during the conversion it seemed to always add ".0" at the final string,
probably because the driver or sqlite3 itself assumes float
please run yarn migrate after pulling this commit
if you skip converting the DB, file duplicates check will fail to
function
and in the future im planning to do size statistics in bigint, which
will also fail if not converted
this will display all information recorded from the specified file, but
only to the users that own them (it requires token)
this page also has a delete file button, allowing us to provide link to
this page for sharex deletion url option
once again, this is only for authenticated users, and will only show
file that the users own, unless said user is a moderator or higher
db is now initiated via utilsController, and gets re-used by main
lolisafe.js script and the other controllers
this should now make sure we only have 1 active db connection pool ever
this is a pretty big refactor due to how we used to reference db,
but it should functionally be identical
every now and then i got curious of what people add to their forks,
and sometimes i'd be like, "hey, that makes sense!"
i remember that back in the days, i switched to nunjucks because of
someone's fork too
we used to use handlebars, which i've had some misgivings with for
awhile back then
either way all i'm saying is that it's the case this time around with
this commit
thanks for reading
49d12004e8
this should properly dispose of unfinished write & hasher streams on
any errors if they have been initiated
also do content-length header check a bit more early
first layer is via sending HEAD request to the url to determine its size
via content-length header
however not all hosts properly set the header, so we ignore it if
it isn't a valid number
next via size option in fetch(), which supposedly limits response body
size during the request itself (?)
lastly via checking actual bytes written to physical file as reported by
fs.createWriteStream()
this supersedes the old temporaryUploadAges, while maintaining full
backwards-compatibility.
please consult config.sample.js if you want to start using this
this required expanding our custom error classes with support for
arbitrary internal api error codes
however it'll only be used for invalid token errors for now (10001)
no plan to assign codes to other existing api errors
at that point it's probably better to redo the whole api infrastructure
change logic to list physical files instead, since the zipGeneratedAt
attribute may still exist despite site owners having already done
physical clean-ups
only usergroup and file extension bypass
real file size can't be determined before passthrough scan,
so there's no bypass by max file size
please read the comments in sample config file
refactored utils.clamscan into utils.scan
GIFs are known to not work without custom globally-installed libvips
with ImageMagick or GraphicsMagick support.
https://sharp.pixelplumbing.com/api-output#gifhttps://sharp.pixelplumbing.com/install#custom-libvips
It's highly recommended to update your config following the changes to
the sample config file.
This also addressed a bug where images would still get recorded to DB
despite them not existing physically due to strip tags errors.
Made the codes for stats generation a bit more readable.
Usage percentage for file systems will now properly reflect "non-root"
usage percentage in ext2/3/4 file systems.
Also Object.freeze() permissions object in permissionController.
I don't think it's much to be worried about, as no "set" will be done to
it during the service's operation, but oh well, might as well.
ClientError will default to 400 HTTP error code.
ServerError will default to 500 HTTP error code.
Following the previous commit, these for now are only being used in
albumsController. More will soon follow.
Additionally fixed existing album names can sometimes be re-used when
editing an album.
This will be used for errors that are to be delivered to users, AND not
to be logged into the server (as in it stacktraces and all).
This will eventually remove the need to throw string literals.
In this commit, this has only been implemented on albumsController.js,
but more will soon to come.
well, that was a pointless endeavor.
i wasn't thinking clearly.
also updated all scripts that previously used them, to use a combo of
for-loop and setInterval to print progress.
it will re-map body of /api/album/get/:id into upstream-compatible body.
prep for lolisafe albums support for magane plugin.
/api/album/:id/:page will stil respond with the old format as that's
what the dashboard use and expect.
list views of uploads, users and albums in dashboard will now show
total items count on the table's top right corner.
they'll also be collapsed on initial page load
this uses a new client-side dependency, bulma-collapsible
https://github.com/creativebulma/bulma-collapsible
/api/albums to fetch albums list now support simple reply, where only
their ids and names will be returned.
this simple reply will also return all of the user's albums, instead of
being limited to only 9 or 25 entires like before.
fixed add to album in dashboard, and album selector in homepage uploader
being limited to only 25 albums.
with preset delay in between (60 secs if rate limited, 5 secs for
unexpected errors)
an alternative to global queue mechanism in #342
not perfect, but easier to implement
closes#342
much more expandable, and should be easier to understand overall.
make more statistics operations run concurrently to speed them up.
make linuxDiskStats config key obsolete by using systeminformation
package to also query for any mounted file systems.
Replaced all Array.concat() with spread operator and/or push().
Fixed some faulty search logics when using exclusion filters.
Fixed capitalization of mac font in _variables.scss. Stylelint somehow
fricked up again in my previous commit.
Now will calculate usage as (total - avail).
In Linux, ext filesystems by default reserves 5% of the space to be
usable by root, making them essentially already "used" space.
Originally we didn't take that into account.
Disk usage percentage will now round down: Math.round() -> Math.floor().
The general behavior in other tools such as "df".
src/js/misc/utils.js: Added page.getPrettyUptime().
Updated dependency:
helmet: 3.23.3 -> 4.1.0
lolisafe.js: Disabled CSP by default.
Since helmet 4, CSP would instead be enabled by default.
I wasn't aware ShareX wouldn't do a basic GET request to the delete URL,
but would instead open your browser to it.
Unfortunately, due to the structure of lolisafe auth setting token
through HTTP header, which had always been done by JS as it has to get
the value from local storage, just visiting the API from browser will
merely respond with "No token provided".
We'd need cookie-based auth, which I do not want to add.
The old me that added deletion API long ago might have been aware of
this fact, but the me of a few hours ago already forgot..
I'll give this further thoughts at a later date, as ShareX isn't really
a priority to me.
Bumped v1 version string and rebuilt client assets.
For registered users only!
This requires adding a basic GET API for file deletion, so that I did.
Configs which guests download will not include pattern for delete URL,
so they won't get notified of unusable delete URL or anything like that.
dev: Improved logger.debug() to support specifying options for node's
Util.inspect() if an object is set as its last param
(assuming >1 params).
Default options now also includes enabling colors.
src/js/utils.js: Simplified dynamic ShareX config generator.
Among other things, it will now use JSON.stringify().
I don't even remember why we didn't use that in the first place..
Some logic improvements in src/js/home.js.
Bumped v1 version string and rebuilt client assets.
The service will now automatically clean up any leftover data from any
inactive & unfinished chunked upload attempts.
Updated config.sample.js for updated description and new sub-option
for chunkSize option.
Inspired by our recent switch to using blake3 for file hashing, chunks
will now be written to a tmp file directly as they're uploaded.
So no more waiting so long for "rebuilding chunks".
There will still be some delay on every following attempts of uploading
each chunks. I'm not sure the specifics, as we're already reusing the
write stream.
For admins, preference for your personal uploads list and Manage uploads
aren't shared, just like thumbs/lists toggle.
Non-keyed keywords for filtering will now apply to original names too.
Added a new fontello icon for this button.
Various other things I'm too lazy to write.
Notable changes, use fluent-ffmpeg's functions instead of passing raw
ffmpeg switches with inputOptions() or outputOptions().
It seems fluent-ffmpeg may be doing some janks with outputOptions().
I can't get select="eq(pict_type\\,I)" to work with it, but it works
with videoFilters(). Weird stuff.
Removed its dependency towards albums' editedAt property.
Editing album's metas (name, description, etc) will no longer update
its editedAt property.
Instead it will now ONLY be updated when adding/removing files to/from
it. Just like how it was meant to be, which was to be used to check
whether it's necessary to re-generate their downloadable ZIPs.
Albums public page cache will still be properly invalidated when
adding/removing files to/from it, as well as after editing their metas.
Added views/album-notice.njk to be used to render okay-ish notice when
an album's public page is still being generated.
I was originally thinking of using it for disabled albums as well, but
I refrained from it to reduce the possibility of disabled album IDs from
being easily scanned (as it just returns 404 now).
Removed invalidatedAt property from stats cache. Instead their caches
will immediately be nullified as they should (thus frees up memory
slightly as well).
Stats cache for albums will now only be cleared when truly necessary.
As in, adding/removing files to/from albums will no longer clear them.
Updated Nunjucks files to properly use h1, h2, h3 tags in actual
hierarchical orders.
Elements that don't need to use hX tags will now use P instead.
Nothing changes visually, only structurally.
Fixed some elements in Nunjucks using single quotes instead of
double quotes. They'd have worked the same, but consistency.
Added h1 title in FAQ page.
Make text for no JS warning a bit bigger, and improved the phrasing
a little bit.
Resolves#194.
Added pagination for Manage your albums page.
Albums sidebar will now only list 9 albums at most.
Use Manage your albums page to view the rest.
Albums in the list will now have View uploads button after all.
Delete album button for albums renamed to Disable album.
Since techincally the server would've always been disabling the albums
instead of deleting them.
It was something upstream dev's decided, and I haven't bothered changing
its behavior.
I'll work on actual Delete album feature some other days.
As the title says, added Manage albums admin page.
Viewing uploads of an album will hook into albumid: filter key.
I'll work on filter and bulk operations some other days.
Updated styling for disabled albums and users.
Instead of havine a line through them, they will be greyed out.
Disable public page of albums will still use line through however.
Links to album's disabled public page are now clickable.
Added a new button styling is-dangerish.
It'll be orange.
Renamed /api/albums/delete to /api/albums/disable.
For backwards compatibility, /api/albums/delete will still work
but automatically re-routed to /api/albums/disable.
/api/uploads/list will no longer print SQLite errors for moderators
or higher when encountering them.
It was originally used to inform moderators of non-existing colum names
when used for sorting.
But on one of the recent commits, I had added a check for allowed colum
names.
Improved some caching in dashboard page.
Added new entries to cookie policy.
Some other small things.
Bumped v1 version string and rebuilt client assets.
ffmpeg can sometimes generate broken thumbnails (sometimes a full grey image) when it seeks in the input instead of the output. When it's added after the input, it is a bit slower but it is more stable and will fixes the issue for various formats like HEVC or Prores.
The resolution could benefit from being increased 3x for users who scale up the webpage in their browser or tablet views. It becomes scaled up 200% or 300% when viewed on a smaller screen.
UPDATE YOUR CONFIG FILE IF YOU USE CHUNKED UPLOADS!
Read more about this at the end.
Added new dependency: blake3
Hashes will be created as the uploads are being written to disk.
With exception for chunked uploads!
For them specifically, their hashes will be created as they're being
rebuilt into a single file.
Should still be a lot better than the previous case where it had to
re-read the already written files.
To support that feature, added a new file
controllers/multerStorageController.js.
It's just a custom storage engine for Multer.
chunkSize option now allows setting max chunk size from config file.
Previously it was hardcoded to 95MB, but assuming you have paid
Cloudflare plans, you can actually have up to 500MB.
Also moved the option to be after maxSize and before urlMaxSize.
Made a lot more sense to me this way, as chunked uploads only work on
regular uploads.
Updated v1 version string and rebuilt client assets.
added additional extensions to the video list for more support in thumbnails on files (all are common filetypes so ffmpeg should support the various codecs needed)
Also reordered the image and video list alphabetically
Fixed browser's timezone offset not being sent with the proper header
name ("minOffset" -> "minoffset").
Server will now actually think client is on UTC timezone if "minoffset"
header is unset.
Stricter "all" header check for uploads list API endpoint.
It will now only accept the header if set to "1" (string).
Fixed server error when either date: or expiry: keys get parsed as
empty strings.
"minoffset" header when listing uploads will now only be sent if
date: or expiry: keys are properly used (i.e. match expected patterns).
Updated filters help message about wrapping with double quotes when
specifying both date and time (since there will be a space in between).
Filters input will now have some automatic clean ups other than trims.
- Replace all tabs with whitespaces.
- Remove whitespaces after "<key>:".
Bumped v1 version string and rebuilt client assets.
Added "yarn migrate" as alias for "node ./database/migration.js".
Updated README.md about it.
Added a new column to users database: registration.
It will be used to store user's registration timestamp.
Registration date will be displayed in Dashboard's Manage Users.
Since this is a new column,
existing users will not have registration dates.
Last token change date will now be displayed in Dashboard as well.
<code> elements will now properly have relative font size.
User ID will now be displayed in Edit user dialog for reference purpose.
Bumped v1 version string and rebuilt client assets.
No longer initiate expired uploads check if
config.uploads.temporaryUploadsInterval is not set.
"yarn delete-expired" will now list names of the expired uploads too.
This works when listing all uploads as well, but Album column will only
be shown when albumid key is used in the filters.
I plan to someday add Manage Albums menu, which will use "View uploads"
buttons, just like the ones in Manage Users.
Updated ESLint's ECMA version to 9 (2018).
I'll need to use some lookbehind regex directives from now on.
It's supported since Node 10, which is the oldest version I'll support.
Refactored "can not" -> "cannot".
Filtering for regular users is still work in progress.
Some features aren't working as expected yet.
e.g. -1 means last page, -2 means the 2nd from last, and so on
will only accept up to -N where N is the amount of pages
anything lower will alwasy return the first page
this works for both list uploads and list users APIs
fixed some Object.assign in dashboard.js
added bottom control buttons in manage users
Restore disk usage stats even when config.linuxDiskStats is off
The said config will now only toggle the 'extended' stats which are
disk usage of each directories within the uploads directory
/api/check and /api/token/verify will now pass along v1 version string
if used (when config.cacheControl is used) for both home.js and
dashboard.js to use to compare version strings.
This should help notify users using browsers with a bit more overbearing
caching (generally mobile browsers).
Rebuilt client assets and bumped v1 version string
Slight update to yarn.lock (a small syntax change for node-gyp, idk why)
Renamed "orderby" key to "sort" (for sorting uploads).
Fixed non-keyed keyword exclusions not working as expected when
more than one are used at the same time.
Support not specifying "from" date when filtering with range keys
(date and expiry).
Proper logic for NULL values inclusion/exclusion when filtering with
user and/or ip keys.
Improved Help? prompt again!!
Also clarify about timezone differences.
Added logger.debug() function.
Basically a shorthand for console.log(require('util').inspect()).
Rebuilt client asssets and bumped v1 version string.
Their thumbs can be made with ffmpeg
Run "yarn thumbs 2" to create thumbs for existing uploads with those
extensions
Rebuilt client assets and bumped v1 version string
Please consult the Help? button again to learn all the syntax changes!
The prompt will now also have its width expanded!
Updated dependency, knex: 0.20.13 -> 0.20.15.
Added new dependency: search-query-parser.
Updated all sub-dependencies.
Critical? Admins-only API /users/edit will no longer return NEW password
salt of the user when randomizing their password.
Added page.escape() function to js/misc/utils.js.
This will be used to escape input in upload filters input box.
The same function used in utilsController.js.
Pretty dates will now use / instead of - for date separator.
This is due to the fact that date range key for filtering uploads
can not accepts dates with - separator.
To avoid inconsistency, we will now use / separator.
Caching system of album public pages will now be disabled during
development (yarn develop).
Cleaned up domClick() function in js/dashboard.js.
If using date or expiry range keys when filtering uploads, attach
client's timezone offset to the API requets.
This will be used by the server to calculate timezone differences.
Success prompt when changing token will now auto-close.
Removed ID column from Manage Users.
Improved success prompt when editing users.
This will properly list all of the edited fields at once,
excluding user group change.
Success message for user group change will require a bit more changes
on the API endpoint, which is a bit annoying.
Rebuilt client-side assets and bumped v1 version string.
New admins-only API route: /api/users/create
Restored checkboxes and bulk buttons in Manage Users
Currently useless, as bulk operators are still WIP
Added filter input in Manage Users, currently WIP
Rebuilt client-side assets and bumped v1 version string
Filters input in Manage Uploads will now display error if it's filled
with invalid filter/sort keys.
SQLITE_ERROR will also now be forwarded to users if they're moderators.
The expectation is to only display it when used through Manage Uploads.
The option is disabled by default.
Meaning you need to explicitly set it to true to restore
the old behavior.
The one that uses both "df" and "du" binaries.
I found them to be very slow with 100k+ uploads
on my ancient potato server.
This adds support for a new key named 'orderby'.
The key needs to be used with internal column names used in the db.
The key can be used more than once.
Help message in Dashboard's Manage Uploads have been updated to include
examples using the said key.
