Mirror/filesafe
1
0
Fork 0
mirror of https://github.com/BobbyWibowo/lolisafe.git synced 2025-01-18 17:21:33 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

feat: merge 'safe.fiery.me-feat/hyper-express'

Browse Source
This commit is contained in:
Bobby Wibowo 2022-07-22 03:23:07 +07:00
commit eb41aea5f2
No known key found for this signature in database
GPG Key ID: 51C3A1E1E22D26CF
31 changed files with 3968 additions and 3983 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

40
README.md
View File

@ -4,21 +4,47 @@
[![GitHub license](https://img.shields.io/badge/license-MIT-blue.svg?style=flat-square)](https://raw.githubusercontent.com/WeebDev/lolisafe/master/LICENSE)
## `safe.fiery.me`
[![JavaScript Style Guide](https://cdn.rawgit.com/standard/standard/master/badge.svg)](https://github.com/standard/standard)
This fork is the one being used at [https://safe.fiery.me](https://safe.fiery.me). If you are looking for the original, head to [WeebDev/lolisafe](https://github.com/WeebDev/lolisafe).
## Features
* Powered by [uWebSocket.js](https://github.com/uNetworking/uWebSockets.js/) & [HyperExpress](https://github.com/kartikk221/hyper-express) for a much more performant web server, due to being a Node.js binding of [uWebSockets](https://github.com/uNetworking/uWebSockets) written in C & C++.
* Powered by [better-sqlite3](https://github.com/WiseLibs/better-sqlite3) for performant SQLite3 database (using [Knex.js](https://knexjs.org/) for abstraction, thus support for other database engines *may* also come in the future).
* Faster file hashing for duplicates detection by using [BLAKE3](https://github.com/BLAKE3-team/BLAKE3) hash function, powered by [blake3](https://github.com/connor4312/blake3) Node.js library.
* ClamAV virus scanning support for Linux/OS X servers ([read more](#clamav-support)).
* Front-end pages templating with [Nunjucks](https://mozilla.github.io/nunjucks/).
* A more integrated Cloudflare support (automatically purge files remote cache upon deletion, and more).
* Chunked uploads to support 100MB+ files when hosted behind Cloudflare, or any other proxies with file upload size limits.
* Performant rate limits powered by [rate-limiter-flexible](https://github.com/animir/node-rate-limiter-flexible).
* Albums with shareable pretty public pages.
* User dashboard to manage own uploads and albums.
* Admin dashboard to manage all uploads, albums, and users.
* Robust files search/filters and sorting in the dashboard.
* Usergroups-based permissions.
* Configurable file retention periods per-usergroups.
* Strip images EXIF tags if required (can be set to be toggleable by users, and with experimental support for stripping videos tags as well).
* Various options configurable via header tags upon file uploads (selected file retention period, whether to strip EXIF tags, and more).
* ShareX support with config file builder in the homepage.
* Token-based authentication on all APIs, allowing you to easily integrate it with anything.
* ... and more!
## Differences with Upstream/Chibisafe
This fork is the one being used at [https://safe.fiery.me](https://safe.fiery.me).
It was originally based on [WeebDev/lolisafe](https://github.com/WeebDev/lolisafe) v3, but later have been so heavily rewritten that it is now simply its own thing.
Chibisafe is an upstream rewrite & rebrand, and technically is lolisafe v4.
If you want to use an existing lolisafe v3 database with this fork, copy over `database/db` file from your previous installation, then run `yarn migrate` at least once to create the new database columns introduced in this fork (don't forget to make a backup).
> **Said migration script is NOT COMPATIBLE with chibisafe (a.k.a lolisafe v4/rewrite).**
> **Said migration script is NOT COMPATIBLE with Chibisafe's database.**
Configuration file of lolisafe v3 (`config.js`) is also NOT fully compatible with this fork. There are some options that had been renamed and/or restructured. Please make sure your config matches the sample in `config.sample.js` before starting.
Configuration file of lolisafe v3 (`config.js`) is also NOT fully compatible with this fork. There are some options that had been renamed and/or restructured. Please make sure your config matches the sample in `config.sample.js` before starting and/or migrating your previous database.
## Running in production mode
1. Ensure you have at least Node v12.22.0 installed (fully compatible up to Node v16.x LTS, untested with v17 or later).
1. Ensure you have at least Node v14 installed (fully compatible up to Node v16.x LTS, untested with v17 or later).
2. Clone this repo.
3. Copy `config.sample.js` as `config.js`.
4. Modify port, domain and privacy options if desired.
@ -35,6 +61,8 @@ When running in production mode, the safe will use pre-built client-side CSS/JS
The pre-built files are processed with [postcss-preset-env](https://github.com/csstools/postcss-preset-env), [cssnano](https://github.com/cssnano/cssnano), [bublé](https://github.com/bublejs/buble), and [terser](https://github.com/terser/terser), and done automatically with [GitHub Actions](https://github.com/BobbyWibowo/lolisafe/blob/safe.fiery.me/.github/workflows/build.yml).
> If you want to this on Docker, please check out [docker directory](https://github.com/BobbyWibowo/lolisafe/tree/safe.fiery.me/docker).
## Running in development mode
This fork has a separate development mode, with which client-side CSS/JS files in `src` directory will be automatically rebuilt using [Gulp](https://github.com/gulpjs/gulp#what-is-gulp) tasks.

85
config.sample.js
View File

@ -84,9 +84,14 @@ module.exports = {
/*
Pages to process for the frontend.
To add new pages, you may create a new Nunjucks-templated pages (.njk) in "views" directory,
or regular HTML files (.html) in "pages/custom" directory,
then simply add the filename without its extension name into the array below.
Alternatively, you may create regular HTML files (.html) in "pages/custom" directory.
If doing so, you don't need to add the filename into the array,
as any changes in said directory will be detected live.
You may even add or remove pages while lolisafe is running.
*/
pages: ['home', 'auth', 'dashboard', 'faq'],
@ -206,42 +211,20 @@ module.exports = {
trustProxy: true,
/*
Rate limits.
Please be aware that these apply to all users, including site owners.
https://github.com/nfriedly/express-rate-limit/tree/v6.3.0#configuration
Rate limiters.
https://github.com/animir/node-rate-limiter-flexible/wiki/Memory
*/
rateLimits: [
{
// 10 requests in 1 second
routes: [
'/api/'
],
config: {
windowMs: 1000,
max: 10,
legacyHeaders: true,
standardHeaders: true,
message: {
success: false,
description: 'Rate limit reached, please try again in a while.'
}
}
},
rateLimiters: [
{
// 2 requests in 5 seconds
routes: [
// If multiple routes, they will share the same points pool
'/api/login',
'/api/register'
],
config: {
windowMs: 5 * 1000,
max: 2,
legacyHeaders: true,
standardHeaders: true,
message: {
success: false,
description: 'Rate limit reached, please try again in 5 seconds.'
}
options: {
points: 2,
duration: 5
}
},
{
@ -249,11 +232,9 @@ module.exports = {
routes: [
'/api/album/zip'
],
config: {
windowMs: 30 * 1000,
max: 6,
legacyHeaders: true,
standardHeaders: true
options: {
points: 6,
duration: 30
}
},
{
@ -261,19 +242,35 @@ module.exports = {
routes: [
'/api/tokens/change'
],
config: {
windowMs: 60 * 1000,
max: 1,
legacyHeaders: true,
standardHeaders: true,
message: {
success: false,
description: 'Rate limit reached, please try again in 60 seconds.'
}
options: {
points: 1,
duration: 60
}
},
/*
Routes, whose scope would have encompassed other routes that have their own rate limit pools,
must only be set after said routes, otherwise their rate limit pools will never trigger.
i.e. since /api/ encompass all other /api/* routes, it must be set last
*/
{
// 10 requests in 1 second
routes: [
'/api/'
],
options: {
points: 10,
duration: 1
}
}
],
/*
Whitelisted IP addresses for rate limiters.
*/
rateLimitersWhitelist: [
'127.0.0.1'
],
/*
Uploads config.
*/

1044
controllers/albumsController.js
View File

File diff suppressed because it is too large Load Diff

588
controllers/authController.js
View File

@ -30,105 +30,108 @@ const self = {
// https://github.com/kelektiv/node.bcrypt.js/tree/v5.0.1#a-note-on-rounds
const saltRounds = 10
self.verify = (req, res, next) => {
Promise.resolve().then(async () => {
const username = typeof req.body.username === 'string'
? req.body.username.trim()
: ''
if (!username) throw new ClientError('No username provided.')
self.verify = async (req, res) => {
// Parse POST body
req.body = await req.json()
const password = typeof req.body.password === 'string'
? req.body.password.trim()
: ''
if (!password) throw new ClientError('No password provided.')
const username = typeof req.body.username === 'string'
? req.body.username.trim()
: ''
if (!username) throw new ClientError('No username provided.')
const user = await utils.db.table('users')
.where('username', username)
.first()
const password = typeof req.body.password === 'string'
? req.body.password.trim()
: ''
if (!password) throw new ClientError('No password provided.')
if (!user) throw new ClientError('Wrong credentials.', { statusCode: 403 })
const user = await utils.db.table('users')
.where('username', username)
.first()
if (user.enabled === false || user.enabled === 0) {
throw new ClientError('This account has been disabled.', { statusCode: 403 })
}
if (!user) throw new ClientError('Wrong credentials.', { statusCode: 403 })
const result = await bcrypt.compare(password, user.password)
if (result === false) {
throw new ClientError('Wrong credentials.', { statusCode: 403 })
} else {
await res.json({ success: true, token: user.token })
}
}).catch(next)
if (user.enabled === false || user.enabled === 0) {
throw new ClientError('This account has been disabled.', { statusCode: 403 })
}
const result = await bcrypt.compare(password, user.password)
if (result === false) {
throw new ClientError('Wrong credentials.', { statusCode: 403 })
} else {
return res.json({ success: true, token: user.token })
}
}
self.register = (req, res, next) => {
Promise.resolve().then(async () => {
if (config.enableUserAccounts === false) {
throw new ClientError('Registration is currently disabled.', { statusCode: 403 })
}
self.register = async (req, res) => {
// Parse POST body
req.body = await req.json()
const username = typeof req.body.username === 'string'
? req.body.username.trim()
: ''
if (username.length < self.user.min || username.length > self.user.max) {
throw new ClientError(`Username must have ${self.user.min}-${self.user.max} characters.`)
}
if (config.enableUserAccounts === false) {
throw new ClientError('Registration is currently disabled.', { statusCode: 403 })
}
const password = typeof req.body.password === 'string'
? req.body.password.trim()
: ''
if (password.length < self.pass.min || password.length > self.pass.max) {
throw new ClientError(`Password must have ${self.pass.min}-${self.pass.max} characters.`)
}
const username = typeof req.body.username === 'string'
? req.body.username.trim()
: ''
if (username.length < self.user.min || username.length > self.user.max) {
throw new ClientError(`Username must have ${self.user.min}-${self.user.max} characters.`)
}
const user = await utils.db.table('users')
.where('username', username)
.first()
const password = typeof req.body.password === 'string'
? req.body.password.trim()
: ''
if (password.length < self.pass.min || password.length > self.pass.max) {
throw new ClientError(`Password must have ${self.pass.min}-${self.pass.max} characters.`)
}
if (user) throw new ClientError('Username already exists.')
const user = await utils.db.table('users')
.where('username', username)
.first()
const hash = await bcrypt.hash(password, saltRounds)
if (user) throw new ClientError('Username already exists.')
const token = await tokens.generateUniqueToken()
if (!token) {
throw new ServerError('Failed to allocate a unique token. Try again?')
}
const hash = await bcrypt.hash(password, saltRounds)
await utils.db.table('users')
.insert({
username,
password: hash,
token,
enabled: 1,
permission: perms.permissions.user,
registration: Math.floor(Date.now() / 1000)
})
utils.invalidateStatsCache('users')
tokens.onHold.delete(token)
const token = await tokens.generateUniqueToken()
if (!token) {
throw new ServerError('Failed to allocate a unique token. Try again?')
}
await res.json({ success: true, token })
}).catch(next)
await utils.db.table('users')
.insert({
username,
password: hash,
token,
enabled: 1,
permission: perms.permissions.user,
registration: Math.floor(Date.now() / 1000)
})
utils.invalidateStatsCache('users')
tokens.onHold.delete(token)
return res.json({ success: true, token })
}
self.changePassword = (req, res, next) => {
Promise.resolve().then(async () => {
const user = await utils.authorize(req)
self.changePassword = async (req, res) => {
const user = await utils.authorize(req)
const password = typeof req.body.password === 'string'
? req.body.password.trim()
: ''
if (password.length < self.pass.min || password.length > self.pass.max) {
throw new ClientError(`Password must have ${self.pass.min}-${self.pass.max} characters.`)
}
// Parse POST body
req.body = await req.json()
const hash = await bcrypt.hash(password, saltRounds)
const password = typeof req.body.password === 'string'
? req.body.password.trim()
: ''
if (password.length < self.pass.min || password.length > self.pass.max) {
throw new ClientError(`Password must have ${self.pass.min}-${self.pass.max} characters.`)
}
await utils.db.table('users')
.where('id', user.id)
.update('password', hash)
const hash = await bcrypt.hash(password, saltRounds)
await res.json({ success: true })
}).catch(next)
await utils.db.table('users')
.where('id', user.id)
.update('password', hash)
return res.json({ success: true })
}
self.assertPermission = (user, target) => {
@ -141,238 +144,253 @@ self.assertPermission = (user, target) => {
}
}
self.createUser = (req, res, next) => {
Promise.resolve().then(async () => {
const user = await utils.authorize(req)
self.createUser = async (req, res) => {
const user = await utils.authorize(req)
const isadmin = perms.is(user, 'admin')
if (!isadmin) return res.status(403).end()
// Parse POST body
req.body = await req.json()
const username = typeof req.body.username === 'string'
? req.body.username.trim()
: ''
if (username.length < self.user.min || username.length > self.user.max) {
const isadmin = perms.is(user, 'admin')
if (!isadmin) return res.status(403).end()
const username = typeof req.body.username === 'string'
? req.body.username.trim()
: ''
if (username.length < self.user.min || username.length > self.user.max) {
throw new ClientError(`Username must have ${self.user.min}-${self.user.max} characters.`)
}
let password = typeof req.body.password === 'string'
? req.body.password.trim()
: ''
if (password.length) {
if (password.length < self.pass.min || password.length > self.pass.max) {
throw new ClientError(`Password must have ${self.pass.min}-${self.pass.max} characters.`)
}
} else {
password = randomstring.generate(self.pass.rand)
}
let group = req.body.group
let permission
if (group !== undefined) {
permission = perms.permissions[group]
if (typeof permission !== 'number' || permission < 0) {
group = 'user'
permission = perms.permissions.user
}
}
const exists = await utils.db.table('users')
.where('username', username)
.first()
if (exists) throw new ClientError('Username already exists.')
const hash = await bcrypt.hash(password, saltRounds)
const token = await tokens.generateUniqueToken()
if (!token) {
throw new ServerError('Failed to allocate a unique token. Try again?')
}
await utils.db.table('users')
.insert({
username,
password: hash,
token,
enabled: 1,
permission,
registration: Math.floor(Date.now() / 1000)
})
utils.invalidateStatsCache('users')
tokens.onHold.delete(token)
return res.json({ success: true, username, password, group })
}
self.editUser = async (req, res) => {
const user = await utils.authorize(req)
// Parse POST body, if required
req.body = req.body || await req.json()
const isadmin = perms.is(user, 'admin')
if (!isadmin) throw new ClientError('', { statusCode: 403 })
const id = parseInt(req.body.id)
if (isNaN(id)) throw new ClientError('No user specified.')
const target = await utils.db.table('users')
.where('id', id)
.first()
self.assertPermission(user, target)
const update = {}
if (req.body.username !== undefined) {
update.username = String(req.body.username).trim()
if (update.username.length < self.user.min || update.username.length > self.user.max) {
throw new ClientError(`Username must have ${self.user.min}-${self.user.max} characters.`)
}
}
let password = typeof req.body.password === 'string'
? req.body.password.trim()
: ''
if (password.length) {
if (password.length < self.pass.min || password.length > self.pass.max) {
throw new ClientError(`Password must have ${self.pass.min}-${self.pass.max} characters.`)
if (req.body.enabled !== undefined) {
update.enabled = Boolean(req.body.enabled)
}
if (req.body.group !== undefined) {
update.permission = perms.permissions[req.body.group]
if (typeof update.permission !== 'number' || update.permission < 0) {
update.permission = target.permission
}
}
let password
if (req.body.resetPassword) {
password = randomstring.generate(self.pass.rand)
update.password = await bcrypt.hash(password, saltRounds)
}
await utils.db.table('users')
.where('id', id)
.update(update)
utils.invalidateStatsCache('users')
const response = { success: true, update }
if (password) {
response.update.password = password
}
return res.json(response)
}
self.disableUser = async (req, res) => {
// Parse POST body and re-map for .editUser()
req.body = await req.json()
.then(obj => {
return {
id: obj.id,
enabled: false
}
})
return self.editUser(req, res)
}
self.deleteUser = async (req, res) => {
const user = await utils.authorize(req)
// Parse POST body
req.body = await req.json()
const isadmin = perms.is(user, 'admin')
if (!isadmin) throw new ClientError('', { statusCode: 403 })
const id = parseInt(req.body.id)
const purge = req.body.purge
if (isNaN(id)) throw new ClientError('No user specified.')
const target = await utils.db.table('users')
.where('id', id)
.first()
self.assertPermission(user, target)
const files = await utils.db.table('files')
.where('userid', id)
.select('id')
if (files.length) {
const fileids = files.map(file => file.id)
if (purge) {
const failed = await utils.bulkDeleteFromDb('id', fileids, user)
utils.invalidateStatsCache('uploads')
if (failed.length) {
return res.json({ success: false, failed })
}
} else {
password = randomstring.generate(self.pass.rand)
// Clear out userid attribute from the files
await utils.db.table('files')
.whereIn('id', fileids)
.update('userid', null)
}
}
let group = req.body.group
let permission
if (group !== undefined) {
permission = perms.permissions[group]
if (typeof permission !== 'number' || permission < 0) {
group = 'user'
permission = perms.permissions.user
}
}
const albums = await utils.db.table('albums')
.where('userid', id)
.where('enabled', 1)
.select('id', 'identifier')
const exists = await utils.db.table('users')
.where('username', username)
.first()
if (exists) throw new ClientError('Username already exists.')
const hash = await bcrypt.hash(password, saltRounds)
const token = await tokens.generateUniqueToken()
if (!token) {
throw new ServerError('Failed to allocate a unique token. Try again?')
}
await utils.db.table('users')
.insert({
username,
password: hash,
token,
enabled: 1,
permission,
registration: Math.floor(Date.now() / 1000)
})
utils.invalidateStatsCache('users')
tokens.onHold.delete(token)
await res.json({ success: true, username, password, group })
}).catch(next)
}
self.editUser = (req, res, next) => {
Promise.resolve().then(async () => {
const user = await utils.authorize(req)
const isadmin = perms.is(user, 'admin')
if (!isadmin) throw new ClientError('', { statusCode: 403 })
const id = parseInt(req.body.id)
if (isNaN(id)) throw new ClientError('No user specified.')
const target = await utils.db.table('users')
.where('id', id)
.first()
self.assertPermission(user, target)
const update = {}
if (req.body.username !== undefined) {
update.username = String(req.body.username).trim()
if (update.username.length < self.user.min || update.username.length > self.user.max) {
throw new ClientError(`Username must have ${self.user.min}-${self.user.max} characters.`)
}
}
if (req.body.enabled !== undefined) {
update.enabled = Boolean(req.body.enabled)
}
if (req.body.group !== undefined) {
update.permission = perms.permissions[req.body.group]
if (typeof update.permission !== 'number' || update.permission < 0) {
update.permission = target.permission
}
}
let password
if (req.body.resetPassword) {
password = randomstring.generate(self.pass.rand)
update.password = await bcrypt.hash(password, saltRounds)
}
await utils.db.table('users')
.where('id', id)
.update(update)
utils.invalidateStatsCache('users')
const response = { success: true, update }
if (password) response.update.password = password
await res.json(response)
}).catch(next)
}
self.disableUser = (req, res, next) => {
req.body = { id: req.body.id, enabled: false }
return self.editUser(req, res, next)
}
self.deleteUser = (req, res, next) => {
Promise.resolve().then(async () => {
const user = await utils.authorize(req)
const isadmin = perms.is(user, 'admin')
if (!isadmin) throw new ClientError('', { statusCode: 403 })
const id = parseInt(req.body.id)
const purge = req.body.purge
if (isNaN(id)) throw new ClientError('No user specified.')
const target = await utils.db.table('users')
.where('id', id)
.first()
self.assertPermission(user, target)
const files = await utils.db.table('files')
.where('userid', id)
.select('id')
if (files.length) {
const fileids = files.map(file => file.id)
if (purge) {
const failed = await utils.bulkDeleteFromDb('id', fileids, user)
if (failed.length) return res.json({ success: false, failed })
utils.invalidateStatsCache('uploads')
} else {
// Clear out userid attribute from the files
await utils.db.table('files')
.whereIn('id', fileids)
.update('userid', null)
}
}
const albums = await utils.db.table('albums')
.where('userid', id)
.where('enabled', 1)
.select('id', 'identifier')
if (albums.length) {
const albumids = albums.map(album => album.id)
await utils.db.table('albums')
.whereIn('id', albumids)
.del()
utils.deleteStoredAlbumRenders(albumids)
// Unlink their archives
await Promise.all(albums.map(async album => {
try {
await paths.unlink(path.join(paths.zips, `${album.identifier}.zip`))
} catch (error) {
// Re-throw non-ENOENT error
if (error.code !== 'ENOENT') throw error
}
}))
}
await utils.db.table('users')
.where('id', id)
if (albums.length) {
const albumids = albums.map(album => album.id)
await utils.db.table('albums')
.whereIn('id', albumids)
.del()
utils.invalidateStatsCache('users')
utils.deleteStoredAlbumRenders(albumids)
await res.json({ success: true })
}).catch(next)
// Unlink their archives
await Promise.all(albums.map(async album => {
try {
await paths.unlink(path.join(paths.zips, `${album.identifier}.zip`))
} catch (error) {
// Re-throw non-ENOENT error
if (error.code !== 'ENOENT') throw error
}
}))
}
await utils.db.table('users')
.where('id', id)
.del()
utils.invalidateStatsCache('users')
return res.json({ success: true })
}
self.bulkDeleteUsers = (req, res, next) => {
self.bulkDeleteUsers = async (req, res) => {
// TODO
}
self.listUsers = (req, res, next) => {
Promise.resolve().then(async () => {
const user = await utils.authorize(req)
self.listUsers = async (req, res) => {
const user = await utils.authorize(req)
const isadmin = perms.is(user, 'admin')
if (!isadmin) throw new ClientError('', { statusCode: 403 })
const isadmin = perms.is(user, 'admin')
if (!isadmin) throw new ClientError('', { statusCode: 403 })
const count = await utils.db.table('users')
.count('id as count')
.then(rows => rows[0].count)
if (!count) return res.json({ success: true, users: [], count })
const count = await utils.db.table('users')
.count('id as count')
.then(rows => rows[0].count)
if (!count) {
return res.json({ success: true, users: [], count })
}
let offset = Number(req.params.page)
if (isNaN(offset)) offset = 0
else if (offset < 0) offset = Math.max(0, Math.ceil(count / 25) + offset)
let offset = req.path_parameters && Number(req.path_parameters.page)
if (isNaN(offset)) offset = 0
else if (offset < 0) offset = Math.max(0, Math.ceil(count / 25) + offset)
const users = await utils.db.table('users')
.limit(25)
.offset(25 * offset)
.select('id', 'username', 'enabled', 'timestamp', 'permission', 'registration')
const users = await utils.db.table('users')
.limit(25)
.offset(25 * offset)
.select('id', 'username', 'enabled', 'timestamp', 'permission', 'registration')
const pointers = {}
for (const user of users) {
user.groups = perms.mapPermissions(user)
delete user.permission
user.uploads = 0
user.usage = 0
pointers[user.id] = user
}
const pointers = {}
for (const user of users) {
user.groups = perms.mapPermissions(user)
delete user.permission
user.uploads = 0
user.usage = 0
pointers[user.id] = user
}
const uploads = await utils.db.table('files')
.whereIn('userid', Object.keys(pointers))
.select('userid', 'size')
const uploads = await utils.db.table('files')
.whereIn('userid', Object.keys(pointers))
.select('userid', 'size')
for (const upload of uploads) {
pointers[upload.userid].uploads++
pointers[upload.userid].usage += parseInt(upload.size)
}
for (const upload of uploads) {
pointers[upload.userid].uploads++
pointers[upload.userid].usage += parseInt(upload.size)
}
await res.json({ success: true, users, count })
}).catch(next)
return res.json({ success: true, users, count })
}
module.exports = self

71
controllers/errorsController.js
View File

@ -11,56 +11,61 @@ const self = {
.map(key => Number(key))
}
self.handle = (error, req, res, next) => {
self.handleError = (req, res, error) => {
if (!res || res.headersSent) {
console.error('Unexpected missing "res" object or headers alredy sent.')
return console.trace()
logger.error('Error: Unexpected missing "res" object or headers alredy sent.')
return logger.error(error)
}
// Error messages that can be returned to users
res.header('Cache-Control', 'no-store')
// Errors that should be returned to users as JSON payload
const isClientError = error instanceof ClientError
const isServerError = error instanceof ServerError
const logStack = (!isClientError && !isServerError) ||
(isServerError && error.logStack)
if (logStack) {
logger.error(error)
}
let statusCode = res.statusCode
const statusCode = (isClientError || isServerError)
? error.statusCode
: 500
if (isClientError || isServerError) {
if (isServerError && error.logStack) {
logger.error(error)
}
const json = {}
const json = {
success: false,
description: error.message || 'An unexpected error occurred. Try again?',
code: error.code
}
const description = (isClientError || isServerError)
? error.message
: 'An unexpected error occurred. Try again?'
if (description) {
json.description = description
}
if (statusCode === undefined) {
res.status(error.statusCode || 500)
}
if ((isClientError || isServerError) && error.code) {
json.code = error.code
}
res.setHeader('Cache-Control', 'no-store')
if (Object.keys(json).length) {
json.success = false
return res.status(statusCode).json(json)
return res.json(json)
} else {
// Generic Errors
logger.error(error)
if (statusCode === undefined) {
statusCode = 500
}
if (self.errorPagesCodes.includes(statusCode)) {
return res.status(statusCode).sendFile(path.join(paths.errorRoot, config.errorPages[statusCode]))
return res
.status(statusCode)
.sendFile(path.join(paths.errorRoot, config.errorPages[statusCode]))
} else {
return res.status(statusCode).end()
return res
.status(statusCode)
.end()
}
}
}
self.handleMissing = (req, res, next) => {
res.setHeader('Cache-Control', 'no-store')
return res.status(404).sendFile(path.join(paths.errorRoot, config.errorPages[404]))
self.handleNotFound = (req, res) => {
res.header('Cache-Control', 'no-store')
return res
.status(404)
.sendFile(path.join(paths.errorRoot, config.errorPages[404]))
}
module.exports = self

43
controllers/handlers/apiErrorsHandler.js
View File

@ -1,43 +0,0 @@
const ClientError = require('./../utils/ClientError')
const ServerError = require('./../utils/ServerError')
const logger = require('./../../logger')
module.exports = (error, req, res, next) => {
if (!res) {
return logger.error(new Error('Missing "res" object.'))
}
// Error messages that can be returned to users
const isClientError = error instanceof ClientError
const isServerError = error instanceof ServerError
const logStack = (!isClientError && !isServerError) ||
(isServerError && error.logStack)
if (logStack) {
logger.error(error)
}
const statusCode = (isClientError || isServerError)
? error.statusCode
: 500
const json = {}
const description = (isClientError || isServerError)
? error.message
: 'An unexpected error occurred. Try again?'
if (description) {
json.description = description
}
if ((isClientError || isServerError) && error.code) {
json.code = error.code
}
if (Object.keys(json).length) {
json.success = false
return res.status(statusCode).json(json)
} else {
return res.status(statusCode).end()
}
}

283
controllers/handlers/serveStatic.js Normal file
View File

@ -0,0 +1,283 @@
const contentDisposition = require('content-disposition')
const etag = require('etag')
const fs = require('fs')
const parseRange = require('range-parser')
const path = require('path')
const SimpleDataStore = require('./../utils/SimpleDataStore')
const errors = require('./../errorsController')
const paths = require('./../pathsController')
const utils = require('./../utilsController')
const serveUtils = require('./../utils/serveUtils')
const logger = require('./../../logger')
class ServeStatic {
directory
contentDispositionStore
contentTypesMaps
setContentDisposition
setContentType
#options
constructor (directory, options = {}) {
if (!directory || typeof directory !== 'string') {
throw new TypeError('Root directory must be set')
}
this.directory = directory
if (options.acceptRanges === undefined) {
options.acceptRanges = true
}
if (options.etag === undefined) {
options.etag = true
}
if (options.ignorePatterns) {
if (!Array.isArray(options.ignorePatterns) || options.ignorePatterns.some(pattern => typeof pattern !== 'string')) {
throw new TypeError('Middleware option ignorePatterns must be an array of string')
}
}
if (options.lastModified === undefined) {
options.lastModified = true
}
if (options.setHeaders && typeof options.setHeaders !== 'function') {
throw new TypeError('Middleware option setHeaders must be a function')
}
// Init Content-Type overrides
if (typeof options.overrideContentTypes === 'object') {
this.contentTypesMaps = new Map()
const types = Object.keys(options.overrideContentTypes)
for (const type of types) {
const extensions = options.overrideContentTypes[type]
if (Array.isArray(extensions)) {
for (const extension of extensions) {
this.contentTypesMaps.set(extension, type)
}
}
}
if (this.contentTypesMaps.size) {
this.setContentType = (req, res) => {
// Do only if accessing files from uploads' root directory (i.e. not thumbs, etc.)
if (req.path.indexOf('/', 1) === -1) {
const name = req.path.substring(1)
const extname = utils.extname(name).substring(1)
const contentType = this.contentTypesMaps.get(extname)
if (contentType) {
res.header('Content-Type', contentType)
}
}
}
} else {
this.contentTypesMaps = undefined
}
}
// Init Content-Disposition store and setHeaders function if required
if (options.setContentDisposition) {
this.contentDispositionStore = new SimpleDataStore(
options.contentDispositionOptions || {
limit: 50,
strategy: SimpleDataStore.STRATEGIES[0]
}
)
this.setContentDisposition = async (req, res) => {
// Do only if accessing files from uploads' root directory (i.e. not thumbs, etc.)
if (req.path.indexOf('/', 1) !== -1) return
const name = req.path.substring(1)
try {
let original = this.contentDispositionStore.get(name)
if (original === undefined) {
this.contentDispositionStore.hold(name)
original = await utils.db.table('files')
.where('name', name)
.select('original')
.first()
.then(_file => {
this.contentDispositionStore.set(name, _file.original)
return _file.original
})
}
if (original) {
res.header('Content-Disposition', contentDisposition(original, { type: 'inline' }))
}
} catch (error) {
this.contentDispositionStore.delete(name)
logger.error(error)
}
}
logger.debug('Inititated SimpleDataStore for Content-Disposition: ' +
`{ limit: ${this.contentDispositionStore.limit}, strategy: "${this.contentDispositionStore.strategy}" }`)
}
this.#options = options
}
async #get (fullPath) {
const stat = await paths.stat(fullPath)
if (stat.isDirectory()) return
return stat
}
/*
* Based on https://github.com/pillarjs/send/blob/0.18.0/index.js
* Copyright(c) 2012 TJ Holowaychuk
* Copyright(c) 2014-2022 Douglas Christopher Wilson
* MIT Licensed
*/
async #handler (req, res) {
if (this.#options.ignorePatterns && this.#options.ignorePatterns.some(pattern => req.path.startsWith(pattern))) {
return errors.handleNotFound(req, res)
}
const fullPath = path.join(this.directory, req.path)
const stat = await this.#get(fullPath)
.catch(error => {
// Only re-throw errors if not due to missing files
if (error.code !== 'ENOENT') {
throw error
}
})
if (stat === undefined) {
return errors.handleNotFound(req, res)
}
// ReadStream options
let len = stat.size
const opts = {}
let ranges = req.headers.range
let offset = 0
// set content-type
res.type(req.path)
// set header fields
await this.#setHeaders(req, res, stat)
// conditional GET support
if (serveUtils.isConditionalGET(req)) {
if (serveUtils.isPreconditionFailure(req, res)) {
return res.status(412).end()
}
if (serveUtils.isFresh(req, res)) {
return res.status(304).end()
}
}
// adjust len to start/end options
len = Math.max(0, len - offset)
if (opts.end !== undefined) {
const bytes = opts.end - offset + 1
if (len > bytes) len = bytes
}
// Range support
if (this.#options.acceptRanges && serveUtils.BYTES_RANGE_REGEXP.test(ranges)) {
// parse
ranges = parseRange(len, ranges, {
combine: true
})
// If-Range support
if (!serveUtils.isRangeFresh(req, res)) {
// range stale
ranges = -2
}
// unsatisfiable
if (ranges === -1) {
// Content-Range
res.header('Content-Range', serveUtils.contentRange('bytes', len))
// 416 Requested Range Not Satisfiable
return res.status(416).end()
}
// valid (syntactically invalid/multiple ranges are treated as a regular response)
if (ranges !== -2 && ranges.length === 1) {
// Content-Range
res.status(206)
res.header('Content-Range', serveUtils.contentRange('bytes', len, ranges[0]))
// adjust for requested range
offset += ranges[0].start
len = ranges[0].end - ranges[0].start + 1
}
} else if (req.method === 'GET' && this.setContentDisposition) {
// Only set Content-Disposition on complete GET requests
// Range requests are typically when streaming
await this.setContentDisposition(req, res)
}
// set read options
opts.start = offset
opts.end = Math.max(offset, offset + len - 1)
// content-length
res.header('Content-Length', String(len))
// HEAD support
if (req.method === 'HEAD') {
return res.end()
}
return this.#stream(req, res, fullPath, opts)
}
async #setHeaders (req, res, stat) {
// Override Content-Type if required
if (this.setContentType) {
this.setContentType(req, res)
}
// Always do external setHeaders function first,
// in case it will overwrite the following default headers anyways
if (this.#options.setHeaders) {
this.#options.setHeaders(req, res)
}
if (this.#options.acceptRanges && !res.get('Accept-Ranges')) {
res.header('Accept-Ranges', 'bytes')
}
if (this.#options.lastModified && !res.get('Last-Modified')) {
const modified = stat.mtime.toUTCString()
res.header('Last-Modified', modified)
}
if (this.#options.etag && !res.get('ETag')) {
const val = etag(stat)
res.header('ETag', val)
}
}
async #stream (req, res, fullPath, opts) {
const readStream = fs.createReadStream(fullPath, opts)
readStream.on('error', error => {
readStream.destroy()
logger.error(error)
})
return res.stream(readStream)
}
get handler () {
return this.#handler.bind(this)
}
}
module.exports = ServeStatic

31
controllers/middlewares/expressCompat.js Normal file
View File

@ -0,0 +1,31 @@
// TODO: Currently consulting with author of hyper-express
// about the behavior of Response.get()/.getHeader() not matching ExpressJS
// https://github.com/kartikk221/hyper-express/discussions/97
// This middleware is a workaround, hopefully only temporarily
class ExpressCompat {
#getHeader (res, name) {
// Always return first value in array if it only has a single value
const values = res._getHeader(name)
if (Array.isArray(values) && values.length === 1) {
return values[0]
} else {
return values
}
}
#middleware (req, res, next) {
// Alias Response.get() and Response.getHeader() with a function that is more aligned with ExpressJS
res._get = res.get
res._getHeader = res.getHeader
res.get = res.getHeader = name => this.#getHeader(res, name)
return next()
}
get middleware () {
return this.#middleware.bind(this)
}
}
module.exports = ExpressCompat

60
controllers/middlewares/nunjucksRenderer.js Normal file
View File

@ -0,0 +1,60 @@
const nunjucks = require('nunjucks')
class NunjucksRenderer {
directory
environment
#persistentCaches = new Map()
constructor (directory = '', options = {}) {
if (typeof directory !== 'string') {
throw new TypeError('Root directory must be a string value')
}
this.directory = directory
this.environment = nunjucks.configure(
this.directory,
Object.assign(options, {
autoescape: true
})
)
}
#middleware (req, res, next) {
// Inject render() method into Response on each requests
// If usePersistentCache, the rendered template will be cached forever (thus only use for absolutely static pages)
res.render = (path, params, usePersistentCache) => this.#render(res, path, params, usePersistentCache)
return next()
}
#render (res, path, params, usePersistentCache) {
return new Promise((resolve, reject) => {
const template = `${path}.njk`
const cached = this.#persistentCaches.get(template)
if (usePersistentCache && cached) {
return resolve(cached)
}
this.environment.render(template, params, (err, html) => {
if (err) {
return reject(err)
}
if (usePersistentCache) {
this.#persistentCaches.set(template, html)
}
resolve(html)
})
}).then(html => {
res.type('html').send(html)
return html
})
}
get middleware () {
return this.#middleware.bind(this)
}
}
module.exports = NunjucksRenderer

60
controllers/middlewares/rateLimiter.js Normal file
View File

@ -0,0 +1,60 @@
const { RateLimiterMemory } = require('rate-limiter-flexible')
const ClientError = require('./../utils/ClientError')
class RateLimiter {
rateLimiterMemory
#requestKey
#whitelistedKeys
constructor (requestKey, options = {}, whitelistedKeys) {
if (typeof options.points !== 'number' || typeof options.duration !== 'number') {
throw new Error('Points and Duration must be set with numbers in options')
}
if (whitelistedKeys && typeof whitelistedKeys instanceof Set) {
throw new TypeError('Whitelisted keys must be a Set')
}
this.#requestKey = requestKey
this.#whitelistedKeys = new Set(whitelistedKeys)
this.rateLimiterMemory = new RateLimiterMemory(options)
}
#middleware (req, res, next) {
if (res.locals.rateLimit) {
return next()
}
// If unset, assume points pool is shared to all visitors of each route
const key = this.#requestKey ? req[this.#requestKey] : req.path
if (this.#whitelistedKeys.has(key)) {
// Set the Response local variable for earlier bypass in any subsequent RateLimit middlewares
res.locals.rateLimit = 'BYPASS'
return next()
}
// Always consume only 1 point
this.rateLimiterMemory.consume(key, 1)
.then(result => {
res.locals.rateLimit = result
res.header('Retry-After', String(result.msBeforeNext / 1000))
res.header('X-RateLimit-Limit', String(this.rateLimiterMemory._points))
res.header('X-RateLimit-Remaining', String(result.remainingPoints))
res.header('X-RateLimit-Reset', String(new Date(Date.now() + result.msBeforeNext)))
return next()
})
.catch(reject => {
// Re-throw with ClientError
return next(new ClientError('Rate limit reached, please try again in a while.', { statusCode: 429 }))
})
}
get middleware () {
return this.#middleware.bind(this)
}
}
module.exports = RateLimiter

104
controllers/middlewares/serveLiveDirectory.js Normal file
View File

@ -0,0 +1,104 @@
const LiveDirectory = require('live-directory')
const serveUtils = require('./../utils/serveUtils')
class ServeLiveDirectory {
instance
#options
constructor (instanceOptions = {}, options = {}) {
if (!instanceOptions.ignore) {
instanceOptions.ignore = path => {
// ignore dot files
return path.startsWith('.')
}
}
this.instance = new LiveDirectory(instanceOptions)
if (options.etag === undefined) {
options.etag = true
}
if (options.lastModified === undefined) {
options.lastModified = true
}
if (options.setHeaders && typeof options.setHeaders !== 'function') {
throw new TypeError('Middleware option setHeaders must be a function')
}
this.#options = options
}
/*
* Based on https://github.com/pillarjs/send/blob/0.18.0/index.js
* Copyright(c) 2012 TJ Holowaychuk
* Copyright(c) 2014-2022 Douglas Christopher Wilson
* MIT Licensed
*/
handler (req, res, file) {
// set header fields
this.#setHeaders(req, res, file)
// set content-type
res.type(file.extension)
// conditional GET support
if (serveUtils.isConditionalGET(req)) {
if (serveUtils.isPreconditionFailure(req, res)) {
return res.status(412).end()
}
if (serveUtils.isFresh(req, res)) {
return res.status(304).end()
}
}
// HEAD support
if (req.method === 'HEAD') {
return res.end()
}
return res.send(file.buffer)
}
#middleware (req, res, next) {
// Only process GET and HEAD requests
if (req.method !== 'GET' && req.method !== 'HEAD') {
return next()
}
const file = this.instance.get(req.path)
if (file === undefined) {
return next()
}
return this.handler(req, res, file)
}
#setHeaders (req, res, file) {
// Always do external setHeaders function first,
// in case it will overwrite the following default headers anyways
if (this.#options.setHeaders) {
this.#options.setHeaders(req, res)
}
if (this.#options.lastModified && !res.get('Last-Modified')) {
const modified = new Date(file.last_update).toUTCString()
res.header('Last-Modified', modified)
}
if (this.#options.etag && !res.get('ETag')) {
const val = file.etag
res.header('ETag', val)
}
}
get middleware () {
return this.#middleware.bind(this)
}
}
module.exports = ServeLiveDirectory

103
controllers/tokenController.js
View File

@ -33,72 +33,69 @@ self.generateUniqueToken = async () => {
return null
}
self.verify = (req, res, next) => {
Promise.resolve().then(async () => {
const token = typeof req.body.token === 'string'
? req.body.token.trim()
: ''
self.verify = async (req, res) => {
// Parse POST body
req.body = await req.json()
if (!token) throw new ClientError('No token provided.', { statusCode: 403 })
const token = typeof req.body.token === 'string'
? req.body.token.trim()
: ''
const user = await utils.db.table('users')
.where('token', token)
.select('username', 'permission')
.first()
if (!token) throw new ClientError('No token provided.', { statusCode: 403 })
if (!user) {
throw new ClientError('Invalid token.', { statusCode: 403, code: 10001 })
const user = await utils.db.table('users')
.where('token', token)
.select('username', 'permission')
.first()
if (!user) {
throw new ClientError('Invalid token.', { statusCode: 403, code: 10001 })
}
const obj = {
success: true,
username: user.username,
permissions: perms.mapPermissions(user)
}
const group = perms.group(user)
if (group) {
obj.group = group
if (utils.retentions.enabled) {
obj.retentionPeriods = utils.retentions.periods[group]
obj.defaultRetentionPeriod = utils.retentions.default[group]
}
}
const obj = {
success: true,
username: user.username,
permissions: perms.mapPermissions(user)
}
if (utils.clientVersion) {
obj.version = utils.clientVersion
}
const group = perms.group(user)
if (group) {
obj.group = group
if (utils.retentions.enabled) {
obj.retentionPeriods = utils.retentions.periods[group]
obj.defaultRetentionPeriod = utils.retentions.default[group]
}
}
if (utils.clientVersion) {
obj.version = utils.clientVersion
}
await res.json(obj)
}).catch(next)
return res.json(obj)
}
self.list = (req, res, next) => {
Promise.resolve().then(async () => {
const user = await utils.authorize(req)
await res.json({ success: true, token: user.token })
}).catch(next)
self.list = async (req, res) => {
const user = await utils.authorize(req)
return res.json({ success: true, token: user.token })
}
self.change = (req, res, next) => {
Promise.resolve().then(async () => {
const user = await utils.authorize(req, 'token')
self.change = async (req, res) => {
const user = await utils.authorize(req, 'token')
const newToken = await self.generateUniqueToken()
if (!newToken) {
throw new ServerError('Failed to allocate a unique token. Try again?')
}
const newToken = await self.generateUniqueToken()
if (!newToken) {
throw new ServerError('Failed to allocate a unique token. Try again?')
}
await utils.db.table('users')
.where('token', user.token)
.update({
token: newToken,
timestamp: Math.floor(Date.now() / 1000)
})
self.onHold.delete(newToken)
await utils.db.table('users')
.where('token', user.token)
.update({
token: newToken,
timestamp: Math.floor(Date.now() / 1000)
})
self.onHold.delete(newToken)
await res.json({ success: true, token: newToken })
}).catch(next)
return res.json({ success: true, token: newToken })
}
module.exports = self

2080
controllers/uploadController.js
View File

File diff suppressed because it is too large Load Diff

127
controllers/utils/multerStorage.js
View File

@ -1,127 +0,0 @@
const fs = require('fs')
const path = require('path')
const blake3 = require('blake3')
const mkdirp = require('mkdirp')
const logger = require('./../../logger')
const REQUIRED_WEIGHT = 2
function DiskStorage (opts) {
this.getFilename = opts.filename
if (typeof opts.destination === 'string') {
mkdirp.sync(opts.destination)
this.getDestination = function ($0, $1, cb) { cb(null, opts.destination) }
} else {
this.getDestination = opts.destination
}
this.scan = opts.scan
this.scanHelpers = opts.scanHelpers
}
DiskStorage.prototype._handleFile = function _handleFile (req, file, cb) {
const that = this
// "weighted" callback, to be able to "await" multiple callbacks
let tempError = null
let tempObject = {}
let tempWeight = 0
const _cb = (err = null, result = {}, weight = 2) => {
tempError = err
tempWeight += weight
tempObject = Object.assign(result, tempObject)
if (tempError || tempWeight >= REQUIRED_WEIGHT) {
cb(tempError, tempObject)
}
}
that.getDestination(req, file, function (err, destination) {
if (err) return _cb(err)
that.getFilename(req, file, function (err, filename) {
if (err) return _cb(err)
const finalPath = path.join(destination, filename)
const onerror = err => {
hash.dispose()
_cb(err)
}
let outStream
let hash
let scanStream
if (file._isChunk) {
if (!file._chunksData.stream) {
file._chunksData.stream = fs.createWriteStream(finalPath, { flags: 'a' })
file._chunksData.stream.on('error', onerror)
}
if (!file._chunksData.hasher) {
file._chunksData.hasher = blake3.createHash()
}
outStream = file._chunksData.stream
hash = file._chunksData.hasher
} else {
outStream = fs.createWriteStream(finalPath)
outStream.on('error', onerror)
hash = blake3.createHash()
if (that.scan.passthrough &&
!that.scanHelpers.assertUserBypass(req._user, filename) &&
!that.scanHelpers.assertFileBypass({ filename })) {
scanStream = that.scan.instance.passthrough()
}
}
file.stream.on('error', onerror)
file.stream.on('data', d => hash.update(d))
if (file._isChunk) {
file.stream.on('end', () => {
_cb(null, {
destination,
filename,
path: finalPath
})
})
file.stream.pipe(outStream, { end: false })
} else {
outStream.on('finish', () => {
_cb(null, {
destination,
filename,
path: finalPath,
size: outStream.bytesWritten,
hash: hash.digest('hex')
}, scanStream ? 1 : 2)
})
if (scanStream) {
logger.debug(`[ClamAV]: ${filename}: Passthrough scanning\u2026`)
scanStream.on('error', onerror)
scanStream.on('scan-complete', scan => {
_cb(null, { scan }, 1)
})
file.stream.pipe(scanStream).pipe(outStream)
} else {
file.stream.pipe(outStream)
}
}
})
})
}
DiskStorage.prototype._removeFile = function _removeFile (req, file, cb) {
const path = file.path
delete file.destination
delete file.filename
delete file.path
fs.unlink(path, cb)
}
module.exports = function (opts) {
return new DiskStorage(opts)
}

111
controllers/utils/serveUtils.js Normal file
View File

@ -0,0 +1,111 @@
const fresh = require('fresh')
const self = {
BYTES_RANGE_REGEXP: /^ *bytes=/
}
self.isFresh = (req, res) => {
return fresh(req.headers, {
etag: res.get('ETag'),
'last-modified': res.get('Last-Modified')
})
}
/*
* Based on https://github.com/pillarjs/send/blob/0.18.0/index.js
* Copyright(c) 2012 TJ Holowaychuk
* Copyright(c) 2014-2022 Douglas Christopher Wilson
* MIT Licensed
*/
self.isRangeFresh = (req, res) => {
const ifRange = req.headers['if-range']
if (!ifRange) {
return true
}
// if-range as etag
if (ifRange.indexOf('"') !== -1) {
const etag = res.get('ETag')
return Boolean(etag && ifRange.indexOf(etag) !== -1)
}
// if-range as modified date
const lastModified = res.get('Last-Modified')
return self.parseHttpDate(lastModified) <= self.parseHttpDate(ifRange)
}
self.isConditionalGET = req => {
return req.headers['if-match'] ||
req.headers['if-unmodified-since'] ||
req.headers['if-none-match'] ||
req.headers['if-modified-since']
}
self.isPreconditionFailure = (req, res) => {
// if-match
const match = req.headers['if-match']
if (match) {
const etag = res.get('ETag')
return !etag || (match !== '*' && self.parseTokenList(match).every(match => {
return match !== etag && match !== 'W/' + etag && 'W/' + match !== etag
}))
}
// if-unmodified-since
const unmodifiedSince = self.parseHttpDate(req.headers['if-unmodified-since'])
if (!isNaN(unmodifiedSince)) {
const lastModified = self.parseHttpDate(res.get('Last-Modified'))
return isNaN(lastModified) || lastModified > unmodifiedSince
}
return false
}
self.contentRange = (type, size, range) => {
return type + ' ' + (range ? range.start + '-' + range.end : '*') + '/' + size
}
self.parseHttpDate = date => {
const timestamp = date && Date.parse(date)
return typeof timestamp === 'number'
? timestamp
: NaN
}
self.parseTokenList = str => {
let end = 0
const list = []
let start = 0
// gather tokens
for (let i = 0, len = str.length; i < len; i++) {
switch (str.charCodeAt(i)) {
case 0x20: /* */
if (start === end) {
start = end = i + 1
}
break
case 0x2c: /* , */
if (start !== end) {
list.push(str.substring(start, end))
}
start = end = i + 1
break
default:
end = i + 1
break
}
}
// final token
if (start !== end) {
list.push(str.substring(start, end))
}
return list
}
module.exports = self

597
controllers/utilsController.js
View File

@ -225,38 +225,42 @@ const cloudflarePurgeCacheQueue = cloudflareAuth && fastq.promise(async chunk =>
logger.log(`${prefix}: ${message}`)
}
try {
const purge = await fetch(url, {
method: 'POST',
body: JSON.stringify({ files: chunk }),
headers
})
const response = await purge.json()
const response = await fetch(url, {
method: 'POST',
body: JSON.stringify({ files: chunk }),
headers
})
.then(res => res.json())
.catch(error => error)
const hasErrorsArray = Array.isArray(response.errors) && response.errors.length
if (hasErrorsArray) {
const rateLimit = response.errors.find(error => /rate limit/i.test(error.message))
if (rateLimit && i < MAX_TRIES - 1) {
_log(`${rateLimit.code}: ${rateLimit.message}. Retrying in a minute\u2026`)
await new Promise(resolve => setTimeout(resolve, 60000))
continue
}
}
result.success = response.success
result.errors = hasErrorsArray
? response.errors.map(error => `${error.code}: ${error.message}`)
: []
} catch (error) {
const errorString = error.toString()
// If fetch errors out, instead of API responding with API errors
if (response instanceof Error) {
const errorString = response.toString()
if (i < MAX_TRIES - 1) {
_log(`${errorString}. Retrying in 5 seconds\u2026`)
await new Promise(resolve => setTimeout(resolve, 5000))
continue
}
result.errors = [errorString]
break
}
// If API reponds with API errors
const hasErrorsArray = Array.isArray(response.errors) && response.errors.length
if (hasErrorsArray) {
const rateLimit = response.errors.find(error => /rate limit/i.test(error.message))
if (rateLimit && i < MAX_TRIES - 1) {
_log(`${rateLimit.code}: ${rateLimit.message}. Retrying in a minute\u2026`)
await new Promise(resolve => setTimeout(resolve, 60000))
continue
}
}
// If succeeds or out of retries
result.success = response.success
result.errors = hasErrorsArray
? response.errors.map(error => `${error.code}: ${error.message}`)
: []
break
}
@ -742,9 +746,10 @@ self.purgeCloudflareCache = async (names, uploads, thumbs) => {
}
const results = []
await Promise.all(chunks.map(async chunk =>
results.push(await cloudflarePurgeCacheQueue.push(chunk))
))
for (const chunk of chunks) {
const result = await cloudflarePurgeCacheQueue.push(chunk)
results.push(result)
}
return results
}
@ -784,320 +789,324 @@ self.invalidateStatsCache = type => {
statsData[type].cache = null
}
self.stats = (req, res, next) => {
Promise.resolve().then(async () => {
const user = await self.authorize(req)
const generateStats = async (req, res) => {
const user = await self.authorize(req)
const isadmin = perms.is(user, 'admin')
if (!isadmin) throw new ClientError('', { statusCode: 403 })
const isadmin = perms.is(user, 'admin')
if (!isadmin) throw new ClientError('', { statusCode: 403 })
const hrstart = process.hrtime()
const stats = {}
Object.keys(statsData).forEach(key => {
// Pre-assign object keys to fix their display order
stats[statsData[key].title] = {}
})
const hrstart = process.hrtime()
const stats = {}
Object.keys(statsData).forEach(key => {
// Pre-assign object keys to fix their display order
stats[statsData[key].title] = {}
})
const os = await si.osInfo()
const os = await si.osInfo()
const getSystemInfo = async () => {
const data = statsData.system
const getSystemInfo = async () => {
const data = statsData.system
if (!data.cache && data.generating) {
stats[data.title] = false
} else if (((Date.now() - data.generatedAt) <= 500) || data.generating) {
// Use cache for 500 ms (0.5 seconds)
stats[data.title] = data.cache
} else {
data.generating = true
data.generatedAt = Date.now()
if (!data.cache && data.generating) {
stats[data.title] = false
} else if (((Date.now() - data.generatedAt) <= 500) || data.generating) {
// Use cache for 500 ms (0.5 seconds)
stats[data.title] = data.cache
} else {
data.generating = true
data.generatedAt = Date.now()
const currentLoad = await si.currentLoad()
const mem = await si.mem()
const time = si.time()
const nodeUptime = process.uptime()
const currentLoad = await si.currentLoad()
const mem = await si.mem()
const time = si.time()
const nodeUptime = process.uptime()
if (self.scan.instance) {
try {
self.scan.version = await self.scan.instance.getVersion().then(s => s.trim())
} catch (error) {
logger.error(error)
self.scan.version = 'Errored when querying version.'
}
if (self.scan.instance) {
try {
self.scan.version = await self.scan.instance.getVersion().then(s => s.trim())
} catch (error) {
logger.error(error)
self.scan.version = 'Errored when querying version.'
}
stats[data.title] = {
Platform: `${os.platform} ${os.arch}`,
Distro: `${os.distro} ${os.release}`,
Kernel: os.kernel,
Scanner: self.scan.version || 'N/A',
'CPU Load': `${currentLoad.currentLoad.toFixed(1)}%`,
'CPUs Load': currentLoad.cpus.map(cpu => `${cpu.load.toFixed(1)}%`).join(', '),
'System Memory': {
value: {
used: mem.active,
total: mem.total
},
type: 'byteUsage'
},
'Memory Usage': {
value: process.memoryUsage().rss,
type: 'byte'
},
'System Uptime': {
value: Math.floor(time.uptime),
type: 'uptime'
},
'Node.js': `${process.versions.node}`,
'Service Uptime': {
value: Math.floor(nodeUptime),
type: 'uptime'
}
}
// Update cache
data.cache = stats[data.title]
data.generating = false
}
}
const getFileSystems = async () => {
const data = statsData.fileSystems
if (!data.cache && data.generating) {
stats[data.title] = false
} else if (((Date.now() - data.generatedAt) <= 60000) || data.generating) {
// Use cache for 60000 ms (60 seconds)
stats[data.title] = data.cache
} else {
data.generating = true
data.generatedAt = Date.now()
stats[data.title] = {}
const fsSize = await si.fsSize()
for (const fs of fsSize) {
const obj = {
value: {
total: fs.size,
used: fs.used
},
type: 'byteUsage'
}
// "available" is a new attribute in systeminformation v5, only tested on Linux,
// so add an if-check just in case its availability is limited in other platforms
if (typeof fs.available === 'number') {
obj.value.available = fs.available
}
stats[data.title][`${fs.fs} (${fs.type}) on ${fs.mount}`] = obj
stats[data.title] = {
Platform: `${os.platform} ${os.arch}`,
Distro: `${os.distro} ${os.release}`,
Kernel: os.kernel,
Scanner: self.scan.version || 'N/A',
'CPU Load': `${currentLoad.currentLoad.toFixed(1)}%`,
'CPUs Load': currentLoad.cpus.map(cpu => `${cpu.load.toFixed(1)}%`).join(', '),
'System Memory': {
value: {
used: mem.active,
total: mem.total
},
type: 'byteUsage'
},
'Memory Usage': {
value: process.memoryUsage().rss,
type: 'byte'
},
'System Uptime': {
value: Math.floor(time.uptime),
type: 'uptime'
},
'Node.js': `${process.versions.node}`,
'Service Uptime': {
value: Math.floor(nodeUptime),
type: 'uptime'
}
// Update cache
data.cache = stats[data.title]
data.generating = false
}
// Update cache
data.cache = stats[data.title]
data.generating = false
}
}
const getUploadsStats = async () => {
const data = statsData.uploads
const getFileSystems = async () => {
const data = statsData.fileSystems
if (!data.cache && data.generating) {
stats[data.title] = false
} else if (data.cache) {
// Cache will be invalidated with self.invalidateStatsCache() after any related operations
stats[data.title] = data.cache
} else {
data.generating = true
data.generatedAt = Date.now()
if (!data.cache && data.generating) {
stats[data.title] = false
} else if (((Date.now() - data.generatedAt) <= 60000) || data.generating) {
// Use cache for 60000 ms (60 seconds)
stats[data.title] = data.cache
} else {
data.generating = true
data.generatedAt = Date.now()
stats[data.title] = {
Total: 0,
Images: 0,
Videos: 0,
Audios: 0,
Others: 0,
Temporary: 0,
'Size in DB': {
value: 0,
type: 'byte'
}
stats[data.title] = {}
const fsSize = await si.fsSize()
for (const fs of fsSize) {
const obj = {
value: {
total: fs.size,
used: fs.used
},
type: 'byteUsage'
}
const getTotalCountAndSize = async () => {
const uploads = await self.db.table('files')
.select('size')
stats[data.title].Total = uploads.length
stats[data.title]['Size in DB'].value = uploads.reduce((acc, upload) => acc + parseInt(upload.size), 0)
// "available" is a new attribute in systeminformation v5, only tested on Linux,
// so add an if-check just in case its availability is limited in other platforms
if (typeof fs.available === 'number') {
obj.value.available = fs.available
}
stats[data.title][`${fs.fs} (${fs.type}) on ${fs.mount}`] = obj
}
const getImagesCount = async () => {
stats[data.title].Images = await self.db.table('files')
.where(function () {
for (const ext of self.imageExts) {
this.orWhere('name', 'like', `%${ext}`)
}
})
.count('id as count')
.then(rows => rows[0].count)
// Update cache
data.cache = stats[data.title]
data.generating = false
}
}
const getUploadsStats = async () => {
const data = statsData.uploads
if (!data.cache && data.generating) {
stats[data.title] = false
} else if (data.cache) {
// Cache will be invalidated with self.invalidateStatsCache() after any related operations
stats[data.title] = data.cache
} else {
data.generating = true
data.generatedAt = Date.now()
stats[data.title] = {
Total: 0,
Images: 0,
Videos: 0,
Audios: 0,
Others: 0,
Temporary: 0,
'Size in DB': {
value: 0,
type: 'byte'
}
}
const getVideosCount = async () => {
stats[data.title].Videos = await self.db.table('files')
.where(function () {
for (const ext of self.videoExts) {
this.orWhere('name', 'like', `%${ext}`)
}
})
.count('id as count')
.then(rows => rows[0].count)
}
const getTotalCountAndSize = async () => {
const uploads = await self.db.table('files')
.select('size')
stats[data.title].Total = uploads.length
stats[data.title]['Size in DB'].value = uploads.reduce((acc, upload) => acc + parseInt(upload.size), 0)
}
const getAudiosCount = async () => {
stats[data.title].Audios = await self.db.table('files')
.where(function () {
for (const ext of self.audioExts) {
this.orWhere('name', 'like', `%${ext}`)
}
})
.count('id as count')
.then(rows => rows[0].count)
}
const getImagesCount = async () => {
stats[data.title].Images = await self.db.table('files')
.where(function () {
for (const ext of self.imageExts) {
this.orWhere('name', 'like', `%${ext}`)
}
})
.count('id as count')
.then(rows => rows[0].count)
}
const getOthersCount = async () => {
stats[data.title].Temporary = await self.db.table('files')
.whereNotNull('expirydate')
.count('id as count')
.then(rows => rows[0].count)
}
const getVideosCount = async () => {
stats[data.title].Videos = await self.db.table('files')
.where(function () {
for (const ext of self.videoExts) {
this.orWhere('name', 'like', `%${ext}`)
}
})
.count('id as count')
.then(rows => rows[0].count)
}
await Promise.all([
getTotalCountAndSize(),
getImagesCount(),
getVideosCount(),
getAudiosCount(),
getOthersCount()
])
const getAudiosCount = async () => {
stats[data.title].Audios = await self.db.table('files')
.where(function () {
for (const ext of self.audioExts) {
this.orWhere('name', 'like', `%${ext}`)
}
})
.count('id as count')
.then(rows => rows[0].count)
}
stats[data.title].Others = stats[data.title].Total -
const getOthersCount = async () => {
stats[data.title].Temporary = await self.db.table('files')
.whereNotNull('expirydate')
.count('id as count')
.then(rows => rows[0].count)
}
await Promise.all([
getTotalCountAndSize(),
getImagesCount(),
getVideosCount(),
getAudiosCount(),
getOthersCount()
])
stats[data.title].Others = stats[data.title].Total -
stats[data.title].Images -
stats[data.title].Videos -
stats[data.title].Audios
// Update cache
data.cache = stats[data.title]
data.generating = false
}
// Update cache
data.cache = stats[data.title]
data.generating = false
}
}
const getUsersStats = async () => {
const data = statsData.users
const getUsersStats = async () => {
const data = statsData.users
if (!data.cache && data.generating) {
stats[data.title] = false
} else if (data.cache) {
// Cache will be invalidated with self.invalidateStatsCache() after any related operations
stats[data.title] = data.cache
} else {
data.generating = true
data.generatedAt = Date.now()
if (!data.cache && data.generating) {
stats[data.title] = false
} else if (data.cache) {
// Cache will be invalidated with self.invalidateStatsCache() after any related operations
stats[data.title] = data.cache
} else {
data.generating = true
data.generatedAt = Date.now()
stats[data.title] = {
Total: 0,
Disabled: 0
stats[data.title] = {
Total: 0,
Disabled: 0
}
const permissionKeys = Object.keys(perms.permissions).reverse()
permissionKeys.forEach(p => {
stats[data.title][p] = 0
})
const users = await self.db.table('users')
stats[data.title].Total = users.length
for (const user of users) {
if (user.enabled === false || user.enabled === 0) {
stats[data.title].Disabled++
}
const permissionKeys = Object.keys(perms.permissions).reverse()
permissionKeys.forEach(p => {
stats[data.title][p] = 0
})
const users = await self.db.table('users')
stats[data.title].Total = users.length
for (const user of users) {
if (user.enabled === false || user.enabled === 0) {
stats[data.title].Disabled++
}
user.permission = user.permission || 0
for (const p of permissionKeys) {
if (user.permission === perms.permissions[p]) {
stats[data.title][p]++
break
}
user.permission = user.permission || 0
for (const p of permissionKeys) {
if (user.permission === perms.permissions[p]) {
stats[data.title][p]++
break
}
}
// Update cache
data.cache = stats[data.title]
data.generating = false
}
// Update cache
data.cache = stats[data.title]
data.generating = false
}
}
const getAlbumsStats = async () => {
const data = statsData.albums
const getAlbumsStats = async () => {
const data = statsData.albums
if (!data.cache && data.generating) {
stats[data.title] = false
} else if (data.cache) {
// Cache will be invalidated with self.invalidateStatsCache() after any related operations
stats[data.title] = data.cache
} else {
data.generating = true
data.generatedAt = Date.now()
if (!data.cache && data.generating) {
stats[data.title] = false
} else if (data.cache) {
// Cache will be invalidated with self.invalidateStatsCache() after any related operations
stats[data.title] = data.cache
} else {
data.generating = true
data.generatedAt = Date.now()
stats[data.title] = {
Total: 0,
Disabled: 0,
Public: 0,
Downloadable: 0,
'ZIP Generated': 0
}
const albums = await self.db.table('albums')
stats[data.title].Total = albums.length
const activeAlbums = []
for (const album of albums) {
if (!album.enabled) {
stats[data.title].Disabled++
continue
}
activeAlbums.push(album.id)
if (album.download) stats[data.title].Downloadable++
if (album.public) stats[data.title].Public++
}
await paths.readdir(paths.zips).then(files => {
stats[data.title]['ZIP Generated'] = files.length
}).catch(() => {})
stats[data.title]['Files in albums'] = await self.db.table('files')
.whereIn('albumid', activeAlbums)
.count('id as count')
.then(rows => rows[0].count)
// Update cache
data.cache = stats[data.title]
data.generating = false
stats[data.title] = {
Total: 0,
Disabled: 0,
Public: 0,
Downloadable: 0,
'ZIP Generated': 0
}
const albums = await self.db.table('albums')
stats[data.title].Total = albums.length
const activeAlbums = []
for (const album of albums) {
if (!album.enabled) {
stats[data.title].Disabled++
continue
}
activeAlbums.push(album.id)
if (album.download) stats[data.title].Downloadable++
if (album.public) stats[data.title].Public++
}
await paths.readdir(paths.zips).then(files => {
stats[data.title]['ZIP Generated'] = files.length
}).catch(() => {})
stats[data.title]['Files in albums'] = await self.db.table('files')
.whereIn('albumid', activeAlbums)
.count('id as count')
.then(rows => rows[0].count)
// Update cache
data.cache = stats[data.title]
data.generating = false
}
}
await Promise.all([
getSystemInfo(),
getFileSystems(),
getUploadsStats(),
getUsersStats(),
getAlbumsStats()
])
await Promise.all([
getSystemInfo(),
getFileSystems(),
getUploadsStats(),
getUsersStats(),
getAlbumsStats()
])
return res.json({ success: true, stats, hrtime: process.hrtime(hrstart) })
}).catch(error => {
// Reset generating state when encountering any errors
Object.keys(statsData).forEach(key => {
statsData[key].generating = false
return res.json({ success: true, stats, hrtime: process.hrtime(hrstart) })
}
self.stats = async (req, res) => {
return generateStats(req, res)
.catch(error => {
logger.debug('caught generateStats() errors')
// Reset generating state when encountering any errors
Object.keys(statsData).forEach(key => {
statsData[key].generating = false
})
throw error
})
return next(error)
})
}
module.exports = self

12
docker/README.md
View File

@ -7,19 +7,19 @@ First make sure you have docker and docker composer installed, so please follow
- https://docs.docker.com/compose/install/
After that:
- Navigate to this directory (`docker`).
- Copy the config file called `docker-compose.config.example.yml` and name it `docker-compose.config.yml` with the values you want. Those that are left commented will use the default values.
- Copy either `lolisafe.tld.http.example.conf` or `lolisafe.tld.https.example.conf` and name it `lolisafe.tld.conf` for either HTTP or HTTPS
- - If using HTTPS make sure to put your certificates into the `ssl` folder and name them accordingly:
- - - `lolisafe.tld.crt` for the certificate
- - - `lolisafe.tld.key` for the certificate key
- Navigate to `nginx` directory, then copy either `lolisafe.tld.http.example.conf` or `lolisafe.tld.https.example.conf` and name it `lolisafe.tld.conf` for either HTTP or HTTPS.
If using HTTPS make sure to put your certificates into the `ssl` folder and name them accordingly:
- `lolisafe.tld.crt` for the certificate
- `lolisafe.tld.key` for the certificate key
Once you are done run the following commands:
- `cd docker`
- `./lolisafe.sh prod pull`
- `./lolisafe.sh prod build`
- `./lolisafe.sh prod up -d`
Use `./lolisafe.ps1` instead if you are on a Windows host.
If you are on a Windows host, replace `./lolisafe.sh` with `./lolisafe.ps1`.
Congrats, your lolisafe instance is now running.

5
docker/nginx/nginxconfig.io/proxy.conf
View File

@ -1,6 +1,11 @@
proxy_http_version 1.1;
proxy_cache_bypass $http_upgrade;
# Disable buffering of responses from the proxied server
proxy_buffering off;
# Disable buffering of client request body
proxy_request_buffering off;
# Proxy headers
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";

301
lolisafe.js
View File

@ -9,15 +9,10 @@ process.on('unhandledRejection', error => {
logger.error(error, { prefix: 'Unhandled Rejection (Promise): ' })
})
// Require libraries
const bodyParser = require('body-parser')
const contentDisposition = require('content-disposition')
const express = require('express')
// Libraries
const helmet = require('helmet')
const HyperExpress = require('hyper-express')
const NodeClam = require('clamscan')
const nunjucks = require('nunjucks')
const path = require('path')
const rateLimit = require('express-rate-limit')
const { accessSync, constants } = require('fs')
// Check required config files
@ -32,19 +27,31 @@ for (const _file of configFiles) {
}
}
// Require config files
// Config files
const config = require('./config')
const versions = require('./src/versions')
// lolisafe
logger.log('Starting lolisafe\u2026')
const safe = express()
const safe = new HyperExpress.Server({
trust_proxy: Boolean(config.trustProxy)
})
const errors = require('./controllers/errorsController')
const paths = require('./controllers/pathsController')
paths.initSync()
const utils = require('./controllers/utilsController')
// Middlewares
const ExpressCompat = require('./controllers/middlewares/expressCompat')
const NunjucksRenderer = require('./controllers/middlewares/nunjucksRenderer')
const RateLimiter = require('./controllers/middlewares/rateLimiter')
const ServeLiveDirectory = require('./controllers/middlewares/serveLiveDirectory')
// Handlers
const ServeStatic = require('./controllers/handlers/serveStatic')
// Routes
const album = require('./routes/album')
const api = require('./routes/api')
const file = require('./routes/file')
@ -53,6 +60,28 @@ const player = require('./routes/player')
const isDevMode = process.env.NODE_ENV === 'development'
// Express-compat
const expressCompatInstance = new ExpressCompat()
safe.use(expressCompatInstance.middleware)
// Rate limiters
if (Array.isArray(config.rateLimiters)) {
let whitelistedKeys
if (Array.isArray(config.rateLimitersWhitelist)) {
whitelistedKeys = new Set(config.rateLimitersWhitelist)
}
for (const rateLimit of config.rateLimiters) {
// Init RateLimiter using Request.ip as key
const rateLimiterInstance = new RateLimiter('ip', rateLimit.options, whitelistedKeys)
for (const route of rateLimit.routes) {
safe.use(route, rateLimiterInstance.middleware)
}
}
} else if (config.rateLimits) {
logger.error('Config option "rateLimits" is deprecated.')
logger.error('Please consult the provided sample file for the new option "rateLimiters".')
}
// Helmet security headers
if (config.helmet instanceof Object) {
// If an empty object, simply do not use Helmet
@ -83,7 +112,7 @@ if (config.accessControlAllowOrigin) {
config.accessControlAllowOrigin = '*'
}
safe.use((req, res, next) => {
res.set('Access-Control-Allow-Origin', config.accessControlAllowOrigin)
res.header('Access-Control-Allow-Origin', config.accessControlAllowOrigin)
if (config.accessControlAllowOrigin !== '*') {
res.vary('Origin')
}
@ -91,103 +120,20 @@ if (config.accessControlAllowOrigin) {
})
}
if (config.trustProxy) {
safe.set('trust proxy', 1)
}
// https://mozilla.github.io/nunjucks/api.html#configure
nunjucks.configure('views', {
autoescape: true,
express: safe,
// NunjucksRenderer middleware
const nunjucksRendererInstance = new NunjucksRenderer('views', {
watch: isDevMode
// noCache: isDevMode
})
safe.set('view engine', 'njk')
safe.enable('view cache')
safe.use(nunjucksRendererInstance.middleware)
// Configure rate limits (disabled during development)
if (!isDevMode && Array.isArray(config.rateLimits) && config.rateLimits.length) {
for (const _rateLimit of config.rateLimits) {
const limiter = rateLimit(_rateLimit.config)
for (const route of _rateLimit.routes) {
safe.use(route, limiter)
}
}
}
// Array of routes to apply CDN Cache-Control onto,
// and additionally call Cloudflare API to have their CDN caches purged when lolisafe starts
const cdnRoutes = [...config.pages]
safe.use(bodyParser.urlencoded({ extended: true }))
safe.use(bodyParser.json())
// Defaults to no-op
let setHeadersForStaticAssets = () => {}
const cdnPages = [...config.pages]
let setHeaders
const contentTypes = typeof config.overrideContentTypes === 'object' &&
Object.keys(config.overrideContentTypes)
const overrideContentTypes = contentTypes && contentTypes.length && function (res, path) {
// Do only if accessing files from uploads' root directory (i.e. not thumbs, etc.)
const relpath = path.replace(paths.uploads, '')
if (relpath.indexOf('/', 1) === -1) {
const name = relpath.substring(1)
const extname = utils.extname(name).substring(1)
for (const contentType of contentTypes) {
if (config.overrideContentTypes[contentType].includes(extname)) {
res.set('Content-Type', contentType)
break
}
}
}
}
const initServeStaticUploads = (opts = {}) => {
if (config.setContentDisposition) {
const SimpleDataStore = require('./controllers/utils/SimpleDataStore')
utils.contentDispositionStore = new SimpleDataStore(
config.contentDispositionOptions || {
limit: 50,
strategy: SimpleDataStore.STRATEGIES[0]
}
)
opts.preSetHeaders = async (res, req, path, stat) => {
// Do only if accessing files from uploads' root directory (i.e. not thumbs, etc.),
// AND only if GET requests
const relpath = path.replace(paths.uploads, '')
if (relpath.indexOf('/', 1) !== -1 || req.method !== 'GET') return
const name = relpath.substring(1)
try {
let original = utils.contentDispositionStore.get(name)
if (original === undefined) {
utils.contentDispositionStore.hold(name)
original = await utils.db.table('files')
.where('name', name)
.select('original')
.first()
.then(_file => {
utils.contentDispositionStore.set(name, _file.original)
return _file.original
})
}
if (original) {
res.set('Content-Disposition', contentDisposition(original, { type: 'inline' }))
}
} catch (error) {
utils.contentDispositionStore.delete(name)
logger.error(error)
}
}
// serveStatic is provided with @bobbywibowo/serve-static, a fork of express/serve-static.
// The fork allows specifying an async function by the name preSetHeaders,
// which it will await before creating 'send' stream to client.
// This is necessary due to database queries being async tasks,
// and express/serve-static not having the functionality by default.
safe.use('/', require('@bobbywibowo/serve-static')(paths.uploads, opts))
logger.debug('Inititated SimpleDataStore for Content-Disposition: ' +
`{ limit: ${utils.contentDispositionStore.limit}, strategy: "${utils.contentDispositionStore.strategy}" }`)
} else {
safe.use('/', express.static(paths.uploads, opts))
}
}
// Cache control (safe.fiery.me)
// Cache control
if (config.cacheControl) {
const cacheControls = {
// max-age: 6 months
@ -201,82 +147,72 @@ if (config.cacheControl) {
}
// By default soft cache everything
safe.use('/', (req, res, next) => {
res.set('Cache-Control', cacheControls.validate)
next()
safe.use((req, res, next) => {
res.header('Cache-Control', cacheControls.validate)
return next()
})
switch (config.cacheControl) {
case 1:
case true:
// If using CDN, cache public pages in CDN
cdnPages.push('api/check')
for (const page of cdnPages) {
safe.get(`/${page === 'home' ? '' : page}`, (req, res, next) => {
res.set('Cache-Control', cacheControls.cdn)
next()
})
}
// If using CDN, cache most front-end pages in CDN
// Include /api/check since it will only reply with persistent JSON payload
// that will not change, unless config file is edited and lolisafe is then restarted
cdnRoutes.push('api/check')
safe.use((req, res, next) => {
if (req.method === 'GET' || req.method === 'HEAD') {
const page = req.path === '/' ? 'home' : req.path.substring(1)
if (cdnRoutes.includes(page)) {
res.header('Cache-Control', cacheControls.cdn)
}
}
return next()
})
break
}
// If serving uploads with node
if (config.serveFilesWithNode) {
initServeStaticUploads({
setHeaders: (res, path) => {
// Override Content-Type header if necessary
if (overrideContentTypes) {
overrideContentTypes(res, path)
}
// If using CDN, cache uploads in CDN as well
// Use with cloudflare.purgeCache enabled in config file
if (config.cacheControl !== 2) {
res.set('Cache-Control', cacheControls.cdn)
}
}
})
}
// Function for static assets.
// This requires the assets to use version in their query string,
// as they will be cached by clients for a very long time.
setHeaders = res => {
res.set('Cache-Control', cacheControls.static)
setHeadersForStaticAssets = (req, res) => {
res.header('Cache-Control', cacheControls.static)
}
// Consider album ZIPs static as well, since they use version in their query string
safe.use(['/api/album/zip'], (req, res, next) => {
const versionString = parseInt(req.query.v)
safe.use('/api/album/zip', (req, res, next) => {
const versionString = parseInt(req.query_parameters.v)
if (versionString > 0) {
res.set('Cache-Control', cacheControls.static)
res.header('Cache-Control', cacheControls.static)
} else {
res.set('Cache-Control', cacheControls.disable)
res.header('Cache-Control', cacheControls.disable)
}
next()
return next()
})
} else if (config.serveFilesWithNode) {
const opts = {}
// Override Content-Type header if necessary
if (overrideContentTypes) {
opts.setHeaders = overrideContentTypes
}
initServeStaticUploads(opts)
}
// Static assets
safe.use('/', express.static(paths.public, { setHeaders }))
safe.use('/', express.static(paths.dist, { setHeaders }))
// Init LiveDirectory middlewares for static assets
// Static assets in /public directory
const serveLiveDirectoryPublicInstance = new ServeLiveDirectory({ path: paths.public }, {
setHeaders: setHeadersForStaticAssets
})
safe.use(serveLiveDirectoryPublicInstance.middleware)
// Static assets in /dist directory
const serveLiveDirectoryDistInstance = new ServeLiveDirectory({ path: paths.dist }, {
setHeaders: setHeadersForStaticAssets
})
safe.use(serveLiveDirectoryDistInstance.middleware)
safe.use('/', album)
safe.use('/', file)
safe.use('/', nojs)
safe.use('/', player)
// Routes
safe.use(album)
safe.use(file)
safe.use(nojs)
safe.use(player)
safe.use('/api', api)
;(async () => {
try {
// Init database
await require('./controllers/utils/initDatabase.js')(utils.db)
await require('./controllers/utils/initDatabase')(utils.db)
// Purge any leftover in chunks directory, do not wait
paths.purgeChunks()
@ -297,30 +233,55 @@ safe.use('/api', api)
}
}
const serveLiveDirectoryCustomPagesInstance = new ServeLiveDirectory({
path: paths.customPages,
keep: ['.html']
})
// Cookie Policy
if (config.cookiePolicy) {
config.pages.push('cookiepolicy')
}
// Check for custom pages, otherwise fallback to Nunjucks templates
for (const page of config.pages) {
const customPage = path.join(paths.customPages, `${page}.html`)
if (!await paths.access(customPage).catch(() => true)) {
safe.get(`/${page === 'home' ? '' : page}`, (req, res, next) => res.sendFile(customPage))
} else if (page === 'home') {
safe.get('/', (req, res, next) => res.render(page, {
config, utils, versions: utils.versionStrings
}))
} else {
safe.get(`/${page}`, (req, res, next) => res.render(page, {
config, utils, versions: utils.versionStrings
}))
// Front-end pages middleware
// HTML files in customPages directory can also override any built-in pages,
// if they have matching names with the routes (e.g. home.html can override the homepage)
// Aside from that, due to using LiveDirectory,
// custom pages can be added/removed on the fly while lolisafe is running
safe.use((req, res, next) => {
if (req.method === 'GET' || req.method === 'HEAD') {
const page = req.path === '/' ? 'home' : req.path.substring(1)
const customPage = serveLiveDirectoryCustomPagesInstance.instance.get(`${page}.html`)
if (customPage) {
return serveLiveDirectoryCustomPagesInstance.handler(req, res, customPage)
} else if (config.pages.includes(page)) {
// These rendered pages are persistently cached during production
return res.render(page, {
config, utils, versions: utils.versionStrings
}, !isDevMode)
}
}
return next()
})
// Init ServerStatic last if serving uploaded files with node
if (config.serveFilesWithNode) {
const serveStaticInstance = new ServeStatic(paths.uploads, {
contentDispositionOptions: config.contentDispositionOptions,
ignorePatterns: [
'/chunks/'
],
overrideContentTypes: config.overrideContentTypes,
setContentDisposition: config.setContentDisposition
})
safe.get('/*', serveStaticInstance.handler)
safe.head('/*', serveStaticInstance.handler)
utils.contentDispositionStore = serveStaticInstance.contentDispositionStore
}
// Express error handlers
safe.use(errors.handleMissing)
safe.use(errors.handle)
// Web server error handlers (must always be set after all routes/middlewares)
safe.set_not_found_handler(errors.handleNotFound)
safe.set_error_handler(errors.handleError)
// Git hash
if (config.showGitHash) {
@ -355,7 +316,7 @@ safe.use('/api', api)
}
// Binds Express to port
await new Promise(resolve => safe.listen(utils.conf.port, () => resolve()))
await safe.listen(utils.conf.port)
logger.log(`lolisafe started on port ${utils.conf.port}`)
// Cache control (safe.fiery.me)
@ -363,7 +324,7 @@ safe.use('/api', api)
if (config.cacheControl && config.cacheControl !== 2) {
if (config.cloudflare.purgeCache) {
logger.log('Cache control enabled, purging Cloudflare\'s cache...')
const results = await utils.purgeCloudflareCache(cdnPages)
const results = await utils.purgeCloudflareCache(cdnRoutes)
let errored = false
let succeeded = 0
for (const result of results) {

13
package.json
View File

@ -10,7 +10,7 @@
"url": "https://github.com/BobbyWibowo/lolisafe/issues"
},
"engines": {
"node": ">=12.22.0"
"node": ">=14.0.0"
},
"license": "MIT",
"scripts": {
@ -34,25 +34,26 @@
"full-upgrade": "rm -f ./yarn.lock && yarn"
},
"dependencies": {
"@bobbywibowo/serve-static": "git+https://git@github.com/BobbyWibowo/serve-static#8030f1b3cb17cf75b69d7ead8db3e879f6fc992f",
"bcrypt": "~5.0.1",
"better-sqlite3": "~7.6.1",
"blake3": "~2.1.7",
"body-parser": "~1.20.0",
"clamscan": "~2.1.2",
"content-disposition": "~0.5.4",
"express": "~4.18.1",
"express-rate-limit": "~6.4.0",
"etag": "~1.8.1",
"fastq": "~1.13.0",
"fluent-ffmpeg": "~2.1.2",
"fresh": "~0.5.2",
"helmet": "~5.1.0",
"hyper-express": "~6.4.1",
"jszip": "~3.10.0",
"knex": "~2.1.0",
"live-directory": "~2.3.2",
"markdown-it": "~13.0.1",
"multer": "~1.4.5-lts.1",
"node-fetch": "~2.6.7",
"nunjucks": "~3.2.3",
"randomstring": "~1.2.2",
"range-parser": "~1.2.1",
"rate-limiter-flexible": "~2.3.7",
"search-query-parser": "~1.6.0",
"sharp": "~0.30.7",
"systeminformation": "~5.12.1"

43
routes/album.js
View File

@ -1,13 +1,14 @@
const routes = require('express').Router()
const { Router } = require('hyper-express')
const routes = new Router()
const path = require('path')
const paths = require('./../controllers/pathsController')
const errors = require('./../controllers/errorsController')
const utils = require('./../controllers/utilsController')
const config = require('./../config')
routes.get('/a/:identifier', async (req, res, next) => {
const identifier = req.params.identifier
routes.get('/a/:identifier', async (req, res) => {
const identifier = req.path_parameters && req.path_parameters.identifier
if (identifier === undefined) {
res.status(404).sendFile(path.join(paths.errorRoot, config.errorPages[404]))
return errors.handleNotFound(req, res)
}
const album = await utils.db.table('albums')
@ -19,10 +20,10 @@ routes.get('/a/:identifier', async (req, res, next) => {
.first()
if (!album || album.public === 0) {
return res.status(404).sendFile(path.join(paths.errorRoot, config.errorPages[404]))
return errors.handleNotFound(req, res)
}
const nojs = req.query.nojs !== undefined
const nojs = req.query_parameters.nojs !== undefined
let cacheid
if (process.env.NODE_ENV !== 'development') {
@ -31,7 +32,7 @@ routes.get('/a/:identifier', async (req, res, next) => {
const cache = utils.albumRenderStore.get(cacheid)
if (cache) {
return res.send(cache)
return res.type('html').send(cache)
} else if (cache === null) {
return res.render('album-notice', {
config,
@ -74,27 +75,25 @@ routes.get('/a/:identifier', async (req, res, next) => {
? utils.md.instance.render(album.description)
: null
return res.render('album', {
// This will already end the Response,
// thus may only continue with tasks that will not interface with Response any further
const html = await res.render('album', {
config,
utils,
versions: utils.versionStrings,
album,
files,
nojs
}, (error, html) => {
const data = error ? null : html
if (cacheid) {
// Only store rendered page if it did not error out and album actually have files
if (data && files.length) {
utils.albumRenderStore.set(cacheid, data)
} else {
utils.albumRenderStore.delete(cacheid)
}
}
// Express should already send error to the next handler
if (!error) return res.send(data)
})
if (cacheid) {
// Only store rendered page if it did not error out and album actually have files
if (html && files.length) {
utils.albumRenderStore.set(cacheid, html)
} else {
utils.albumRenderStore.delete(cacheid)
}
}
})
module.exports = routes

82
routes/api.js
View File

@ -1,4 +1,5 @@
const routes = require('express').Router()
const { Router } = require('hyper-express')
const routes = new Router()
const albumsController = require('./../controllers/albumsController')
const authController = require('./../controllers/authController')
const tokenController = require('./../controllers/tokenController')
@ -6,7 +7,7 @@ const uploadController = require('./../controllers/uploadController')
const utilsController = require('./../controllers/utilsController')
const config = require('./../config')
routes.get('/check', (req, res, next) => {
routes.get('/check', async (req, res) => {
const obj = {
private: config.private,
enableUserAccounts: config.enableUserAccounts,
@ -22,44 +23,49 @@ routes.get('/check', (req, res, next) => {
if (utilsController.clientVersion) {
obj.version = utilsController.clientVersion
}
return res.json(obj)
})
routes.post('/login', (req, res, next) => authController.verify(req, res, next))
routes.post('/register', (req, res, next) => authController.register(req, res, next))
routes.post('/password/change', (req, res, next) => authController.changePassword(req, res, next))
routes.get('/uploads', (req, res, next) => uploadController.list(req, res, next))
routes.get('/uploads/:page', (req, res, next) => uploadController.list(req, res, next))
routes.post('/upload', (req, res, next) => uploadController.upload(req, res, next))
routes.post('/upload/delete', (req, res, next) => uploadController.delete(req, res, next))
routes.post('/upload/bulkdelete', (req, res, next) => uploadController.bulkDelete(req, res, next))
routes.post('/upload/finishchunks', (req, res, next) => uploadController.finishChunks(req, res, next))
routes.get('/upload/get/:identifier', (req, res, next) => uploadController.get(req, res, next))
routes.post('/upload/:albumid', (req, res, next) => uploadController.upload(req, res, next))
routes.get('/album/get/:identifier', (req, res, next) => albumsController.get(req, res, next))
routes.get('/album/zip/:identifier', (req, res, next) => albumsController.generateZip(req, res, next))
routes.get('/album/:id', (req, res, next) => albumsController.listFiles(req, res, next))
routes.get('/album/:id/:page', (req, res, next) => albumsController.listFiles(req, res, next))
routes.get('/albums', (req, res, next) => albumsController.list(req, res, next))
routes.get('/albums/:page', (req, res, next) => albumsController.list(req, res, next))
routes.post('/albums', (req, res, next) => albumsController.create(req, res, next))
routes.post('/albums/addfiles', (req, res, next) => albumsController.addFiles(req, res, next))
routes.post('/albums/delete', (req, res, next) => albumsController.delete(req, res, next))
routes.post('/albums/disable', (req, res, next) => albumsController.disable(req, res, next))
routes.post('/albums/edit', (req, res, next) => albumsController.edit(req, res, next))
routes.post('/albums/rename', (req, res, next) => albumsController.rename(req, res, next))
routes.get('/albums/test', (req, res, next) => albumsController.test(req, res, next))
routes.get('/tokens', (req, res, next) => tokenController.list(req, res, next))
routes.post('/tokens/verify', (req, res, next) => tokenController.verify(req, res, next))
routes.post('/tokens/change', (req, res, next) => tokenController.change(req, res, next))
routes.get('/filelength/config', (req, res, next) => authController.getFileLengthConfig(req, res, next))
routes.post('/filelength/change', (req, res, next) => authController.changeFileLength(req, res, next))
routes.get('/users', (req, res, next) => authController.listUsers(req, res, next))
routes.get('/users/:page', (req, res, next) => authController.listUsers(req, res, next))
routes.post('/users/create', (req, res, next) => authController.createUser(req, res, next))
routes.post('/users/edit', (req, res, next) => authController.editUser(req, res, next))
routes.post('/users/disable', (req, res, next) => authController.disableUser(req, res, next))
routes.post('/users/delete', (req, res, next) => authController.deleteUser(req, res, next))
routes.get('/stats', (req, res, next) => utilsController.stats(req, res, next))
routes.post('/login', authController.verify)
routes.post('/register', authController.register)
routes.post('/password/change', authController.changePassword)
routes.get('/uploads', uploadController.list)
routes.get('/uploads/:page', uploadController.list)
routes.post('/upload', uploadController.upload, {
// HyperExpress defaults to 250kb
// https://github.com/kartikk221/hyper-express/blob/6.2.4/docs/Server.md#server-constructor-options
max_body_length: parseInt(config.uploads.maxSize) * 1e6
})
routes.post('/upload/delete', uploadController.delete)
routes.post('/upload/bulkdelete', uploadController.bulkDelete)
routes.post('/upload/finishchunks', uploadController.finishChunks)
routes.get('/upload/get/:identifier', uploadController.get)
routes.post('/upload/:albumid', uploadController.upload)
routes.get('/album/get/:identifier', albumsController.get)
routes.get('/album/zip/:identifier', albumsController.generateZip)
routes.get('/album/:identifier', albumsController.getUpstreamCompat)
routes.get('/album/:albumid/:page', uploadController.list)
routes.get('/albums', albumsController.list)
routes.get('/albums/:page', albumsController.list)
routes.post('/albums', albumsController.create)
routes.post('/albums/addfiles', albumsController.addFiles)
routes.post('/albums/delete', albumsController.delete)
routes.post('/albums/disable', albumsController.disable)
routes.post('/albums/edit', albumsController.edit)
routes.post('/albums/rename', albumsController.rename)
routes.get('/albums/test', albumsController.test)
routes.get('/tokens', tokenController.list)
routes.post('/tokens/verify', tokenController.verify)
routes.post('/tokens/change', tokenController.change)
routes.get('/filelength/config', authController.getFileLengthConfig)
routes.post('/filelength/change', authController.changeFileLength)
routes.get('/users', authController.listUsers)
routes.get('/users/:page', authController.listUsers)
routes.post('/users/create', authController.createUser)
routes.post('/users/edit', authController.editUser)
routes.post('/users/disable', authController.disableUser)
routes.post('/users/delete', authController.deleteUser)
routes.get('/stats', utilsController.stats)
module.exports = routes

11
routes/file.js
View File

@ -1,10 +1,9 @@
const routes = require('express').Router()
const utils = require('../controllers/utilsController')
const config = require('../config')
const { Router } = require('hyper-express')
const routes = new Router()
const utils = require('./../controllers/utilsController')
const config = require('./../config')
routes.get([
'/file/:identifier'
], async (req, res, next) => {
routes.get('/file/:identifier', async (req, res) => {
// Uploads identifiers parsing, etc., are strictly handled by client-side JS at src/js/file.js
return res.render('file', {
config, utils, versions: utils.versionStrings

13
routes/nojs.js
View File

@ -1,9 +1,10 @@
const routes = require('express').Router()
const { Router } = require('hyper-express')
const routes = new Router()
const uploadController = require('./../controllers/uploadController')
const utils = require('./../controllers/utilsController')
const config = require('./../config')
routes.get('/nojs', async (req, res, next) => {
routes.get('/nojs', async (req, res) => {
return res.render('nojs', {
config,
utils,
@ -11,7 +12,7 @@ routes.get('/nojs', async (req, res, next) => {
})
})
routes.post('/nojs', (req, res, next) => {
routes.post('/nojs', async (req, res) => {
res._json = res.json
res.json = (...args) => {
const result = args[0]
@ -23,7 +24,11 @@ routes.post('/nojs', (req, res, next) => {
files: result.files || [{}]
})
}
return uploadController.upload(req, res, next)
return uploadController.upload(req, res)
}, {
// HyperExpress defaults to 250kb
// https://github.com/kartikk221/hyper-express/blob/6.2.4/docs/Server.md#server-constructor-options
max_body_length: parseInt(config.uploads.maxSize) * 1e6
})
module.exports = routes

13
routes/player.js
View File

@ -1,15 +1,16 @@
const routes = require('express').Router()
const { Router } = require('hyper-express')
const routes = new Router()
const utils = require('./../controllers/utilsController')
const config = require('./../config')
routes.get([
'/player/:identifier',
'/v/:identifier'
], async (req, res, next) => {
const playerHandler = async (req, res) => {
// Uploads identifiers parsing, etc., are strictly handled by client-side JS at src/js/player.js
return res.render('player', {
config, utils, versions: utils.versionStrings
})
})
}
routes.get('/player/:identifier', playerHandler)
routes.get('/v/:identifier', playerHandler)
module.exports = routes

4
scripts/clean-up.js
View File

@ -1,6 +1,6 @@
const path = require('path')
const paths = require('../controllers/pathsController')
const utils = require('../controllers/utilsController')
const paths = require('./../controllers/pathsController')
const utils = require('./../controllers/utilsController')
const self = {
getFiles: async directory => {

2
scripts/delete-expired.js
View File

@ -1,4 +1,4 @@
const utils = require('../controllers/utilsController')
const utils = require('./../controllers/utilsController')
;(async () => {
const location = process.argv[1].replace(process.cwd() + '/', '')

4
scripts/rebuild-hashes.js
View File

@ -1,8 +1,8 @@
const blake3 = require('blake3')
const fs = require('fs')
const path = require('path')
const paths = require('../controllers/pathsController')
const utils = require('../controllers/utilsController')
const paths = require('./../controllers/pathsController')
const utils = require('./../controllers/utilsController')
;(async () => {
const location = process.argv[1].replace(process.cwd() + '/', '')

4
scripts/thumbs.js
View File

@ -1,6 +1,6 @@
const path = require('path')
const paths = require('../controllers/pathsController')
const utils = require('../controllers/utilsController')
const paths = require('./../controllers/pathsController')
const utils = require('./../controllers/utilsController')
const self = {
mode: null,

3
views/faq.njk
View File

@ -120,7 +120,6 @@
<div class="container has-text-left">
<h2 id="technical" class='title is-spaced'>Technical</h2>
{% if not globals.is_for_personal_use -%}
<h3 class="subtitle has-text-white-ter">What are the allowed extensions here?</h3>
<article class="message">
<div class="message-body">
@ -140,6 +139,8 @@
</div>
</article>
{% if not globals.is_for_personal_use -%}
{# lolisafe does not do this by default, enable only if you specifically do so #}
<h3 class="subtitle has-text-white-ter">Why are my <strong>.htm/.html</strong> uploads being served as plain text?</h3>
<article class="message">
<div class="message-body">

2014
yarn.lock
View File

File diff suppressed because it is too large Load Diff