Mirror/filesafe
1
0
Fork 0
mirror of https://github.com/BobbyWibowo/lolisafe.git synced 2025-01-18 17:21:33 +00:00
Code Issues Actions Packages Projects Releases Wiki Activity

Updates [!! update config.js !!]

Browse Source 
Added extended support for URL uploads.
Namely URL proxy support and separate extensions filter (as in separate
from the primary extensions filter).
There's also a new option to set a disclaimer message that will be
printed underneath the URL uploads form.

Trust proxy is now toggleable from the configuration file.
I think they should only be enabled when you're behind proxy such as
Cloudflare or Incapsula.
I'm not sure how it behaves with only a bare nginx reverse proxy though.

Empty files can now be filtered.

Sorted preset extensions filter in config.sample.js.

Rephrased some options in config.sample.js as well.

maxTries now default to 3 in config.sample.js.

Various other small changes.
This commit is contained in:
Bobby Wibowo 2018-12-20 18:53:37 +07:00
parent bba6836708
commit d723c0f562
No known key found for this signature in database
GPG Key ID: 51C3A1E1E22D26CF
6 changed files with 208 additions and 94 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

113
config.sample.js
View File

@ -30,7 +30,7 @@ module.exports = {
/*
If you are serving your files with a different domain than your lolisafe homepage,
then fill this option with your lolisafe homepage, otherwise leave it null (or other falsy value).
then fill this option with your lolisafe homepage, otherwise any falsy value.
This will be used when listing album links in the dashboard.
*/
homeDomain: null,
@ -46,30 +46,30 @@ module.exports = {
pages: ['home', 'auth', 'dashboard', 'faq'],
/*
If set to true, all extensions in "extensionsFilter" array will be blacklisted,
otherwise only files with those extensions that can be uploaded.
This can be either 'blacklist' or 'whitelist', which should be self-explanatory.
When this is set to neither, this will fallback to 'blacklist'.
*/
filterBlacklist: true,
extensionsFilterMode: 'blacklist',
extensionsFilter: [
'.jar',
'.bash_profile',
'.bash',
'.bashrc',
'.bat',
'.bsh',
'.cmd',
'.com',
'.csh',
'.exe',
'.exec',
'.jar',
'.msi',
'.com',
'.bat',
'.cmd',
'.nt',
'.scr',
'.profile',
'.ps1',
'.psm1',
'.sh',
'.bash',
'.bsh',
'.csh',
'.bash_profile',
'.bashrc',
'.profile'
'.scr',
'.sh'
],
/*
@ -77,6 +77,13 @@ module.exports = {
*/
filterNoExtension: false,
/*
If set to true, files with zero bytes size will always be rejected.
NOTE: Even if the files only contain whitespaces, as long as they aren't
zero bytes, they will be accepted.
*/
filterEmptyFile: true,
/*
Show hash of the current git commit in homepage.
*/
@ -84,7 +91,7 @@ module.exports = {
/*
Path to error pages. Only 404 and 500 will be used.
Note: rootDir can either be relative or absolute path.
NOTE: rootDir can either be relative or absolute path.
*/
errorPages: {
rootDir: './pages/error',
@ -92,18 +99,24 @@ module.exports = {
500: '500.html'
},
/*
Trust proxy.
Only enable if you are running this behind a proxy like Cloudflare, Incapsula, etc.
*/
trustProxy: true,
/*
Uploads config.
*/
uploads: {
/*
Folder where images should be stored.
Folder where files should be stored.
*/
folder: 'uploads',
/*
Max file size allowed. Needs to be in MB.
Note: When maxSize is greater than 1 MiB and using nginx as reverse proxy,
NOTE: When maxSize is greater than 1 MiB and using nginx as reverse proxy,
you must set client_max_body_size to the same as maxSize.
https://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size
*/
@ -111,10 +124,58 @@ module.exports = {
/*
Max file size allowed for upload by URLs. Needs to be in MB.
NOTE: Set to falsy value (false, null, etc.) to disable upload by URLs.
NOTE: Set to falsy value to disable upload by URLs.
*/
urlMaxSize: '32MB',
/*
Proxy URL uploads.
NOTE: Set to falsy value to disable.
Available templates:
{url} = full URL (encoded & with protocol)
{url-noprot} = URL without protocol (images.weserv.nl prefers this format)
Example:
https://images.weserv.nl/?url={url-noprot}
will become:
https://images.weserv.nl/?url=example.com/assets/image.png
*/
urlProxy: 'https://images.weserv.nl/?url={url-noprot}',
/*
Disclaimer message that will be printed in the URL uploads form.
Supports HTML. Be safe though.
*/
urlDisclaimerMessage: 'URL uploads are being proxied and compressed by <a href="https://images.weserv.nl/" target="_blank" rel="noopener">images.weserv.nl</a>. By using this feature, you agree to their <a href="https://github.com/weserv/images/blob/4.x/Privacy-Policy.md" target="_blank" rel="noopener">Privacy Policy</a>.',
/*
Filter mode for URL uploads.
Can be 'blacklist', 'whitelist', or 'inherit'.
'inherit' => inherit primary extensions filter (extensionsFilter option).
The rest are paired with urlExtensionsFilter option below and should be self-explanatory.
When this is not set to any of the 3 values, this will fallback to 'inherit'.
*/
urlExtensionsFilterMode: 'whitelist',
/*
An array of extensions that are allowed for URL uploads.
Intented for URL proxies that only support certain extensions.
This will parse the extensions from the URLs, so URLs that do not end with
the file's extensions will always be rejected
Queries and segments in the URLs will be bypassed when parsing.
NOTE: Set to falsy value to disable filters.
*/
urlExtensionsFilter: [
'.gif',
'.jpg',
'.jpeg',
'.png',
'.bmp',
'.xbm',
'.webp'
],
/*
Scan files using ClamAV through clamd.
*/
@ -130,7 +191,7 @@ module.exports = {
by the size specified in "chunkSize". People will still be able to upload bigger files with
the API as long as they don't surpass the limit specified in the "maxSize" option above.
Total size of the whole chunks will also later be checked against the "maxSize" option.
NOTE: Set to falsy value (false, null, etc.) to disable chunked uploads.
NOTE: Set to falsy value to disable chunked uploads.
*/
chunkSize: '10MB',
@ -184,13 +245,13 @@ module.exports = {
/*
This option will limit how many times it will try to
generate a new random name when a collision occurrs.
The shorter the length is, the higher the chance for a collision to occur.
Generally, the shorter the length is, the higher the chance for a collision to occur.
This applies to both file name and album identifier.
*/
maxTries: 1,
maxTries: 3,
/*
Thumbnails are only for the dashboard.
Thumbnails are only used in the dashboard and album's public pages.
You need to install a separate binary called ffmpeg (https://ffmpeg.org/) for video thumbnails.
*/
generateThumbs: {
@ -214,7 +275,7 @@ module.exports = {
No-JS uploader page will not chunk the uploads, so it's recommended to change this
into the maximum upload size you have in Cloudflare.
This limit will only be applied to the subtitle in the page.
NOTE: Set to falsy value (false, null, etc.) to inherit "maxSize" option.
NOTE: Set to falsy value to inherit "maxSize" option.
*/
noJsMaxSize: '100MB',
@ -223,7 +284,7 @@ module.exports = {
API route (homeDomain/api/album/zip/*), with this option you can limit the
maximum total size of files in an album that can be zipped.
Cloudflare will not cache files bigger than 512MB.
NOTE: Set to falsy value (false, null, etc.) to disable max total size.
NOTE: Set to falsy value to disable max total size.
*/
zipMaxTotalSize: '512MB',

97
controllers/uploadController.js
View File

@ -72,27 +72,34 @@ const upload = multer({
delete req.body[key]
}
if (req.body.chunkindex)
if (chunkedUploads && parseInt(req.body.totalfilesize) > maxSizeBytes) {
// This will not be true if "totalfilesize" key does not exist, since "NaN > number" is false.
// eslint-disable-next-line standard/no-callback-literal
return cb('Chunk error occurred. Total file size is larger than the maximum file size.')
} else if (!chunkedUploads) {
if (req.body.chunkindex) {
if (!chunkedUploads)
// eslint-disable-next-line standard/no-callback-literal
return cb('Chunked uploads are disabled at the moment.')
const totalfilesize = parseInt(req.body.totalfilesize)
if (!isNaN(totalfilesize)) {
if (config.filterEmptyFile && totalfilesize === 0)
// eslint-disable-next-line standard/no-callback-literal
return cb('Empty files are not allowed.')
if (totalfilesize > maxSizeBytes)
// eslint-disable-next-line standard/no-callback-literal
return cb('Chunk error occurred. Total file size is larger than the maximum file size.')
}
}
return cb(null, true)
}
}).array('files[]')
uploadsController.isExtensionFiltered = extname => {
// If empty extension needs to be filtered
if (!extname && config.filterNoExtension) return true
// If there are extensions that have to be filtered
if (extname && config.extensionsFilter && config.extensionsFilter.length) {
if (extname && Array.isArray(config.extensionsFilter) && config.extensionsFilter.length) {
const match = config.extensionsFilter.some(extension => extname === extension.toLowerCase())
if ((config.filterBlacklist && match) || (!config.filterBlacklist && !match))
return true
const whitelist = config.extensionsFilterMode === 'whitelist'
if ((!whitelist && match) || (whitelist && !match)) return true
}
return false
}
@ -195,6 +202,13 @@ uploadsController.actuallyUpload = async (req, res, user, albumid) => {
}
})
if (config.filterEmptyFile && infoMap.some(file => file.data.size === 0)) {
infoMap.forEach(file => {
utils.deleteFile(file.data.filename, req.app.get('uploads-set')).catch(console.error)
})
return erred('Empty files are not allowed.')
}
if (config.uploads.scan && config.uploads.scan.enabled) {
const scan = await uploadsController.scanFiles(req, infoMap)
if (scan) return erred(scan)
@ -225,11 +239,29 @@ uploadsController.actuallyUploadByUrl = async (req, res, user, albumid) => {
let iteration = 0
const infoMap = []
for (const url of urls) {
for (let url of urls) {
const original = path.basename(url).split(/[?#]/)[0]
const extension = utils.extname(original)
if (uploadsController.isExtensionFiltered(extension))
return erred(`${extension.substr(1).toUpperCase()} files are not permitted due to security reasons.`)
const extname = utils.extname(original)
// Extensions filter
let filtered = false
if (['blacklist', 'whitelist'].includes(config.uploads.urlExtensionsFilterMode))
if (Array.isArray(config.uploads.urlExtensionsFilter) && config.uploads.urlExtensionsFilter.length) {
const match = config.uploads.urlExtensionsFilter.some(extension => extname === extension.toLowerCase())
const whitelist = config.uploads.urlExtensionsFilterMode === 'whitelist'
filtered = ((!whitelist && match) || (whitelist && !match))
} else {
return erred('config.uploads.urlExtensionsFilter is not an array or is an empty array, please contact site owner.')
}
else filtered = uploadsController.isExtensionFiltered(extname)
if (filtered)
return erred(`${extname ? `${extname.substr(1).toUpperCase()} files` : 'Files with no extension'} are not permitted due to security reasons.`)
if (config.uploads.urlProxy)
url = config.uploads.urlProxy
.replace(/{url}/g, encodeURIComponent(url))
.replace(/{url-noprot}/g, encodeURIComponent(url.replace(/^https?:\/\//, '')))
try {
const fetchHead = await fetch(url, { method: 'HEAD' })
@ -244,7 +276,10 @@ uploadsController.actuallyUploadByUrl = async (req, res, user, albumid) => {
if (size > urlMaxSizeBytes)
return erred('File too large.')
// limit max response body size with content-length
if (config.filterEmptyFile && size === 0)
return erred('Empty files are not allowed.')
// Limit max response body size with the size reported by Content-Length
const fetchFile = await fetch(url, { size })
if (fetchFile.status !== 200)
return erred(`${fetchHead.status} ${fetchHead.statusText}`)
@ -252,7 +287,7 @@ uploadsController.actuallyUploadByUrl = async (req, res, user, albumid) => {
const file = await fetchFile.buffer()
const length = uploadsController.getFileNameLength(req)
const name = await uploadsController.getUniqueRandomName(length, extension, req.app.get('uploads-set'))
const name = await uploadsController.getUniqueRandomName(length, extname, req.app.get('uploads-set'))
const destination = path.join(uploadsDir, name)
fs.writeFile(destination, file, async error => {
@ -342,12 +377,12 @@ uploadsController.actuallyFinishChunks = async (req, res, user, albumid) => {
}
if (file.count < chunkNames.length) return erred('Chunks count mismatch.')
const extension = typeof file.original === 'string' ? utils.extname(file.original) : ''
if (uploadsController.isExtensionFiltered(extension))
return erred(`${extension.substr(1).toUpperCase()} files are not permitted due to security reasons.`)
const extname = typeof file.original === 'string' ? utils.extname(file.original) : ''
if (uploadsController.isExtensionFiltered(extname))
return erred(`${extname ? `${extname.substr(1).toUpperCase()} files` : 'Files with no extension'} are not permitted due to security reasons.`)
const length = uploadsController.getFileNameLength(req)
const name = await uploadsController.getUniqueRandomName(length, extension, req.app.get('uploads-set'))
const name = await uploadsController.getUniqueRandomName(length, extname, req.app.get('uploads-set'))
.catch(erred)
if (!name) return
@ -360,12 +395,19 @@ uploadsController.actuallyFinishChunks = async (req, res, user, albumid) => {
const chunksTotalSize = await uploadsController.getTotalSize(uuidDir, chunkNames)
.catch(erred)
if (typeof chunksTotalSize !== 'number') return
if (chunksTotalSize > maxSizeBytes) {
const isEmpty = config.filterEmptyFile && (chunksTotalSize === 0)
const isBigger = chunksTotalSize > maxSizeBytes
if (isEmpty || isBigger) {
// Delete all chunks and remove chunks dir
const chunksCleaned = await uploadsController.cleanUpChunks(uuidDir, chunkNames)
.catch(erred)
if (!chunksCleaned) return
return erred(`Total chunks size is bigger than ${maxSize}.`)
if (isEmpty)
return erred('Empty files are not allowed.')
else
return erred(`Total chunks size is bigger than ${maxSize}.`)
}
// Append all chunks
@ -523,13 +565,7 @@ uploadsController.formatInfoMap = (req, res, user, infoMap) => {
timestamp: Math.floor(Date.now() / 1000)
})
} else {
utils.deleteFile(info.data.filename).catch(console.error)
const set = req.app.get('uploads-set')
if (set) {
const identifier = info.data.filename.split('.')[0]
set.delete(identifier)
// console.log(`Removed ${identifier} from identifiers cache (formatInfoMap)`)
}
utils.deleteFile(info.data.filename, req.app.get('uploads-set')).catch(console.error)
existingFiles.push(dbFile)
}
@ -548,7 +584,7 @@ uploadsController.scanFiles = (req, infoMap) => {
for (const info of infoMap)
scanner.scanFile(info.path).then(reply => {
if (!reply.includes('OK') || reply.includes('FOUND')) {
// eslint-disable-next-line no-control-regex
// eslint-disable-next-line no-control-regex
const virus = reply.replace(/^stream: /, '').replace(/ FOUND\u0000$/, '')
console.log(`ClamAV: ${info.data.filename}: ${virus} FOUND.`)
return resolve(virus)
@ -696,9 +732,8 @@ uploadsController.list = async (req, res) => {
// Only push usernames if we are a moderator
if (all && ismoderator)
if (file.userid !== undefined && file.userid !== null && file.userid !== '') {
if (file.userid !== undefined && file.userid !== null && file.userid !== '')
userids.push(file.userid)
}
file.extname = utils.extname(file.name)
if (utils.mayGenerateThumb(file.extname))

15
controllers/utilsController.js
View File

@ -197,14 +197,19 @@ utilsController.generateThumbs = (name, force) => {
})
}
utilsController.deleteFile = file => {
utilsController.deleteFile = (filename, set) => {
return new Promise((resolve, reject) => {
const extname = utilsController.extname(file)
return fs.unlink(path.join(uploadsDir, file), error => {
const extname = utilsController.extname(filename)
return fs.unlink(path.join(uploadsDir, filename), error => {
if (error && error.code !== 'ENOENT') return reject(error)
const identifier = filename.split('.')[0]
// eslint-disable-next-line curly
if (set) {
set.delete(identifier)
// console.log(`Removed ${identifier} from identifiers cache (deleteFile)`)
}
if (utilsController.imageExtensions.includes(extname) || utilsController.videoExtensions.includes(extname)) {
const thumb = file.substr(0, file.lastIndexOf('.')) + '.png'
const thumb = `${identifier}.png`
return fs.unlink(path.join(thumbsDir, thumb), error => {
if (error && error.code !== 'ENOENT') return reject(error)
resolve(true)

59
lolisafe.js
View File

@ -31,7 +31,7 @@ fs.existsSync(`./${config.uploads.folder}/thumbs`) || fs.mkdirSync(`./${config.u
fs.existsSync(`./${config.uploads.folder}/zips`) || fs.mkdirSync(`./${config.uploads.folder}/zips`)
safe.use(helmet())
safe.set('trust proxy', 1)
if (config.trustProxy) safe.set('trust proxy', 1)
// https://mozilla.github.io/nunjucks/api.html#configure
nunjucks.configure('views', {
@ -64,31 +64,38 @@ safe.use('/', album)
safe.use('/', nojs)
safe.use('/api', api)
for (const page of config.pages)
if (fs.existsSync(`./pages/custom/${page}.html`)) {
safe.get(`/${page}`, (req, res, next) => res.sendFile(`${page}.html`, {
root: './pages/custom/'
}))
} else if (page === 'home') {
safe.get('/', (req, res, next) => res.render('home', {
maxSize: config.uploads.maxSize,
urlMaxSize: config.uploads.urlMaxSize,
gitHash: safe.get('git-hash'),
urlDuckDuckGoProxy: config.uploads.urlDuckDuckGoProxy
}))
} else if (page === 'faq') {
const fileLength = config.uploads.fileLength
safe.get('/faq', (req, res, next) => res.render('faq', {
filterBlacklist: config.filterBlacklist,
extensionsFilter: config.extensionsFilter,
fileLength,
tooShort: (fileLength.max - fileLength.default) > (fileLength.default - fileLength.min),
noJsMaxSize: parseInt(config.cloudflare.noJsMaxSize) < parseInt(config.uploads.maxSize),
chunkSize: config.uploads.chunkSize
}))
} else {
safe.get(`/${page}`, (req, res, next) => res.render(page))
}
if (Array.isArray(config.pages) && config.pages.length) {
for (const page of config.pages)
if (fs.existsSync(`./pages/custom/${page}.html`)) {
safe.get(`/${page}`, (req, res, next) => res.sendFile(`${page}.html`, {
root: './pages/custom/'
}))
} else if (page === 'home') {
safe.get('/', (req, res, next) => res.render('home', {
maxSize: config.uploads.maxSize,
urlMaxSize: config.uploads.urlMaxSize,
urlDisclaimerMessage: config.uploads.urlDisclaimerMessage,
urlExtensionsFilterMode: config.uploads.urlExtensionsFilterMode,
urlExtensionsFilter: config.uploads.urlExtensionsFilter,
gitHash: safe.get('git-hash')
}))
} else if (page === 'faq') {
const fileLength = config.uploads.fileLength
safe.get('/faq', (req, res, next) => res.render('faq', {
whitelist: config.extensionsFilterMode === 'whitelist',
extensionsFilter: config.extensionsFilter,
fileLength,
tooShort: (fileLength.max - fileLength.default) > (fileLength.default - fileLength.min),
noJsMaxSize: parseInt(config.cloudflare.noJsMaxSize) < parseInt(config.uploads.maxSize),
chunkSize: config.uploads.chunkSize
}))
} else {
safe.get(`/${page}`, (req, res, next) => res.render(page))
}
} else {
console.error('config.pages is not an array or is an empty array. This won\t do!')
process.exit(1)
}
safe.use((req, res, next) => {
res.status(404).sendFile(config.errorPages[404], { root: config.errorPages.rootDir })

4
views/faq.njk
View File

@ -87,9 +87,9 @@
<h2 class='subtitle'>What are the allowed extensions here?</h2>
<article class="message">
<div class="message-body">
{% if extensionsFilter.length and filterBlacklist -%}
{% if extensionsFilter.length and not whitelist -%}
We support any file extensions except the following: {{ extensionsFilter | join(', ') }}.
{%- elif extensionsFilter.length and not filterBlacklist -%}
{%- elif extensionsFilter.length and whitelist -%}
We only support the following extensions: {{ extensionsFilter | join(', ') }}.
{%- else -%}
We support any file extensions.

14
views/home.njk
View File

@ -78,10 +78,16 @@
{% if urlMaxSize !== maxSize -%}
Maximum file size for URL upload is <span style="font-weight: bold">{{ urlMaxSize }}</span>.
{%- endif %}
{%- if urlMaxSize and urlMaxSize !== maxSize and urlDuckDuckGoProxy %}<br>{% endif -%}
{%- if urlDuckDuckGoProxy %}
Since we're using DuckDuckGo's proxy, the URLs have to be direct links.
{% endif -%}
{% if urlExtensionsFilter.length and (urlExtensionsFilterMode === 'blacklist') -%}
Blacklisted extensions: {{ urlExtensionsFilter | join(', ') }}.
{%- elif urlExtensionsFilter.length and (urlExtensionsFilterMode === 'whitelist') -%}
Whitelisted extensions: {{ urlExtensionsFilter | join(', ') }}.
{%- endif %}
{%- if urlDisclaimerMessage %}
{{ urlDisclaimerMessage | safe }}
{% endif -%}
</p>
</div>
<div class="field">