Commit Graph

490 Commits

Author SHA1 Message Date
rr-
c9cb9aa539 server/password-reset: try to construct full URL 2018-07-08 10:10:06 +02:00
rr-
d85e746a65 server/tests: fix failing info api tests 2018-07-08 09:42:13 +02:00
rr-
b6a5be74cf config: fix camelCase 2018-07-08 09:38:41 +02:00
Shyam Sunder
60ab9246c6 client: improved build.js, use relative links
* Removed unnecessary require('config.js') calls
* 'markdown.js' now uses rel. links in EntityPermalinkWrapper
* 'password_reset.py' now generates rel. links
* Removed 'Base URL' config parameter
* Removed 'API URL' config parameter
* 'build.js' no longer reads/requires config.yaml
* Updated documentation
* Removed unnecessary node packages used in 'build.js'

abandon api_url parameter
2018-07-06 19:40:20 +02:00
Shyam Sunder
3972b902d8 client: fetch configurations from server at runtime
Permissions, regex filters, app title, email info,
and safety now fetched using server's Info API
2018-06-27 21:20:03 +02:00
ReAnzu
2a69f0193f server/auth: add token authentication
* Users are only authenticated against their password on login,
  and to retrieve a token
* Passwords are wiped from the GUI frontend and cookies
  after login and token retrieval
* Tokens are revoked at the end of the session/logout
* If the user chooses the "remember me" option,
  the token is stored in the cookie
* Tokens correctly delete themselves on logout
* Tokens can expire at user-specified date
* Tokens have their last usage time
* Tokens can have user defined descriptions
* Users can manage login tokens in their account settings
2018-03-25 22:23:29 +02:00
ReAnzu
3f52aceca4 server/users: harden password hashes
- Changed password setup to use libsodium and argon2id (regular SHA256
  hashing for passwords is inadequate as modern GPU's can hash generate
  billions of hashes per second).
- Added code to auto migrate old passwords to the new password_hash if
  the existing password_hash matches either of the legacy password
  generation schemes (SHA1 or SHA256).
- Added migration to support new password_hash format length
- Added column password_revision. This field will default to 0, which
  all passwords will have till they're updated. After that each password
  hash method has a revision.
2018-03-08 23:40:47 +01:00
ReAnzu
7519e071e7 server/posts: deleting a post purges its artifacts
Specifically, its thumbnail and post source.
2018-03-08 23:37:37 +01:00
ReAnzu
12ec43f098 server/posts: auto convert GIFs to WEBMs/MP4s
- Default setting is false for both conversions, as this will require
  additional resources of the server, but is bandwidth friendly for
  viewers
- WEBM conversion is slow, but better quality than MP4 conversion with
  a typically smaller file size
- Tags are copied over from the original upload
- Snapshots are generated for the new auto posts
2018-03-08 07:48:45 +01:00
ReAnzu
4ff8be6a2f server/posts: ignore ffmpeg warnings
Poorly formatted MP4 and WEBM sources can cause ffmpeg to throw a lot
of warnings. However when there is byte ouptut, the generated thumbnail
is valid. Add a bypass for the resize_fill function to allow ffmpeg to
error.
2018-03-08 07:48:44 +01:00
ReAnzu
4b3529272e server/users: let administrators add new users
* Added functionality for administrators to directly add users to the
  application
* Added permission users:create:any to handle level that users are
  allowed to create other users
* Moved old permission users:create to users:create:self
2018-03-07 21:30:24 +01:00
rr-
a1fbeb91a0 server/users: fix checking passwords with colons 2018-02-10 14:04:02 +01:00
rr-
f8c7375b01 server/tags: allow uppercase tag category colors
i.e. colors such as "#FF0000"
2017-10-08 21:38:38 +02:00
rr-
1c4c5c5f91 remove tags.json 2017-10-01 21:48:00 +02:00
rr-
674d6c35d7 server/posts: add posts:view:featured privilege 2017-08-24 17:17:09 +02:00
rr-
4afece8d50 server/posts: add non-guessable IDs to post URLs 2017-08-24 17:17:09 +02:00
rr-
3c138685ea server/images: handle resizing errors 2017-05-03 12:10:04 +02:00
rr-
a1b762c65f api: fix getting cached disk usage with empty dirs 2017-05-01 20:26:53 +02:00
rr-
4bc58a3c95 server: lint 2017-04-24 23:30:53 +02:00
rr-
467b4a7630 server/tags: fix nondeterministic siblings order 2017-04-24 22:48:11 +02:00
rr-
8e5798ab8c server/tests: fix content sync tests on postgres 2017-04-24 22:36:41 +02:00
rr-
e4aa38f159 server/search: fix errors on negative page offsets 2017-04-24 22:12:12 +02:00
rr-
ba4df16499 server/search: add search term escaping 2017-04-24 21:59:38 +02:00
rr-
9814b132c3 server/search: fix searching for ---
Allow only one negation sign.
Also throw an error if user searches only for "-".
2017-04-24 19:55:02 +02:00
rr-
0014721053 server/tags: fix retrieving many tags 2017-04-19 14:44:54 +02:00
rr-
7044d2aaee server/posts: ignore old elasticsearch results 2017-03-12 18:30:42 +01:00
rr-
5681fd11ef server/net: make the user-agent configurable
Fixes #127
2017-03-03 17:27:23 +01:00
rr-
fdad08e176 server: use index-based paging (#123) 2017-02-09 22:40:00 +01:00
Alice Ryhl
a3b3532ca4 server/api: patch timing attack on password reset form 2017-02-07 20:29:37 +01:00
rr-
7f09306dde server/api: fix unicode urls (#121) 2017-02-07 18:03:35 +01:00
rr-
74c583f11d server/build: fix alembic environment script 2017-02-05 23:29:21 +01:00
rr-
72056e0cd2 server/requirements: fix skimage package name...
Brain fart during previous commit...
2017-02-05 23:27:59 +01:00
rr-
ee6b66329b server/posts: fix search by aspect ratio
It was being rounded to nearest integer because of the width/height
columns' data type.
2017-02-05 23:21:43 +01:00
rr-
49e5975254 server/model: use new sqlalchemy import style 2017-02-05 23:21:43 +01:00
rr-
f40a8875c4 server/files: fix import for Py3.5
os.DirEntry is available only from Python3.6+.
2017-02-05 22:38:55 +01:00
rr-
4caa980bf8 server/build: add missing dependency
Althought szurubooru is now no longer dependent from image-match, the
pulled code still needs the skimage library.
2017-02-05 22:38:05 +01:00
rr-
00c3a4320b server/posts: support aspect-ratio search query 2017-02-05 22:09:33 +01:00
rr-
0b21d98c9b server/posts: support note-text search query 2017-02-05 21:51:53 +01:00
rr-
e725f4f99c server/api: extra validation of list fields 2017-02-05 16:34:45 +01:00
rr-
705967d0fb server/scripts: remove lint
Any configuration for pycodestyle should go to the new setup.cfg file.
2017-02-05 16:34:45 +01:00
rr-
350e9dd331 server/scripts: replace ./test with setup.cfg 2017-02-05 16:34:45 +01:00
rr-
e490080347 server/scripts: remove migration script
It was unmaintained for months (years?) anyway
2017-02-05 16:34:45 +01:00
rr-
ad842ee8a5 server: refactor + add type hinting
- Added type hinting (for now, 3.5-compatible)
- Split `db` namespace into `db` module and `model` namespace
- Changed elastic search to be created lazily for each operation
- Changed to class based approach in entity serialization to allow
  stronger typing
- Removed `required` argument from `context.get_*` family of functions;
  now it's implied if `default` argument is omitted
- Changed `unalias_dict` implementation to use less magic inputs
2017-02-05 16:34:45 +01:00
rr-
abf1fc2b2d server: make linters happier 2017-02-03 22:42:14 +01:00
rr-
fd30675124 server/image-hash: do not depend on image-match
While I hold this library in great esteem for its excellent work on
implementing the original paper, I have several problems with it:

- as of this commit, it (again) has bug fixes unreleased on pip
- its code is badly structured
    - forces OOP and then proceeds @staticmethod everything
    - bad class design, parameters are repeated in several places
    - terrible contract of make_record() and generate_signature()
    - ambiguous parameters: path vs. image path vs. image content
    - doesn't adhere to PEP-8
- depends on cairo just to render svg images almost no one uses this
  library with
2017-02-03 21:20:52 +01:00
rr-
894cd29511 server/tests: test image hash 2017-02-03 19:53:10 +01:00
rr-
b21ffac820 server/scripts: make pytest happier 2017-02-03 19:22:33 +01:00
rr-
f828c375e6 server/posts: fix reverse search late evaluation
Uploading webms caused 'Not an image.' error to be shown, cause
generators are evaluated lazily, so the `catch` never worked.
2017-02-02 21:52:52 +01:00
rr-
accdb51c0b server/migrations: add default tag category 2017-02-02 20:26:22 +01:00
rr-
f2fd769767 server/migrations: fix imports for alembic
`alembic revision -m 'blah blah'` rightfully complained about imports
(in case of `upgrade`, that module was being populated by some other
module.)
2017-02-02 20:06:20 +01:00
rr-
e92bd2fd80 server/tags: fix getting default category name
No categories? Should have thrown an error rather than returning None.
2017-02-02 20:04:09 +01:00
rr-
cce543e0b6 server/posts: commit reverse search population 2017-02-02 19:46:35 +01:00
rr-
af6c35ed6b server/rest: rollback session on query exception
Kills complaints from sqlalchemy when an error happens during
insertion/update hook.
2017-02-02 19:46:03 +01:00
rr-
07d0b43d4c server/posts: reduce warnings from sqlalchemy
...regarding empty IN() statements
2017-02-02 19:46:03 +01:00
rr-
8be0e731a7 server/facade: run without elasticsearch
...but don't let user upload any images until they fix their
configuration
2017-02-02 19:46:03 +01:00
rr-
ec9c70ba68 server/facade: disable elasticsearch logs
Errors are covered by new safety mechanisms in image hash.
2017-02-02 19:46:03 +01:00
rr-
aa1faa3ccb server/image-hash: improve exception handling 2017-02-02 19:46:03 +01:00
rr-
f42fbbdc56 server/images: support webm with multiple streams 2017-01-25 17:13:39 +01:00
rr-
0cfc9bcafd server/posts: fix handling corrupt files
In case of a ProcessingError, the image dimensions are set to None. But
after that, they are compared with 0, which resulted in a TypeError.
2017-01-25 17:11:05 +01:00
rr-
9b27e113b3 server/search: escape backslashes in search 2017-01-21 00:22:53 +01:00
rr-
783171729f server: remove unneeded waitress wrapper 2017-01-21 00:22:53 +01:00
rr-
e5f250260d server: make gunicorn friendly 2017-01-21 00:22:53 +01:00
rr-
6b42d787a7 server: fix problems with escaping 2017-01-21 00:22:53 +01:00
rr-
7414d1f7a6 server/posts: fix getting posts around
Querying this undocumented API resulted in 500 ISE unless the client
asked only for the "id" field.
2017-01-20 22:17:26 +01:00
rr-
8d8165a0d7 server/tags: fix order of aliases in export
fixes #112
2017-01-08 22:29:05 +01:00
rr-
036fa9ee39 server/uploads: add file upload api 2017-01-08 10:25:29 +01:00
rr-
4cb613a5c9 server/posts: change reverse image search API
Add exact duplicates search; refactor to use classes over dictionaries
2017-01-07 14:07:31 +01:00
rr-
9edaaffec2 server/posts: fix post relations
Trying to relate post to itself resulted in 500 ISE.
2017-01-03 21:37:38 +01:00
rr-
627574a9c2 server: make pylint happier 2017-01-03 21:35:08 +01:00
rr-
902a0d3fe0 server/db: fix closing DB sessions
Certain exception scenarios led to small disasters. Moved database
session management directly to router, since it's that sensitive.
2017-01-03 21:29:48 +01:00
rr-
ef079121a9 server/rest: simplify error handling flow 2017-01-03 21:17:41 +01:00
rr-
f301ca9a8a server/image-hash: fix handling invalid input 2016-12-26 19:03:04 +01:00
rr-
a7a5cc8180 server/posts: expose reverse image search 2016-12-26 15:00:16 +01:00
rr-
1a59a74d63 server/image-hash: add image search engine 2016-12-26 15:00:16 +01:00
rr-
42666706d9 server/util: fix API queries for empty ?options 2016-11-20 16:02:45 +01:00
rr-
c0d484689b server: postpone circular dependency evaluation
Hopefully this improves importing with python 3.4
2016-11-07 19:28:54 +01:00
rr-
141c9fcdc9 server/tags: merge also tag relations 2016-10-22 18:02:50 +02:00
rr-
995cd4610d server: drop old style class declarations 2016-10-22 14:43:52 +02:00
rr-
e71718c50d server/posts: add replaceContent to post merging 2016-10-21 22:34:45 +02:00
rr-
9d6a0e0173 server/posts: add post merging 2016-10-21 21:48:38 +02:00
rr-
b853caf6f5 server/posts: fix relation updating
Fixes #103
2016-10-02 17:21:15 +02:00
rr-
b0c5031001 client+server/posts: reverse next/prev post role
In the post list, when we navigate to the page with ">" button, we
navigate to older posts.
In the post view, when we navigate to the page with ">" button, we
navigate to older posts as well.

However, in the post list, the ">" button is called "next page".
At the same time, in the post view, the ">" button was called "previous
post". Now it's called "next post".

The difference isn't visible to normal users nor even API consumers as
the "get posts around post X" request isn't documented.

The change is motivated not only by consistency, but to also improve
compatibility with Vimperator's `[[` and `]]`. Vimperator assumes the
word "next" refers to ">" and the word "previous" refers to "<".
2016-10-02 17:07:08 +02:00
rr-
049a0dc351 server/mime: fix GIF animation heuristics
Closes #100
2016-09-29 12:59:40 +02:00
rr-
e8c93cd735 server: fix constructing of HTTP errors
When I added error codes, I missed these exceptions.
2016-09-26 22:51:07 +02:00
rr-
0c61e85340 server: fix lint 2016-09-26 22:51:00 +02:00
rr-
560a7d6839 server/search: prefer arrays over ranges
(No, it doesn't work recursively.)
Also fix tests.
2016-09-26 22:48:09 +02:00
rr-
1e65622daf server/search: don't be a hardass about strings
Let range criteria (values that contain ..) that end up being used as
strings, to be used as if they were simple criteria. So let the user
search for "when_you_see_it..." and don't throw a warning.
2016-09-26 22:48:09 +02:00
rr-
1bd8af47b0 server/search: match only [a-z-]* for named tokens
Adds ability to search for *:* for example. Still not perfect, but it's
a start.
2016-09-26 22:06:18 +02:00
rr-
0e31e1fd14 server/search: fix underscores and percentages
Escape them for LIKE statements.
2016-09-26 21:58:27 +02:00
rr-
71a4ce8764 server/func: handle download errors 2016-09-25 14:52:47 +02:00
rr-
d1cad99e87 server/middleware: fix reporting auth errors 2016-09-24 08:38:15 +02:00
rr-
42b7a9b94f server/errors: fix serializing errors 2016-09-10 15:28:32 +02:00
rr-
16d04adde0 server/errors: add and document error codes 2016-09-10 11:36:01 +02:00
rr-
8674c8b50e server/posts: report duplicate post ID and URL 2016-09-10 10:16:14 +02:00
rr-
0a19e7bbd0 server/errors: allow extra info in errors 2016-09-10 10:16:14 +02:00
rr-
c516030c66 server/tests: fix info api tests 2016-09-10 10:12:43 +02:00
rr-
7fa8593b0a client/general: improve URL escaping
Specifically, cater for /, + and % in URL components.
2016-09-04 02:07:22 +02:00
rr-
a22fe306d1 server/posts: fix deleting posts with relations 2016-08-31 22:49:45 +02:00
rr-
eff0e002f2 server/info: increase hdd usage cache time to 48h 2016-08-31 22:22:06 +02:00
rr-
243ab15b85 server/tags: add order to tag names
The better implementation of a224297.

Fixes ability to reorder tag aliases, especially - the ability to change
the tag's primary name after it was created. Until now, both of these
scenarios needed sad workarounds on the user part.
2016-08-28 20:00:50 +02:00
rr-
c366b608da server/search: fix negating complex searches
Entering:

    miko -miko

is a contradiction that shouldn't have been returning any matches, but
it has nonetheless. This change fixes the construction of negated
expressions that use subqueries.
2016-08-28 18:43:05 +02:00
rr-
3c5878cb16 server/tags: improve tag list performance 2016-08-27 22:19:01 +02:00
rr-
ef0f74297f server/tag-categories: fix default categories
- Don't cache default category in its entirety - cache only its name
- Purge cache on category name changes and default category changes
- Lock records for updates where applicable
2016-08-27 12:39:59 +02:00
rr-
06ab98fa70 server/search: fix sort:random breaking tags
Using sqlalchemy's subqueryload to fetch tags works like this:

1. Get basic info about posts with query X
2. Copy query X
3. SELECT all tags WHERE post_id IN (SELECT post_ids FROM query X)
4. Associate the resulting tags with the posts

When original query contains .order_by(func.random()), it looks like
this:

1. SELECT post.* FROM post ORDER BY random() LIMIT 10
2. Copy "ORDER BY random() LIMIT 10"
3. SELECT tag.* FROM tag WHERE tag.post_id IN (
       SELECT id FROM post ORDER BY random() LIMIT 10)
4. Disaster! Each post now has completely arbitrary tags!

To circumvent this, we replace eager loading with lazy loading. This
generates one extra query for each result row, but it has no chance of
producing such anomalies. This behavior is activated only for
queries containing "sort:random" and derivatives so it shouldn't hit
performance too much.
2016-08-27 01:21:59 +02:00
rr-
f8e91a10e8 server/search: refactor query factories 2016-08-27 01:19:29 +02:00
rr-
6d26b5c37a server/search: fix sort:random 2016-08-26 23:27:33 +02:00
rr-
fa60b42f65 server/search: improve post list performance 2016-08-26 17:57:20 +02:00
rr-
422b99ac8d server/search: add content-checksum 2016-08-26 16:26:06 +02:00
rr-
ffb87f1650 server/posts: defer flush; save content lazily
Rather than flushing the post right away only to find out that there
were validation errors, try to postpone flushing for as long as
possible.

The previous behavior has led to too eager spending of post IDs - each
flush calls nextval(post_id_seq), and postgres sequences are not
affected by transaction rollbacks, so each erroneous post creation
discarded a post ID, which has led to gaps in post IDs.
2016-08-26 15:09:08 +02:00
rr-
bb369efa99 server/general: disable autoflush 2016-08-26 14:41:05 +02:00
rr-
280a55046a server/db: make query counter thread-local 2016-08-24 12:31:55 +02:00
rr-
73a8542220 server/posts: make anon snapshots for anon uploads 2016-08-22 20:07:39 +02:00
rr-
61d084cc66 server/search: support 'submit:' for anon uploads 2016-08-22 19:45:25 +02:00
rr-
b7e9cbd541 server/posts: allow tagless posts 2016-08-21 23:40:01 +02:00
rr-
28bcbd33b9 server/posts: use SHA1 checksums
This changes the checksums to ones that are compatible with 1.x, which
relieves the migration script from recalculating the checksums for all
the posts.
2016-08-20 13:06:19 +02:00
rr-
80af79779d server/snapshots: rewrite 2016-08-16 21:51:25 +02:00
rr-
65119d69ab server/rest: urldecode incoming paths 2016-08-16 21:42:11 +02:00
rr-
0c2efc0be5 server/rest: work around Python bug 27777
(See http://bugs.python.org/issue27777)
2016-08-16 21:42:11 +02:00
rr-
e89a086d58 server/rest: fix reporting parameter type errors 2016-08-16 21:42:11 +02:00
rr-
ff7bbbdd8a server/db: fix upgrade (regression from 9aea55e) 2016-08-16 21:42:11 +02:00
rr-
6c29377f6b server/db: allow full DSN; use memdb in tests
The earlier commit is still relevant as it allows to integrate real
database when needed.
2016-08-16 21:42:09 +02:00
rr-
e688f39887 server/db: specify nullable for foreign keys
This is mostly stylistic change, but it clearly shows the intended
behavior should we detect more integrity violation errors.
2016-08-16 17:22:33 +02:00
rr-
522886ae6c server/db: fix user cascade deletions 2016-08-16 17:22:33 +02:00
rr-
a224297c4f server/db: poor fix for tag aliases' order
It's a hack, but it seems to work okay.
2016-08-16 17:22:33 +02:00
rr-
48af5160df server/search: fix negative offsets causing ISE 2016-08-16 17:22:33 +02:00
rr-
87b1ee4564 server/tests: use real database
I'm experimenting with snapshots and found following limitation of
SQLite: https://www.sqlite.org/isolation.html
2016-08-16 17:22:33 +02:00
rr-
0320a0b55b server/general: improve versioning effectiveness
...by integrating it with sqlalchemy that adds WHERE conditions for each
UPDATE and DELETE statement.
2016-08-16 17:22:33 +02:00
rr-
ef4af697c4 server/tags: fix tag sorting
Brainfart from d6942121e5
2016-08-14 17:54:15 +02:00
rr-
663aacdf82 server/tools: add lint script
Integrated both pylint and pycodestyle.
2016-08-14 16:46:50 +02:00
rr-
9aea55e3d1 server/general: embrace most of PEP8
Ignored only the rules about continuing / hanging indentation.

Also, added __init__.py to tests so that pylint discovers them. (I don't
buy pytest's BS about installing your package.)
2016-08-14 16:44:03 +02:00
rr-
af62f8c45a server/general: ditch falcon for in-house WSGI app
For quite some time, I hated Falcon's class maps approach that caused
more chaos than good for Szurubooru. I've taken a look at the other
frameworks (hug, flask, etc) again, but they all looked too
bloated/over-engineered. I decided to just talk to WSGI myself.

Regex-based routing may not be the fastest in the world, but I'm fine
with response time of 10 ms for cached /posts.
2016-08-14 16:43:35 +02:00
rr-
d102c9bdba server/tests: update func.posts tests 2016-08-14 16:43:35 +02:00
rr-
264f9ee70b server/tests: update func.mime tests 2016-08-14 16:43:35 +02:00
rr-
c23c401c4d server/tests: add func.tags tests 2016-08-14 16:43:35 +02:00
rr-
53e96ba41f server/tests: add func.tag_categories tests 2016-08-14 16:43:35 +02:00
rr-
81dfbaec98 server/tests: add func.users tests 2016-08-14 16:43:35 +02:00
rr-
03c74cb5a3 server/tests: add func.comments tests 2016-08-14 16:43:35 +02:00
rr-
65efc309a8 server/comments: catch bad IDs 2016-08-14 16:43:31 +02:00
rr-
56b3eb9674 server/util: fix case conversion 2016-08-14 16:43:04 +02:00
rr-
86452019a3 server/util: improve catching bad field names
KeyError could catch exceptions that happened inside the serializer
routine and mistakenly report them as an error with user input.
2016-08-14 16:43:04 +02:00
rr-
f6f07a35df server/general: authenticated_user->auth_user 2016-08-14 16:43:04 +02:00
rr-
c2bbf7b62c server/general: add assertions 2016-08-14 16:43:04 +02:00
rr-
bb86e9bf56 server/posts: add more safety checks for notes 2016-08-14 16:43:04 +02:00
rr-
7cd4a1a530 server/tags: verify description size 2016-08-14 16:43:01 +02:00
rr-
3db4f39545 server/tag-categories: correct exception type 2016-08-14 12:31:46 +02:00
rr-
92075bb455 server/tags: don't verify tag names while getting 2016-08-14 12:31:46 +02:00
rr-
d6942121e5 server/tags: change tags sort order 2016-08-14 12:31:21 +02:00
rr-
07237bc2bc server/tags: fix changing name case 2016-08-14 11:39:04 +02:00
rr-
7e5deee76b server/tags: fix tag merging 2016-08-14 11:39:04 +02:00
rr-
2b3d193b7c server/tags: don't auto-create tag categories 2016-08-14 11:38:59 +02:00