Branch: external-maps

cursor/RSP-108-fix-rspamd-dkim-key-loading-for-ed25519-9e75

cursor/RSP-246-parse-esmtp-arguments-for-lua-api-55e6

cursor/RSP-251-add-vault-kv-version-2-support-7418

cursor/RSP-259-fix-dkim-relaxed-bodyhash-calculation-for-spaces-b6ff

cursor/RSP-268-fix-duplicate-symbol-in-once-received-plugin-3c4b

cursor/RSP-271-implement-metadefender-hash-lookup-module-113a

cursor/RSP-289-add-auid-support-to-dkim-signing-96d3

cursor/RSP-76-fix-ed25519-key-loading-in-arc-module-f263

external-maps

libev-migration

log_json

master

mime-rework

rdns-tcp-rework

rework-symcache

rspamd-0.5

rspamd-0.6

rspamd-0.7

rspamd-0.8

rspamd-0.9

rspamd-1.0

rspamd-1.1

rspamd-1.2

rspamd-1.3

rspamd-1.4

rspamd-1.5

rspamd-1.6

rspamd-1.9

rspamd-3.10

rspamd-3.7

rspamd-3.8

rspamd-3.9

sync-public-suffix-list

torch-removal

vstakhov-another-grow-factor-fix

vstakhov-dkim2

vstakhov-fasttext-langdet

vstakhov-fuzzy-cxx

vstakhov-fuzzy-tcp

vstakhov-fuzzy-tcp-test

vstakhov-known-senders

vstakhov-learn-framework

vstakhov-llm-embeddings

vstakhov-lua-text-api

vstakhov-mailto-dedup

vstakhov-move-obfuscation-flags

vstakhov-neural-force-train

vstakhov-new-build-workflows

vstakhov-smtp-proxy-exp

vstakhov-stringzilla

0.2.7

0.3.0

0.3.1

0.3.10

0.3.11

0.3.12

0.3.13

0.3.14

0.3.2

0.3.3

0.3.7

0.3.8

0.3.9

0.4.0

0.4.1

0.4.2

0.4.3

0.4.4

0.4.5

0.4.6

0.4.7

0.5.0

0.5.1

0.5.2

0.5.3

0.5.4

0.5.5

0.5.6

0.6.0

0.6.1

0.6.10

0.6.2

0.6.3

0.6.4

0.6.5

0.6.6

0.6.7

0.6.8

0.6.9

0.7.0

0.7.1

0.7.2

0.7.3

0.7.4

0.7.5

0.7.6

0.8.0

0.8.1

0.8.2

0.8.3

0.9.0

0.9.1

0.9.10

0.9.2

0.9.3

0.9.4

0.9.5

0.9.6

0.9.7

0.9.8

0.9.9

1.0.0

1.0.1

1.0.10

1.0.11

1.0.2

1.0.3

1.0.4

1.0.5

1.0.6

1.0.7

1.0.8

1.0.9

1.1.0

1.1.1

1.1.2

1.1.3

1.1.4

1.2.0

1.2.1

1.2.2

1.2.3

1.2.4

1.2.5

1.2.6

1.2.7

1.2.8

1.3.0

1.3.1

1.3.2

1.3.3

1.3.4

1.3.5

1.4.0

1.4.1

1.4.2

1.4.3

1.4.4

1.5.0

1.5.1

1.5.2

1.5.3

1.5.4

1.5.5

1.5.6

1.5.7

1.5.8

1.5.9

1.6.0

1.6.1

1.6.2

1.6.3

1.6.4

1.6.5

1.6.6

1.7.0

1.7.1

1.7.2

1.7.3

1.7.4

1.7.5

1.7.6

1.7.7

1.7.8

1.7.9

1.8.0

1.8.1

1.8.2

1.8.3

1.9.0

1.9.1

1.9.2

1.9.3

1.9.4

2.0

2.1

2.2

2.3

2.4

2.5

2.6

2.7

3.0

3.1

3.10.0

3.10.1

3.10.2

3.11.0

3.11.1

3.12.0

3.12.1

3.13.0

3.13.1

3.13.2

3.14.0

3.14.1

3.14.2

3.14.3

3.2

3.3

3.4

3.5

3.6

3.7.1

3.7.2

3.7.3

3.7.4

3.7.5

3.8.0

3.8.1

3.8.2

3.8.3

3.8.4

3.9.0

3.9.1

4.0.0

4.0.1

4.1.0

4.1.1

4.1.2

4.1.3

4.1.4

Branches Tags

${ item.name }

Create tag ${ searchTerm }

Create branch ${ searchTerm }

from 'external-maps'

${ noResults }

Commit Graph

114 Commits (external-maps)

Author	SHA1	Message	Date
Vsevolod Stakhov	05da2709be	Release 3.4 * [CritFix] Restore compatibility with the integrations and headers alterations * [Feature] Milter_headers: Add `x-rspamd-action` routine * [Feature] Share hyperscan database among processes * [Fix] Another corner case in url parsing * [Fix] Another fix for the enable password * [Fix] Another try to fix close method in lua_tcp * [Fix] Fix additional fields in the Redis schema * [Fix] Fix emoji joiner FP * [Fix] Fix favicon.ico Content-Type header * [Fix] Fix hang when close is used * [Fix] Lua_tcp: Sigh, another try to fix `close` invocation * [Fix] Mx_check: Cache the fact of a missing MX record * [Fix] Try to fix parsing of the unencoded `>` characters in html attributes * [Fix] Try to fix the case where password == enable_password * [Project] (Re)implement hyperscan caching * [Project] Rework cleanup * [Project] Synchronize hyperscan caches via the main process * [Rework] Convert multipattern to use hyperscan tools * [Rework] Make http normalize path function a generic function * [Rework] Split locked and unlocked files, as mmap does not need flock normally * [Rework] Start movement of the hyperscan related routines into a single unit * [Rework] Store the current worker, so other libraries could use this information * [Rework] Use blocking socket for IPC between main and workers * [Rework] Use more predictable size for commands buffers * [Rules] Do not insert ONCE_RECEIVED_STRICT on RDNS missing * [Rules] Reduce score of HTTP_TO_HTTPS - subject to remove completely	4 years ago
Vsevolod Stakhov	16819f6fd0	Release 3.3 * [Conf] Add missing groups for whitelist module symbols * [CritFix] Neural: Fix keys regression after #3968 * [Feature] Accept upstream in lua_tcp * [Feature] Add ability to statically maintain disabled/enabled patterns * [Feature] Add function to store upstreams for HTTP urls * [Feature] Allow augmentations set in Lua API * [Feature] Allow lua_http module to accept upstreams * [Feature] Allow to limit write access to fuzzy storage by key * [Feature] Allow to sort symbols output * [Feature] Check content for binary stuff before dumping it to Lua * [Feature] Implement symbols augmentations * [Fix] Add missing flags * [Fix] Add more sanity checks for rua in dmarc_report * [Fix] Adjust length of the fuzzy checks for short text parts * [Fix] Another try to fix add headers compatibility logic * [Fix] Another try to fix race condition in the runtime destruction * [Fix] Avoid cyclic references in symcache and fix memory leaks * [Fix] Avoid overriding IP with Sender IP * [Fix] BAD_REP_POLICIES did not trigger when message was classified as spam by Bayes * [Fix] Bind AF_UNIX DGRAM client connection to annonymous address * [Fix] Disable IPv6 lookups for Blocklist.de RBL * [Fix] Distinguish dynamic and static items * [Fix] Dkim: Ignore unknown DKIM kv pairs as stated in RFC * [Fix] Dmarc report: Use local timezone instead of GMT * [Fix] Do not exclude authenticated users from URIBL lookups * [Fix] Empty envelopes should not be emitted as arrays (json+messagepack) when populated envelopes are objects. This greatly complicates decoding in strictly typed languages. * [Fix] External_relay: Restore the originating hostname check * [Fix] Fix DKIM keys with spaces still allowing errors on invalid base64 * [Fix] Fix copying of sockaddr_un addresses * [Fix] Fix crash with cname replies * [Fix] Fix dependencies propagation * [Fix] Fix iteration over milter headers * [Fix] Fix ordering when sorting symcache * [Fix] Fix reading of the cached maps * [Fix] Fix several issues with the HTTP keepalive parsing * [Fix] Fix stack smashing * [Fix] Fix synchronous auth/select in lua_redis * [Fix] Fix various symcache issues * [Fix] Ignore all (I hope) unknown DKIM signature KV pairs * [Fix] Ignore directories in RarV5 archives * [Fix] Libucl: avoid memory leak on objects merging * [Fix] Lua_tcp: Another try to fix closing logic * [Fix] Mempool: Fix alloc_array function to actually multiply nmembers by size * [Fix] Only check allowed fuzzy worker update ips for non-unix sockets * [Fix] Plug memory leak in regexp destruction with pcre2 * [Fix] Properly check the original email flag * [Fix] Properly deal with `get_symbol/get_metric_symbol` ambiguity * [Fix] Properly parse expressions atoms * [Fix] Properly set `Host` in rspamd_proxy * [Fix] Rbl: Fix received positioned checks * [Fix] Remove check for a score with no symbol being registered * [Fix] Same fix for lua_tcp * [Fix] Skip cname records when processing SPF records * [Fix] Skip sending dmarc reports in no-opt mode fixes https://github.com/rspamd/rspamd/issues/4241 * [Fix] Stop slow timer on task destruction * [Fix] Symcache: Do not use C style comparators in C++ sorts * [Fix] Try to avoid a corner case for `@` pattern * [Fix] Try to fix dkim reputation adjustements * [Fix] Try to fix passthrough results processing logic * [Fix] Try to fix the mess with read only flag * [Fix] Upstreams: Don't ignore revive_time config option * [Fix] Use proper format string, sigh... * [Fix] Use space category in ragel automata to resolve space characters * [Fix] Zstd: Fix compression with the new Zstd API * [Fix] milter_headers: Header fields may be inserted at wrong position. * [Project] Add experimental HTTP statistics backend * [Project] Add more methods for symbols addition * [Project] Add raii_sink file helper * [Project] Add some more methods * [Project] Add symbols processing methods * [Project] Allow `=` separated augmentations to be treated as kv pairs * [Project] Allow to extract augmentation values * [Project] Few more methods * [Project] Fix on conditions * [Project] Further efforts to make a more consistent architecture * [Project] Further rework * [Project] Further rework tracking * [Project] Further split of the code * [Project] Get rid of C style ctors/dtors * [Project] Http_stat: Notice statfiles when creating runtime * [Project] Implement dynamic items lookup and processing * [Project] Implement item finalization * [Project] Implement more methods * [Project] Implement runtime creation * [Project] Implement settings processing + some neats * [Project] Implement some conditions checks * [Project] Implement validation logic * [Project] More methods * [Project] Move runtime cache part to a separate unit * [Project] Move some more methods * [Project] Re-implement counters method * [Project] Reimplement dependencies processing * [Project] Remove obsoleted methods * [Project] Remove old code (finally) * [Project] Rework symbols execution * [Project] Some more adjustments in symbols registration * [Project] Start rewrite symcache in c++ * [Project] Support augmentations with values * [Project] Symcache: Use ordered filters to avoid extra lookups * [Rework] Another movement * [Rework] Augmentations can now imply flags * [Rework] Further steps * [Rework] Further work on deps processing * [Rework] Implement cache resorting * [Rework] Isolate disable/enable logic for the configuration ucl objects * [Rework] Move item implementation to a separate header * [Rework] Multimap: Avoid prefilters usage where augmentations can be used * [Rework] Pass upstream when sending TCP requests * [Rework] Re-implement cache sorting * [Rework] Reimplement saving/loading the cache items * [Rework] Reiterate on priorities * [Rework] Rework files structure * [Rework] Rewrite rspamc in C++ * [Rework] Simplify scores check and extend it to pre/post filters * [Rework] Switch minimum C++ standard version to C++20 * [Rework] Try to fix the mess with types & flags * [Rework] Use another version of hash table from the same author * [Rework] Use dynamic items for calling callbacks * [Rework] Use dynamic items in the callbacks * [Rework] Use hash map for id->symbol mappings * [Rework] Use khash instead of uthash in rdns compression logic	4 years ago
Vsevolod Stakhov	c1b767c234	Release 3.2 * [Conf] Score MIME_OBFUSCATED_ARCHIVE to 8 points * [Conf] Set one_shot for URIBL rules by default * [CritFix] Fix upstreams name resolution when there is also a port * [Feature] Add ROC feature to neural network plugin * [Feature] Add public suffic compilation utility * [Feature] Add support of Cloudmark * [Feature] Allow hyperscan for ppc64, as vectorscan now suports it. * [Feature] Allow to skip DNS resolution for keep-alive connections * [Feature] Aws_s3: Allow to store large parts separately * [Feature] BIMI: Add preliminary version of the BIMI plugin * [Feature] JSON endpoint for querying maps * [Feature] Lua_magic: Add a sane CSV heuristic * [Feature] Lua_mime: Add schema for message transfer * [Feature] Output average scan time in /stat endpoint * [Feature] Show average scan time in `rspamc stat` output * [Fix] Add guards to avoid race condition on TCP connection * [Fix] Allow spaces in DKIM key records * [Fix] Apply the similar fix to the url_reputation * [Fix] Avoid overwriting whitelisted_signers_map * [Fix] Backport PR from libucl * [Fix] Clear SSL errors * [Fix] ClickHouse cleanup of old partitions * [Fix] Do not double call error handler on ssl errors in the timeout path * [Fix] Do not forget to clear pointers on IOC reset * [Fix] External_relay: Remove useless check of the map value * [Fix] Find suspicious url encodings that could break url extraction * [Fix] Fix HTTP(s) client timeout * [Fix] Fix exclude flags setting * [Fix] Fix expanding of the variables * [Fix] Fix host header usage in lua_http * [Fix] Fix http maps shared memory cache cleanup * [Fix] Fix logic in HTML processing FSM * [Fix] Fix parsing of the compound mailto urls * [Fix] Fix processing captures from pcre2 * [Fix] Fix removing from khash * [Fix] Fix stuctured headers pushing * [Fix] Further fix for i386 compilation * [Fix] Improve duplicate settings error reporting * [Fix] Lua: task:remove_result didn't work in some cases * [Fix] Output service parts as well * [Fix] Phishing: Deal with phishing + redirected URL * [Fix] Phishing: Fix finding domains in the phishing map * [Fix] Plug memory leak by using mempool for a copied address * [Fix] Properly find the request and the number of requested entries * [Fix] Rbl: Fix inversed logic of the url_full_hostname * [Fix] Read file maps if they were not pre-read during preload * [Fix] Restrict x86_64 assembly to x86_64 * [Fix] Return a real number of recipients when dealing with aliases * [Fix] Rework unshedule DNS request function * [Fix] Support definition of ungrouped symbol in conf file, use group info from lua or other conf file * [Fix] Unschedule DNS request when clearing IO channel * [Fix] When checking for phishing, we need to convert punicode -> UTF8, not vice versa * [Fix] lua_cfg_transform - actions without score (discard) * [Fix] lua_cfg_transform - silly break break actions * [Fix] ratelimit - symbol per bucket * [Project] BIMI: Fix helper integration issues * [Project] Further DNS over TCP architecturing * [Project] Rdns: Add more functions for TCP based requests * [Project] Rdns: Add preliminary reading logic for TCP channels * [Project] Rdns: Add reaper for inactive TCP connections * [Project] Rdns: Add timeout logic for TCP requests * [Project] Rdns: Do not treat TCP channels failure as fatal * [Project] Rdns: Fix TCP connection mess * [Project] Rdns: Fix TCP stuff cleanup * [Project] Rdns: Fix various ownership issues * [Project] Rdns: Implement TCP writing logic * [Project] Rdns: Initial support of TCP IO channels * [Project] Rdns: More fixes in TCP handling * [Project] Rdns: Restore the previous EDNS0 size * [Project] Rdns: Send truncated replies via TCP * [Project] Rdns: Unregister TCP requests * [Rework] Allow to restore SSL handlers after keepalive pooling * [Rework] Allow to set a different behaviour for actions from settings * [Rework] Include SSL flag into keepalive hash * [Rework] Make `rspamadm dmarc_report` default behaviour more sane * [Rework] Mempool: Use explicit alignment * [Rework] Rdns: Use faster and more compact hash table for DNS requests * [Rework] Rework SSL flag operations * [Rework] Take disabled flag into account * [Rework] Timeouts are now global per event and not reseted by IO activity * [Rework] Use xxh3 as a default hash and fix memory/alignment issues * [Rules] Fix old rules to stop global functions usage * [Rules] Fix symbol for DKIM temporary failure * [Rules] Remove ancient and inefficient rules * [Rules] Slightly reduce MULTIPLE_FROM score	4 years ago
Vsevolod Stakhov	87a4c54609	Release 3.1 * [Feature] Add junk_threshold for autolearn * [Feature] Add neural test command * [Feature] Antivirus: Allow to set fake eicar patterns for testing AV engines * [Feature] Lua_cdb: Add cdb building interface * [Feature] Ratelimit: Add per bucket configurations * [Feature] S3: Allow to store structured data in messagepack * [Fix] Add concept of uncancellable events to prevent use-after-free * [Fix] Add temporary guard to prevent linked list exploitation * [Fix] Another rework of the ucl hashing * [Fix] Another try to fix references safety * [Fix] Another try to fix rspamd_text passing in the selectors * [Fix] Avoid copy for received structure as it has raw C pointers * [Fix] Avoid dangling reference * [Fix] Correctly check numeric URLs in URL DNS lists * [Fix] Delete the correct pointer type * [Fix] Dmarc: Always lowercase domain * [Fix] Fix compilation of the hyperscan databases with errors * [Fix] Fix hash table lookup * [Fix] Fix http message flag shift * [Fix] Fix parsing of the from_hostname when it is an IP address * [Fix] Fix parsing of the unquoted attributes in HTML * [Fix] Fix passing of rspamd_text in selectors pipelines * [Fix] Fix rubbish QP sequences decoding * [Fix] Fix some complicated case with the closing tags parsing * [Fix] Fix the case when l tag is too small * [Fix] Html: Fix the case where only bgcolor is explicitly set * [Fix] Libucl: Fix deletion from ucl objects * [Fix] Namespace and add metadata for OpenMetrics, fix interleaving * [Fix] Plug memory leak in http settings reload * [Fix] Preserve SPF top record in the mempool variable * [Fix] Remove aarch64 GC64 workaround * [Fix] Remove bogus G_LIKELY * [Fix] Spf: Do not parse non TXT DNS replies as TXT replies * [Fix] Try to use on_connect/on_disconnect callbacks to handle internal Redis failures * [Fix] buffer overflow in rspamc counters * [Fix] fix static building * [Fix] lua_scanners - message_min_words logic * [Fix] src/lua/lua_mimepart.c: fix null dereference * [Project] Add constant iterators * [Project] Add helper library to handle mime strings in a more safe matter * [Project] Add preliminary support of CDB bayes dump * [Project] Add trim operations * [Project] Allow mempool allocated mime strings * [Project] Cdb: Finish backend implementation * [Project] Cdb: Fix configuration load * [Project] Cdb: Use shared data between cdb statfiles * [Project] Cdb: continue statistics backend implementation * [Project] Finish received headers rework part * [Project] Move C++ specific declarations to C++ header * [Project] Rework received headers parsing to C++ * [Project] Start using of the new received structure * [Project] Start work on cdb backend * [Rework] Further rework of the redis pool * [Rework] Redis_pool: fix issues found * [Rework] Rework learn and add classify condition * [Rework] Save invisible content to a separate buffer * [Rework] Start rewriting of the redis pool logic * [Rules] Improve zero font rule Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	5 years ago
Vsevolod Stakhov	4fc136a7f8	Release 3.0 * [Conf] Align ARC scores with DKIM scores * [CritFix] Neural: Fix sorting application * [Feature] Add a simple dumper for bayes tokens * [Feature] Add lua_maps.fill_config_maps function * [Feature] Add preliminary exporter to AWS S3 * [Feature] Add preliminary restore bayes support * [Feature] Add race condition protection against hs_helper restarts * [Feature] Add rspamd_utf8_strcmp utility * [Feature] Add zstd streaming API * [Feature] Allow to log severity level explicitly * [Feature] Allow to save and show attachment name when inserting AV scan results * [Feature] Allow to sort urls for Lua * [Feature] Allow to specify different timeouts/retransmits for fuzzy rules * [Feature] Aws_s3: Allow to compress data stored * [Feature] CMakeLists.txt: Change check and run-test to use rspamd-test-cxx * fixes #3807 * [Feature] Dmarc_report: allow sending reports in batches * [Feature] Fuzzy_check: Allow to disable subject when making short text hash * [Feature] Lua_cryptobox: Add keyed ssl hash functions via HMAC * [Feature] Lua_task: Add get_urls_filtered method * [Feature] Make monitored checks less frequent * [Feature] Milter_headers: Add x-rspamd-pre-result header * [Feature] Neural: Allow to balance FP/FN for the network * [Feature] Ppopagate monitored errors from rbl module * [Feature] Pyzor calculate score dynamically Count - WL-Count of default_score in percent * [Feature] Rbl: Distinguish flattened and non-flattened selectors in RBL requests * [Feature] Re-add pyzor support * [Feature] Settings: add ip_map check and rework structure slightly * [Feature] Spamassassin: Allow to set the default priority for SA scores * [Feature] Strip smtp comments from message id * [Feature] add SYSTEM_ZSTD cmake option To use the system zstd instead on the bundled version * [Feature] external_relay plugin * [Feature] rspamadm clickhouse neural_train subcommand * [Fix] #3400 milter_headers: fix inverted logic for extended_headers_rcpt * [Fix] ASN: fix _FAIL symbol for when main symbol is disabled * [Fix] Add a special logic for text part with no text extraction * [Fix] Add diacritics flag for several eu languages * [Fix] Another FSM fix to accomodate possibility of multiple consequent ? * [Fix] Avoid curse of dynamic array referencing * [Fix] Avoid reinitialising neural settings * [Fix] Check remain before processing TXT records * [Fix] Enable error multiplier on http errors * [Fix] Finally rework parsing entities logic * [Fix] Fix '==' parsing in the content type attributes parser * [Fix] Fix IPv6 expansion for SPF macros * [Fix] Fix Mozilla Message-ID detection * [Fix] Fix an edge case in BITCOIN_ADDR rule * [Fix] Fix brain-damaged behaviour when http request has a custom Host header * [Fix] Fix check of limits in email address parsing * [Fix] Fix copy&paste error and rework * [Fix] Fix expressions logic for and/or and float values * [Fix] Fix fuzzy retransmits * [Fix] Fix http maps with no or invalid expires data * [Fix] Fix last quote character parsing in the content-type state machine * [Fix] Fix normalisation flags propagation * [Fix] Fix overflow when appending many broken tags * [Fix] Fix parsing of rfc2047 tokens with '?' inside * [Fix] Fix phishing flag set * [Fix] Fix rfc2047 embedded into rfc2231 pieces in special headers * [Fix] Fix round-robin rotation * [Fix] Fix searching for symbols * [Fix] Fix storing of the regexps inside variant * [Fix] Fix tokenization near exceptions * [Fix] Fix visibility calculations * [Fix] Html: Attach inline tags to the structure * [Fix] Html: Do not treat empty tags as block tags * [Fix] Ical: Do not extract urls from all flags using merely specific ones * [Fix] Initialise symcache even if it cannot be loaded properly * [Fix] Lua_fuzzy: Remove text parts check when checking image dimensions * [Fix] Lua_maps: Fix adjustments for the map type in the complex map definitions * [Fix] Lua_task: Fix deleted symbols in has_symbol/get_symbol * [Fix] Move metric and symcache link from validation to the init stage * [Fix] Oletools: Another try to fix table sorting * [Fix] One more default behaviour fix * [Fix] Phishing: Rework urls processing * [Fix] RBL: was missing some config schema * [Fix] Replies: Fix 'Reply-To' handling in task:get_reply_sender * [Fix] Rework metrics handling * [Fix] Save symcache on exit * [Fix] Selectors: Filter nil elements in lists * [Fix] Selectors: Properly fix implicit tostring for nils * [Fix] Try to fix some broken code in DMARC reporting plugin * [Fix] Urls: Fix processing of html urls when it comes to the flags * [Fix] Use proper buffer length * [Fix] Various visibility fixes * [Fix]: ASN: dns cb func should also return in case of an error * [Project] Add a simple css rule definition * [Project] Add css style skeleton * [Project] Add css syntax (adopted from ebnf) * [Project] Add css_selectors * [Project] Add doctest unit testing library * [Project] Add expected library * [Project] Add fmt library for simple string ops * [Project] Add fu2 library to better functions abstractions * [Project] Add hashing method * [Project] Add parsers skeleton * [Project] Add preliminary support of vcard parser * [Project] Add process exceptions for invisible text * [Project] Add some methods for css parser * [Project] Allow static libstdc++ * [Project] Another whitespace hack * [Project] CSS: Various fixes in the declarations and values parsing * [Project] Cpp: Add robin-hood hash map library * [Project] Css: Add AST debug * [Project] Css: Add colors conversion functions * [Project] Css: Add dimensions handling * [Project] Css: Add display value support * [Project] Css: Add frozen library from https://github.com/serge-sans-paille/frozen/ * [Project] Css: Add opacity support * [Project] Css: Add parser helpers to simplify debugging * [Project] Css: Add preliminary stylesheet support * [Project] Css: Add rules processing functions and tests * [Project] Css: Add simple selectors unit tests * [Project] Css: Add some c++ unit tests * [Project] Css: Add some debug methods * [Project] Css: Add some debug statements for the css parser * [Project] Css: Add some logical skeleton for declarations parser * [Project] Css: Add url/function tokens * [Project] Css: Allow at rules parsing * [Project] Css: Declarations parsing logic skeleton * [Project] Css: Enable conditional css parsing support from the HTML parser * [Project] Css: Finish generic lexer cases * [Project] Css: Fix HSL conversion * [Project] Css: Fix minus parsing * [Project] Css: Fix parser consumers nesting * [Project] Css: Fix parsing of the qualified rules * [Project] Css: Fix rules merging * [Project] Css: Further fixes to lexer * [Project] Css: Further steps to parse css colors + rework * [Project] Css: Further work on parser's methods * [Project] Css: Implement backlog of css tokens * [Project] Css: Implement numbers and ident parsers * [Project] Css: Implement simple css selectors lookup * [Project] Css: Implement styles merging * [Project] Css: Make debug strings json like to simplify tests * [Project] Css: Minor adjustments * [Project] Css: More meat to the lexer * [Project] Css: Move some of the tests to the doctest * [Project] Css: Projected a parser * [Project] Css: Properties attachment logic * [Project] Css: Remove ragel from build targets (maybe keep for reference) * [Project] Css: Rework css block structure * [Project] Css: Rework flags of css properties * [Project] Css: Rework tokens structure * [Project] Css: Several fixes + tests * [Project] Css: Simplify checks * [Project] Css: Simplify debug code * [Project] Css: Start css selectors parsing logic * [Project] Css: Start semantic parsing for rules * [Project] Css: Start stylesheet implementation * [Project] Css: Tidy up lambdas * [Project] Css: rework tokeniser * [Project] Dmarc: Add dmarc report tool (WIP) * [Project] Dmarc: Add munging configuration * [Project] Dmarc: Add preliminary munging logic * [Project] Dmarc: Fix header removal * [Project] Dmarc: Fix munging logic * [Project] Dmarc: Use full recipient address instead of a domain map * [Project] Dmarc: Use zlists for dmarc reports * [Project] Dmarc_report: Add message generation logic * [Project] Dmarc_report: Add preliminary sending support * [Project] Fix lua bindings * [Project] Fix xml/sgml tags processing * [Project] Handle new modification * [Project] Html/CSS: Add transform from a CSS rule to html block * [Project] Html/CSS: Link html and css styles * [Project] Html/CSS: Switch styles parsing to css parser * [Project] Html/Css: Fix some issues found * [Project] Html/Css: Implement visibility rules for a block * [Project] Html: Add more tests cases and fix some more corner issues * [Project] Html: Add rows display type support * [Project] Html: Allow decode entities function to normalise spaces + unit tests * [Project] Html: Another rework of the tags structure * [Project] Html: Another try to fix unbalanced cases * [Project] Html: Fix crossing spans * [Project] Html: Fix parent propagation * [Project] Html: Further rework of the html parsing stuff * [Project] Html: Implement logic for tags pairing * [Project] Html: Implement rawtext state machine * [Project] Html: Insert closing tags as well :( * [Project] Html: More fixes * [Project] Html: More fixes * [Project] Html: More spaces logic fixes * [Project] Html: One more attempt to write text content * [Project] Html: Replace \0 in html content * [Project] Html: Rework img/a tags handling * [Project] Html: Rework propagation method * [Project] Html: Rework tags placement * [Project] Html: Rework transparency logic * [Project] Html: Support 'hidden' attribute * [Project] Html: Try another approach to append tags content * [Project] Html: Try to deal with bad unknown tags properly * [Project] Lua_aws: Add canonicalisation utility * [Project] Lua_aws: Add function to produce AWS Authorisation header * [Project] Lua_aws: Implement request signing * [Project] Lua_mime: Add lua_mime.modify_headers routine * [Project] Lua_task: Add modify_header method * [Project] Lua_task: Allow to extract modified headers * [Project] Make unescape code public for unit testing * [Project] More fixes for closed tags * [Project] More fixes to calculations * [Project] Rework API for the modified headers * [Project] Rework html visibility rule * [Project] Skeleton of the css library * [Project] Start headers modification API structure * [Project] Start working on AWS Lua API * [Project] Use lua_mime to modify headers * [Project] Use modified headers on dkim signing * [Project] Use string_view to constexpr variant unpacking * [Rework] Add composites manager concept * [Rework] Add tags definitions * [Rework] Allow C code to be compiled with C++ compiler * [Rework] Clickhouse: Store url flags * [Rework] Composites: Rewrite the composites logic * [Rework] Composites: Start rework of the composites framework * [Rework] Dmarc: Move check policy function to the common utils * [Rework] Dmarc: Rework reports keys structure * [Rework] Further work to make html content private * [Rework] Html/CSS: Remove css C bindings as they are useless now * [Rework] Html/CSS: Rework Lua bindings * [Rework] Html/Css: Start rework of the html blocks * [Rework] Html: Add images processing logic * [Rework] Html: Add traverse function * [Rework] Html: Another steps to get rid of gnode * [Rework] Html: Convert to variant * [Rework] Html: Deal with the utf_content part * [Rework] Html: Final rework part for the html processing code * [Rework] Html: Fix Lua bindings * [Rework] Html: Forgot to add the internal include * [Rework] Html: Further html urls rework * [Rework] Html: Further rework of the tags content extraction * [Rework] Html: Make parameters as a vector again * [Rework] Html: Move blocks part * [Rework] Html: Move images processing stuff * [Rework] Html: Rework lua bindings * [Rework] Html: Start html text extraction rework * [Rework] Html: Start refactoring of the html tags handling * [Rework] Html: Start removing of GNode stuff * [Rework] Html: Start rework of the html content structure * [Rework] Lua_magic: Try to detect text parts with 8bit characters for non-utf8 encodings * [Rework] Move HTML url functions and rework them * [Rework] Move and adopt entities handling logic * [Rework] Move common and rarely used dmarc code to the library * [Rework] Move compression routines outside of rspamd_util library * [Rework] Move entities/tags handling * [Rework] Phishing: Split from redirectors usage * [Rework] Redesign html blocks propagation logic * [Rework] Remove tag name string * [Rework] Rename phished url to a linked url * [Rework] Reorganize dmarc plugin and remove unsupported reporting code * [Rework] Reputation: Use more flexible types in get/set functions * [Rework] Require proper C++ environment for Rspamd build * [Rework] Rework extended urls output * [Rework] Rework tags parsing machine * [Rework] Slightly improve old regexp API * [Rework] Start conversion of the redis pool code to c++ * [Rework] Try to resolve failed upstreams more agressively * [Rework] Use C++ utf8 library with unit tests to trim whitespaces * [Rework] Use C++ version for unicode normalisation * [Rework] Use C++ version of the lua threads pool * [Rules] Add raw addresses to MULTIPLE_FROM options * [Rules] Another fix to HTTP_TO_HTTPS rule * [Rules] Do not trigger HTML_SHORT_LINK_IMG on external images * [Rules] Extend FORGED_X_MAILER * [Rules] Extend OLD_X_MAILER * [Rules] Fix CTYPE_MIXED_BOGUS for text attachments * [Rules] Fix FPs for CTYPE_MIXED_BOGUS * [Rules] Fix HTTP_TO_HTTPS rule * [Rules] Fix HTTP_TO_HTTPS rule * [Rules] Fix zerofont rule (partially) * [Rules] Micro-optimize X_PHP_EVAL * [Rules] Reduce default weight for R_MISSING_CHARSET Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	5 years ago
Vsevolod Stakhov	4c56c1210d	Release 2.7 * [Conf] Add R_DKIM_PERMFAIL to the metric * [CritFix] Dkim: Fix simple canonicalisation if multiple signatures are presented * [CritFix] Fix controller paths normalisation * [Feature] Add INVALID_DATE rule * [Feature] Add controller endpoint for training neural * [Feature] Add sanity checks for actions thresholds * [Feature] Add support of '==' and '!=' in Rspamd expressions * [Feature] Composites: Improve composite atoms parser * [Feature] Docker: use Debian slim variant * [Feature] Elastic: Add some missing fields * [Feature] Extract text from img alt attributes * [Feature] Improve charset detection logic * [Feature] Lua_clickhouse: Add optional row callback for large selections * [Feature] Lua_dns_resolver: Add idna_convert_utf8 method * [Feature] Lua_mime: Add ability to do multipattern replacement * [Feature] Lua_trie: Allow to report start of the match * [Feature] Multimap: support adding map values as extra options * [Feature] Neural: Move PCA learning to a subprocess * [Feature] RBL: support matching content/image URLs only * [Feature] RBL: support use of multiple selectors * [Feature] Reputation: Allow to specify ip masks * [Feature] Support SMIME signed messages container * [Feature] Support multiple conditions for symbols * [Feature] Support ping in milter mode * [Feature] Support rspamd_text in selector regexps * [Feature] Use own daemonization routine * [Feature] Vadesecure: Implement settings_outbound feature as recommended by Vade * [Feature] `rspamadm clickhouse` command * [Feature] allow hyperscan for aarch64 * [Fix] Allow to set priorities between post init scripts * [Fix] Allow to use maps for strings that are not zero terminated * [Fix] Apply max_lua_urls limit for emails as well * [Fix] Arc: Fix CV check on signing * [Fix] Arc: Fix signing of the broken ARC chains * [Fix] Clickhouse: escape carriage return * [Fix] Composites: Allow partial match * [Fix] Deduct type of a table methods * [Fix] Do not load errored hyperscan database * [Fix] Do not process links in ignored html tags * [Fix] Fix ClamAV result for cached encrypted file (#3395) * [Fix] Fix canonicalisation when l= tag is presented * [Fix] Fix flag shift * [Fix] Fix handling of skip/skip_process http flags * [Fix] Fix html attachments checks * [Fix] Fix issue with pushing binary formats to Lua strings * [Fix] Fix logging for rspamadm * [Fix] Fix off-by-one with init check * [Fix] Fix parsing of escape characters in quoted pairs * [Fix] Fix pushing ucl strings with \0 inside * [Fix] Fix quoted-printable soft newlines bugged case * [Fix] Fix settings in case actions are set to null (#3415) * [Fix] Fix several issues with auth results producing * [Fix] Fix smtp comments exclusion * [Fix] Fix smtp date syntax definition * [Fix] Fix substring search in case if srchlen == inlen * [Fix] Fix text selectors * [Fix] Honour `systemd` setting when logging to console (#3514) * [Fix] Html: Add entities collisions prevention logic (e.g. for mathml entities) * [Fix] Lua_auth_results: Quote potentially bad values in AR header * [Fix] Multimap: Fix flags usage * [Fix] Multimap: Fix scoring for combined maps * [Fix] Plug GList * leak in redis pool * [Fix] RBL: allow for multiple matches of the same label if types are different * [Fix] Rely on libev checks for file maps * [Fix] Restore simple dkim canonicalisation mode * [Fix] Return MimeCharset as we work with emails... * [Fix] Spamassassin: Fix pcre_only flags * [Fix] Spamassassin: Preserve 'pcre_only' flag when dealing with regexp replacements * [Fix] Try to fix GError leak * [Fix] Try to fix a mess with settings loading by adding priorities * [Fix] Try to move setings initialisation to a later stage * [Fix] Use dup fd in milter handler to avoid races with the proxy * [Fix] Use message pointer to avoid obsolete data to be cached * [Project] Rbl: Migrate to `checks` * [Project] Rbl: Move config code outside of the plugin * [Project] Ressurect empty prefilters as connection filters * [Project] Support connection filters registration from Lua * [Rework] Add final cleanup logic * [Rework] Add preliminary support of hyperscan caching for re maps * [Rework] Add stale cache removal * [Rework] Clickhouse: Improve performance * [Rework] Distinguish between strict config test mode * [Rework] Furhter logging improvements * [Rework] Milter_headers: improve extended_headers_rcpt support * [Rework] Move parsers to a separate lua library * [Rework] Neural: Skip composite symbols * [Rework] Rbl: Rework defaults logic * [Rework] Some tunes to cache saving * [Rework] Track maps origins * [Rework] Use full crypto hash for regexp maps * [Rules] Remove broken rule Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	6 years ago
Vsevolod Stakhov	eec4438705	Release 2.6 * [Conf] Add missing symbols * [Conf] Add missing symbols * [Conf] Fix fat-fingers typo * [Conf] Fix wrong comment in options.inc * [Conf] Neural: Fix the default name for max_trains * [Conf] Register a known symbol * [Conf] Spf: Add R_SPF_PERMFAIL symbol * [CritFix] Arc: Fix ARC validation for chains of signatures * [CritFix] Distinguish socketpairs between different fuzzy workers * [CritFix] Fix IDNA dots parsing * [CritFix] Fix test assertion method * [CritFix] Fix usage of crypto_sign it should be crypto_sign_detached! * [Feature] Add BOUNCE rule * [Feature] Add controller plugins support and selectors plugin * [Feature] Add maps query method * [Feature] Add minimal delay to fuzzy storage * [Feature] Add multiple base32 alphabets for decoding * [Feature] Add preliminary support of BCH addresses * [Feature] Add query_specific endpoint * [Feature] Allow multiple base32 encodings in Lua API * [Feature] Allow to specify nonces manually * [Feature] Controller: Allow to pass query arguments to the lua webui plugins * [Feature] Fuzzy_check: Add gen_hashes command * [Feature] Fuzzy_check: Add weight_threshold option for fuzzy rules * [Feature] Implement address retry on connection failure * [Feature] Improve limits in pdf scanning * [Feature] Initial support of subscribe command in lua_redis * [Feature] Lua_cryptobox: Add secretbox API * [Feature] Lua_text: Add encoding methods * [Feature] Milter_headers: Allow to activate routines via users settings * [Feature] PDF: Add timeouts for expensive operations * [Feature] Preliminary maps addon for controller * [Feature] Split pdf processing object and output object to allow GC * [Feature] Support BLIS blas library * [Feature] Support input vectorisation by recvmmsg call * [Feature] Support multiple base32 alphabets * [Feature] add queueid, uid, messageid and specific symbols to selectors [Minor] use only selectors to fill vars in force_actions message * [Feature] allow variables in force_actions messages * [Feature] extend lua api * [Fix] #3249 * [Fix] Allow to adjust neurons in the hidden layer * [Fix] Another try to fix email names parsing * [Fix] Arc: Allow to reuse authentication results when doing multi-stage signing * [Fix] Arc: Fix bug with arc chains verification where i>1 * [Fix] Arc: Sort headers by their i= value * [Fix] Change neural plugin's loss function * [Fix] Deal with double eqsigns when decoding headers * [Fix] Default ANN names in clickhouse * [Fix] Disable reuseport for TCP sockets as it causes too many troubles * [Fix] Disable text detection heuristics for encrypted parts * [Fix] Distinguish DKIM keys by md5 * [Fix] Distinguish type from flags in register_symbol * [Fix] Dmarc: Unbreak reporting after cf2ae3292a * [Fix] Do not flag pre-result of virus scanners as least if action is reject * [Fix] Do not use GC64 workaround on 32bit platforms, omg * [Fix] Exclude damaged urls from html parser * [Fix] Fix FREEMAIL_REPLYTO_NEQ_FROM_DOM * [Fix] Fix FROM_NEQ_ENVFROM * [Fix] Fix FWD_GOOGLE rule (#1815) * [Fix] Fix adding of the empty archive file for gzip * [Fix] Fix aliases in forged recipients and limit number of iterations * [Fix] Fix authentication results insertion * [Fix] Fix calling of methods in selectors * [Fix] Fix clen length for hiredis... * [Fix] Fix endless loop if broken arc chain has been found * [Fix] Fix false - operation * [Fix] Fix get_urls table invocation * [Fix] Fix group based composites * [Fix] Fix headers passing in rspamd_proxy * [Fix] Fix incomplete utf8 sequences handling * [Fix] Fix lua_next invocation * [Fix] Fix lua_parse_symbol_type function logic * [Fix] Fix multiple listen configuration * [Fix] Fix occasional encryption of the cached data * [Fix] Fix parsing boundaries with spaces * [Fix] Fix passing of methods arguments * [Fix] Fix poor man allocator algorithm * [Fix] Fix regexp selector and add flattening * [Fix] Fix rfc base32 encode ordering (skip inverse bits) * [Fix] Fix rfc based base32 decoding * [Fix] Fix sockets leak in the client * [Fix] Fix storing of the original smtp from * [Fix] Fix types check and types usage in lua_cryptobox * [Fix] Fix unused results * [Fix] Fuzzy_check: Disable shingles for short texts (really) * [Fix] Ical: Fix identation grammar * [Fix] Improve part:is_attachment logic * [Fix] Mmap return value must be checked versus MAP_FAILED * [Fix] One more fix to skip images that are not urls * [Fix] Pdf: Support some weird objects with no newline before endobj * [Fix] Rbl: Fix ignore_defaults in conjunction with ignore_whitelists * [Fix] Restore support for `for` and `id` parts in received headers * [Fix] Segmentation fault in contrib/lua-lpeg/lpvm.c on ppc64el * [Fix] Skip spaces at the boundary end * [Fix] Slashing fix: fix captures matching API * [Fix] Spamassassin: Rework metas processing * [Fix] Store reference of upstream list in upstreams objects * [Fix] Understand utf8 in content-disposition parser * [Fix] Unify selectors digest functions * [Fix] Use `abs` value when checking composites * [Fix] Use strict IDNA for utf8 DNS names + add sanity checks for DNS names * [Fix] Use unsigned char and better support of utf8 in ragel parser * [Fix] add missing selector_cache declaration * [Project] Add `L` flag for regexps to save start of the match in Hyperscan * [Project] Add `lower` method to lua_text * [Project] Add a simple matrix Lua library * [Project] Add implicit bitcoincash prefix * [Project] Add linalg ffi library for prototyping * [Project] Add methods to append data to fuzzy requests * [Project] Add routine to call a generic lua function * [Project] Add ssyev method interface * [Project] Add tensors index method * [Project] Add text:sub method * [Project] Allow rspamd_text based selectors * [Project] Allow to specify re_conditions for regular expressions * [Project] Attach extensions to the binary fuzzy commands * [Project] Bitcoin: BTC cash addresses needs some checksum validation * [Project] Cleanup the redis script * [Project] Convert bitcoin rules to the new regexp conditions feature * [Project] Detect memrchr in systems that supports it * [Project] Do not listen sockets in the main process * [Project] Implement 'probabilistic' learn mode for ANN * [Project] Implement BTC polymod in C as it requires 64 bit ops * [Project] Implement bitcoin cash validation in a proper way * [Project] Implement extensions logic for fuzzy storage * [Project] Implement symbols insertion in multiple results mode * [Project] Lua_text: Add method memchr * [Project] Neural: Add PCA loading logic * [Project] Neural: Fix PCA based learning * [Project] Neural: Fix matrix gemm * [Project] Neural: Further PCA fixes * [Project] Neural: Implement PCA in learning * [Project] Neural: Implement PCA learning * [Project] Neural: Implement PCA on ANN forward * [Project] Neural: Implement PCA serialisation * [Project] Neural: Start PCA implementation * [Project] Neural: Use C version of scatter matrix producing * [Project] Preliminary support of lua conditions for regexps * [Project] Preliminary usage of the reuseport * [Project] Process composites separately for each shadow result * [Project] Remove old code * [Project] Rework scan result functions to support shadow results * [Project] Rework some more functions to work with shadow results * [Project] Some more fixes * [Project] Start results chain implementation * [Project] Support fun iterators on rspamd_text objects * [Project] Support multiply, minus and divide operators in expressions * [Project] Tensor: Move scatter matrix calculation to C * [Rework] Allow to specify exat metric result when adding a symbol * [Rework] Change and improve openblas detection and usage * [Rework] Close listen sockets in main after fork * [Rework] Further rework of lua urls extraction API * [Rework] Lua_cryptobox: Allow to store output of the hash function * [Rework] Lua_task: Add more methods to deal with shadow results * [Rework] Modernize logging for expressions * [Rework] Remove empty prefilters feature - we are not prepared... * [Rework] Remove old FindLua module, disable lua fallback when LuaJIT is enabled * [Rework] Rework and refactor forged recipients plugin * [Rework] Rework expressions processing * [Rework] Rework fuzzy commands processing * [Rework] Rework url flags handling API * [Rework] Rework urls extraction * [Rework] Split operations processing and add more debug logs * [Rework] Update zstd to 1.4.5 * [Rework] Use google-ced instead of libicu chardet as the former sucks * [Rework] add alias util:parse_addr for util:parse_mail_address * [Rework] get rid of util:parse_addr duplicating the util:parse_mail_address, replace where used * [Rules] Allow prefix for bitcoin cash addresses * [Rules] More fixes for bitcoin cash addresses decoding * [Rules] Refactor bleach32 addresses handling Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	6 years ago
Vsevolod Stakhov	61cbcd0483	Release 2.5 * [Conf] Mark Rspamd emailbl as ignore whitelist * [Conf] RBL: Add missing emails = true option * [Feature] Add support for scripts in fuzzy storage * [Feature] Arc: Add whitelisted_signers_map option * [Feature] Implement hosts file processing * [Feature] Neural: Introduce classes bias that allows non-equal classes learning * [Feature] Update libev to 4.33 * [Fix] Another brain damage html standard adoptions * [Fix] Another fix for brain damaged obs-fws state * [Fix] Fix flags that caused force_actions failure * [Fix] Fix logging issue * [Fix] Fix lua symbols scores registration when config does not define scores * [Fix] Fix opaque maps logic * [Fix] Fix parsing of the html tags with no spaces after attributes * [Fix] Fix some corner cases in urls parsing, add limits * [Fix] Fix tlds extraction if custom composition rules are used * [Fix] Fix variables replacement in mempool * [Fix] Improve base64 detection * [Fix] Normalize dynamic scores in ANN correctly * [Fix] Plug memory leak introduced by #3153 * [Fix] Stat_redis_backend: Fix memory leak and simplify learn path * [Fix] Try hard to deal with ghost workers * [Fix] metadata_exporter default formatter * [Rework] Change the way to extract URLs when dealing with alternative parts * [Rework] Fix various url extraction issues * [Rework] Re cache: Load compiled hyperscan in the main process as well * [Rework] Re cache: Load hyperscan early * [Rework] Rework URL structure: adjust tld part * [Rework] Rework URL structure: host field * [Rework] Rework URL structure: more structure optimisations * [Rework] Rework URL structure: user field * [Rework] URL: Another update for urls extraction logic * [Rework] Urls: Improve query urls handling * [Rework] Urls: adopt html related stuff * [Rework] Urls: more rework of the urls sets * [Rework] Urls: process query urls in HTML urls correctly * [Rework] Urls: rework urls hash structure * [Rework] Urls: update lua libraries * [Rework] Use multiple search tries for different url extraction types Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	6 years ago
Vsevolod Stakhov	a74e458e15	Release 2.4 * [CritFix] Fix parsing of the content type attributes * [Feature] Clickhouse: Add extra columns support * [Feature] Rbl: Add url_compose_map option for RBL rules * [Fix] 'R' flag is for all headers regexp * [Fix] Allow to reset settings id from Lua (e.g. because of the priority) * [Fix] Avoid collisions in mempool variables by changing fuzzy caching logic * [Fix] Avoid strdup usage for symbols options * [Fix] Do not trust stat(2) it lies * [Fix] Filter all options for symbols to have sane characters * [Fix] Fix all headers iteration * [Fix] Fix allowed_settings for neural * [Fix] Fix listen socket parsing * [Fix] Fix maps expressions evaluation * [Fix] Fix sentinel connections leak by using async connections * [Fix] Fix smtp message on passthrough result * [Fix] Fix tld compositon rules * [Fix] Fuzzy_storage: Do not check for shingles if a direct hash has been found * [Fix] Lua_mime: Do not perform QP encoding for 7bit parts * [Fix] Neural: Distinguish missing symbols from symbols with low scores * [Fix] Support listening on systemd sockets by name * [Project] Add lua_urls_compose library * [Project] Allow to set a custom log function to the logger * [Project] CDB maps: Start making cdb a first class citizen * [Project] Clickhouse: Add extra columns concept * [Project] Fix urls composition rules, add unit tests * [Project] Unify cdb maps * [Rework] Logger infrastructure rework * [Rework] Refactor libraries structure * [Rework] Rework SSL caching * [Rework] Update snowball stemmer to 2.0 and remove all crap aside of UTF8 Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	7 years ago
Vsevolod Stakhov	11a3b09a5d	Release 2.3 * [Conf] SPF is no longer a C module * [Conf] Update spamtrap map path example * [CritFix] Fix html entities decoding * [CritFix] Fix re cache when mix of pcre and hyperscan is used * [Feature] Allow milter code to deal with multiple headers * [Feature] Antivirus: Add avast support * [Feature] Dkim_signing: Allow to sign via milter_headers * [Feature] Implement content hashes * [Feature] Lua_text: Add regexp split iterator method * [Feature] Lua_text: Implement flattening of the input tables * [Feature] Send quit command to Redis * [Feature] Speed up is_ascii function * [Feature] Spf: Add external_relay option * [Fix] Avoid double escaping * [Fix] Fix O(N^2) algorithm * [Fix] Fix arc seal validation * [Fix] Fix base tag processing according to stupid HTML renderer behaviour * [Fix] Fix dealing with `\0` in ucl strings and JSON * [Fix] Fix gpg parts misdetection * [Fix] Fix ignored symbols exporting * [Fix] Fix processing of numeric url's * [Fix] Fix processing of the closed tcp connections * [Fix] Fix regexp type check for pcre2 * [Fix] Fix urls encode function * [Fix] Fix urls shifting when doing decode to include separators * [Fix] Fix white on white rule and add is_leaf flag * [Fix] Further fixes in charset detection * [Fix] Ignore diacritics in chartable module for specific languages * [Fix] Limit size of symbols options by max_opts_len option * [Fix] More fixes in html tag content calculations * [Fix] Plug memory leak in fuzzy storage * [Fix] Process high priority settings even if settings/id has been specified * [Fix] Select a different upstream on last retransmit * [Fix] Treat soft hyphen as zero width space * [Fix] Try harder to watch the lifetime of the key_stat * [Fix] Use ipv6-mapped-ipv4 addresses in radix trie * [Project] Add logic to break execution when processing symbols* * [Project] Add methods to set specific content for mime parts from Lua * [Project] Lua_content: support PDF files * [Project] Move dns_tool to using of the rspamd_spf from FFI module * [Project] Preliminary SPF plugin in Lua * [Project] Show debug stat for memory pool * [Project] Some rework about specific data that is now tagged * [Project] Start reworking of the mempool structure * [Rework] Allow to add userdata as symbols options * [Rework] Change mime part specifics handling * [Rework] Move LRU SPF cache from spf plugin * [Rework] Rework HTML tags content attachment * [Rework] Rework options hash structure * [Rework] Start lua_content library * [Rework] Stop using of uthash for http headers * [Rework] Use faster hashing approach for memory pools variables * [Rules] Add PDF related rules Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	7 years ago
Vsevolod Stakhov	f87b6eb314	Release 2.2 * [Conf] Antivirus: Fix the default config * [Feature] Add verdict library in lua * [Feature] Allow exception when choosing upstream * [Feature] Allow to disable symbols from the metric config * [Feature] Allow to limit maps per specific worker * [Feature] Always validate Rspamd protocol output * [Feature] Antivirus: Add preliminary virustotal support * [Feature] Clickhouse: Rework Clickhouse collection logic * [Feature] Improve base64 usage * [Feature] Shutdown timeout is now associated with task timeout * [Fix] #3129 Multiple classifiers on redis working incorrectly * [Fix] Allow real upstreams configuration * [Fix] Another try to fix slow callbacks and timers * [Fix] Check results of write message as SSL can bork them * [Fix] Clickhouse: Avoid potential races in collection * [Fix] Clickhouse: Fix periodic script * [Fix] Fail DNS upstream on each retransmit attempt * [Fix] Fix consistent hashing when upstreams are marked inactive * [Fix] Fix issues found * [Fix] Fix off-by-one in retries for the proxy * [Fix] Fix termination * [Fix] Fix upstreams exclusion logic * [Fix] Fix utf8 validation for symbols options and empty strings * [Fix] Oops, fix maps reload * [Fix] Rbl: Allow utf8 lookups for IDN domains * [Fix] Sigh, another try to fix brain-damaged openssl * [Project] Add fast utf8 validation library * [Project] Use own utf8 validation instead of glib * [Rework] Another phase of finish actions rework * [Rework] Further cmake system rework * [Rework] Further isolation of the controller's functions * [Rework] Make cmake structure more modular * [Rework] Move cmake modules to a dedicated path * [Rework] Replace controller functions by any scanner worker if needed * [Rework] Rework final scripts logic * [Rework] Rewrite rspamd_str_make_utf_valid function Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	7 years ago
Vsevolod Stakhov	601e28d2ea	Release 2.1 * [Conf] Update neural.conf * [CritFix] Fix dkim verification for multiple headers listed * [Feature] Add support of uudecode * [Feature] Allow to explicitly set events backend * [Feature] Implement configurable limits for SPF lookups * [Feature] Lua_scanners: Use lua magic for inclusion/exclusion logic * [Feature] Multimap: Do not check files in office archives * [Feature] Neural: Add sampling when storing training vectors * [Feature] SPF: Allow to disable AAAA checks in configuration * [Feature] Spf: Add limits configuration support * [Feature] Store etag in cached HTTP maps + better logging * [Feature] Support segwit BTC addresses, fix LTC verification * [Feature] Support uuencoding * [Fix] Add configurable number of threads for OpenBLAS * [Fix] Add workaround for ragel 7 in hyperscan related maps code * [Fix] Another fix for numeric urls parsing * [Fix] Correct EMA time calculations * [Fix] Do not treat archives as text * [Fix] Do not use strdup on data extracted from lua * [Fix] Fix a failure calcuating URL reputation. * [Fix] Fix crash due to constructors init order * [Fix] Fix crash on parts with no cd * [Fix] Fix empty prefilters that require mime structures * [Fix] Fix event loop creation * [Fix] Fix issues sending DMARC reports. * [Fix] Fix misprint * [Fix] Fix saving of the file maps * [Fix] Fix size calculations when converting from utf16 * [Fix] Fix support of disable_monitoring in rbl * [Fix] Fix use-after-free * [Fix] Fix zip files check to relax requirements * [Fix] Important hiredis fixes * [Fix] Lot's of fixes in maps check logic * [Fix] Lua_tcp: Deal with temporary fails on write * [Fix] Lua_tcp: Make write errors fatal and rework error handlers * [Fix] Meta: Filter some more values * [Fix] Neural: Add protection agains infinities * [Fix] Oops, fix math.huge invocation * [Fix] Plug memory leak * [Fix] Sigh, another email to string fix * [Fix] Try to fix another ownership race in ssl connection * [Fix] Uuencode: Fix parsing of corrupted uuencode * [Fix] lua_scanners - razor rename need_check function * [Rework] Require CMake 3.9 to work, remove manual lto crap Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	7 years ago
Vsevolod Stakhov	8d31adb5cb	Release 2.0 * [Conf] Add BROKEN_HEADERS_MAILLIST composite * [Conf] Add path to greylist-whitelist-domains.inc * [Conf] Clarify documentation in the config files * [Conf] Introduce maps.d directories * [Conf] Log settings id by default * [Conf] Make LEAKED_PASSWORD_SCAM a composite rule again * [Conf] Move all surbl/emails rules to rbl * [Conf] Register new Spamhaus codes * [Conf] Remove configs for deleted modules * [Conf] Remove surbl parts, fix hash_format attribute * [Conf] Show autolearn sample * [Conf] Slashing: Change default stats backend to Redis * [Conf] Surbl: Utilise new `check_emails` option * [Conf] Update header * [Conf] Use multi-prefixes RBLs in the default config * [CritFix] Deal with case-sensivity in Content-Disposition parser * [CritFix] Eliminate old endpoint * [CritFix] Fix case sensivity when parsing Content-Type * [CritFix] Fix loading of DKIM public keys * [CritFix] Fix procesing of urls * [CritFix] Fix whitelisting when both spf and dkim are required to be valid * [CritFix] Langdet: Fix language detection where no stop words found * [Feature] Add description to the groups * [Feature] Add limit for number of URLs in Lua * [Feature] Add logging of groups to the log_format * [Feature] Add lua_smtp library * [Feature] Add maps cache and type refinement * [Feature] Add p0f scanner * [Feature] Adopt emails module to use lua_maps * [Feature] Allow options matching in composites * [Feature] Allow selectors in rbl module * [Feature] Allow to output group results * [Feature] Asn: Allow to use bgpdump when NET::MRT is broken * [Feature] Calculate tokens occurrences distribution * [Feature] Clickhouse: Add authenticated user and settings id columns * [Feature] Clickhouse: Store groups data * [Feature] Clickhouse: Utilise LowCardinality feature * [Feature] Implement Redis prefixes registration logic * [Feature] Implement settings id propagation between deps * [Feature] Improve AV results caching * [Feature] Improve autolearning * [Feature] Improve logging locking logic (remove it actually) * [Feature] Improve settings processing * [Feature] Langdet: Limit number of stop words to be checked * [Feature] Libucl: Allow to sort keys in ucl objects * [Feature] Lua_config: Extend get symbols method * [Feature] Lua_maps: Allow static maps for key-value pairs * [Feature] Lua_mimepart: Add function filter_words * [Feature] Lua_selectors: Add `words` selector * [Feature] Lua_selectors: Add sort and uniq transform functions * [Feature] Lua_selectors: Allow table arguments for selectors * [Feature] Lua_tcp: Add preliminary support of SSL connections * [Feature] Lua_trie: More flexible API * [Feature] Lua_util: Add filter_specific_url function * [Feature] Lua_util: table_digest can now recursively traverse tables * [Feature] Maillist: Improve detection * [Feature] Maps: Allow caching for complex maps * [Feature] Monitored: Support random lookups * [Feature] Multimap: Add combined maps prototype * [Feature] Multimap: Add dependend maps via redis keys selectors * [Feature] Multimap: Allow multiple email addresses matches * [Feature] Multimap: Also check detected charset when do filename checks * [Feature] Output number of messages processed to proctitle * [Feature] Perform clean SSL shutdown * [Feature] Performance: Do not use base64 SIMD version for bad inputs * [Feature] RBL: Support bit results in replies * [Feature] RBL: Support type specific prefixes * [Feature] Ratelimit: Consider number of SMTP recipients * [Feature] Rbl: Add ability to check urls * [Feature] Rbl: Add resolve_ip based RBLs * [Feature] Rbl: Make config checks much more strict * [Feature] Rbl: Support per-rule whitelists * [Feature] Rbl: Support process script * [Feature] Rbl: Support replyto addresses * [Feature] SURBL: Allow to check email domains * [Feature] Selectors: Add `list` generator * [Feature] Selectors: Add `specific_urls` extractor * [Feature] Selectors: Add flatten function * [Feature] Selectors: Support filter_map and apply_map functions * [Feature] Store Clickhouse data outside of lua alloc * [Feature] Support caching for encrypted files and macros * [Feature] Support images when extracting urls * [Feature] Support more hyperscan flags * [Feature] Support protocol flags * [Feature] URL: Apply stringprep to hostnames to filter garbage * [Feature] Upstreams: Add lazy resolving logic to all upstreams * [Feature] Upstreams: Set noresolve flag on numeric upstreams * [Feature] Use `scores` in apply section * [Feature] Use maps logic from lua_maps for multimap * [Feature] Use random monitored in rbl module * [Feature] lua_scanners - add Razor support * [Fix] Add another safe-guard in urls processing * [Fix] Add debug to ssl, fixed write hangs * [Fix] Add missing groups to C callback symbols * [Fix] Add more checks for ghosts symbols * [Fix] Allow to enable or add new actions via settings * [Fix] Allow to set 0 size for spf/dkim caches * [Fix] Another bunch of fixes towards protocol mess * [Fix] Another fix to deal with bad URLs * [Fix] Arc: Another bunch of fixes for arc signing * [Fix] Arc: More arc signing fixes * [Fix] Avoid another overflow in fpconv * [Fix] Clickhouse: Fix quoting * [Fix] Clickhouse: Fix retention query quoting * [Fix] Distinguish empty and non-empty prefilters * [Fix] Distinguish remote and local addrs parsing * [Fix] Do not assert if length of sig is bad, just fail verification * [Fix] Do not assert if we have broken mime boundary in the headers * [Fix] Do not call implicit strlen to avoid issues * [Fix] Do not count images urls when checking url regexps for compatibility * [Fix] Do not output rbl suffix in symbol option * [Fix] Do not use config pool to avoid issues with double reload * [Fix] Do not use ephemeral string * [Fix] Do not use lightuserdata for traceback * [Fix] Do not use priority in metric registration * [Fix] Emails: Check email sanity before testing on BL * [Fix] Emails: Fix misprint in key name * [Fix] Escape utf in regexp to dodge ragel/hyperscan issue * [Fix] Extend task_timeout to postfilters stage * [Fix] Fix ARC signing after fixing another bug in it... * [Fix] Fix AV scan logic * [Fix] Fix DMARC_NA behaviour in case of no valid policies * [Fix] Fix LRU hash iteration logic * [Fix] Fix alignment mess * [Fix] Fix configuring symbols without scores * [Fix] Fix disabling of the actions * [Fix] Fix dkim signing exceptions * [Fix] Fix embedded images linking logic * [Fix] Fix events leak * [Fix] Fix eviction corner case * [Fix] Fix fuzzy image score calculation #2962 * [Fix] Fix hang in fuzzy_learn when explicit rotation is set * [Fix] Fix headers propagation logic * [Fix] Fix hearbeats restart issue * [Fix] Fix history reset * [Fix] Fix log parameter * [Fix] Fix lua_ip_equal logic * [Fix] Fix more issues with nested messages + tests * [Fix] Fix normalization of non-alphabet based languages * [Fix] Fix offsets when parsing message/rfc822 in multipart * [Fix] Fix options in rbl symbols * [Fix] Fix out of bound access in lua logger * [Fix] Fix out-of-bound read in qp decode * [Fix] Fix parent CTE propagation * [Fix] Fix parsing of the received headers with empty part * [Fix] Fix pending checks for events * [Fix] Fix printing of NULL pointer with fixed length * [Fix] Fix race condition in watcher handler * [Fix] Fix read-after-end in quoted printable decoding * [Fix] Fix redis sentinel support * [Fix] Fix registry leak in case of DNS errors * [Fix] Fix reload logic * [Fix] Fix sending of large entries via HTTPS * [Fix] Fix settings reload * [Fix] Fix some more corner cases for fpconv * [Fix] Fix trie code when there are regexps and Hyperscan is absent * [Fix] Further fixes to printing of the FP numbers * [Fix] Fuzzy_check: Fix timeouts * [Fix] Grrr, fix empty ip case * [Fix] Html: Fix processing of fjlig entity * [Fix] Lang_det: Try better to distinguish Chinese and Japanese * [Fix] Lua_mime: Fix reversed extensions map * [Fix] Lua_task: Fix message-less API * [Fix] Lua_tcp: Report connection failures * [Fix] Lua_tcp: Various fixes and debugging improvements * [Fix] Metadata_exporter: This plugin is idempotent not a postfilter * [Fix] More fixes to extract_specific_urls * [Fix] More stages fixes * [Fix] Neural: Another bunch of fixes * [Fix] Neural: use version in ANN key profile * [Fix] Postpone lua state destruction to allow lua dtors to be used * [Fix] Prefer surbl/emails rule on rbl to preserve compatibility * [Fix] RBL: Fix behaviour of emails_domainonly * [Fix] Ratelimit: Fix dynamic score * [Fix] Rbl: Fix emailbl functions * [Fix] Really fix hyperscan workaround * [Fix] Set sanity limits for pcre2 * [Fix] Settings: Fix settings check flags * [Fix] Sort keys when getting data from Lua when filling rules * [Fix] Statistics: Do not query Redis tokens when there are no learns * [Fix] Stop IO event on write finished in http connection * [Fix] Use heuristically detected text parts data * [Fix] Various fixes to QP encoding algorithm * [Fix] Various fixes to SSL state machine handler * [Fix] Various fixes to asn module * [Fix] Workaround for empty charset in rfc2231 encoding * [Project] Switch from torch to KANN * [Project] Add heartbeat events * [Project] Add preliminary support of the Kaspersky Scan Engine * [Project] Add preliminary version of maps expressions * [Project] Add preprocessed settings to the config structure * [Project] Add simple forward propagation function * [Project] Add small helpers for migration simplifications * [Project] Allow to replace body in milter * [Project] Bundle libev * [Project] First refactoring step libevent->libev * [Project] Implement syntax highlighting for Lua * [Project] Lua_magic: Adopt lua_magic stuff in mime_types * [Project] Remove libfann, gd and other unsupported stuff * [Project] Remove torch * [Project] Rework upstreams * [Rework] Allow execution of async events when hs compiles regexps * [Rework] Bayes expiry: eliminate `default` expiration mode * [Rework] Dkim: Remove signing code * [Rework] Dkim_signing: Move sign condition to dkim_signing * [Rework] Do not lowercase all data send to ClickHouse * [Rework] Drop url tags * [Rework] Eliminate lua_squeeze as it has shown no improvements * [Rework] Eliminate virtual scan time as it is useless * [Rework] Lua core: Use lightuserdata to index classes * [Rework] Lua_util: Another rework for extract_specific_urls * [Rework] Migrate from ip_score to reputation * [Rework] Move mime modification functions to lua_mime library * [Rework] Rbl: Major whitelisting logic rework * [Rework] Remove deprecated plugins * [Rework] Remove log helper worker * [Rework] Remove rspamd.classifiers.lua * [Rework] Rename filter.h to a more sane name * [Rework] Reorganise selectors implementation * [Rework] Replace linenoise with replxx * [Rework] Reputation: Remove ipnet from the ip reputation * [Rework] Reputation: Slashing - change name of symbols * [Rework] Rework children operations * [Rework] Rework config reload * [Rework] Rework expression API * [Rework] Rework image urls processing * [Rework] Rework initialisation to reduce static leaks count * [Rework] Rework request headers processing * [Rework] Slashing: Change versioning schema - move to 2.0 * [Rework] Slashing: Turn off postfilters when passthrough result is set * [Rework] Start moving to replxx * [Rework] Stop support of signed HTTP maps to simplify code * [Rework] Store ASN as UInt32 in ClickHouse * [Rework] Url_redirector: Rewrite plugin * [Rework] Use a dedicated library for autolearn * [Rework] Use libsodium instead of hand crafted crypto implementations * [Rework] Use opaque structure to store a table of mime headers * [Rules] Add dedicated bitcoin addresses filter rule * [Rules] Add more detection to LEAKED_PASSWORD_SCAM * [Rules] Catch LTC addresses * [Rules] Reduce weight of RSPAMD_EMAILBL * [Rules] Rework LEAKED_PASSWORD_SCAM rule one more time Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	7 years ago
Vsevolod Stakhov	cc1aef5ea4	[Minor] Add Changelog for 1.9.4	7 years ago
Vsevolod Stakhov	fe91396af3	Release 1.9.3 * [Conf] Add IP_SCORE_FREEMAIL composite rule * [Feature] Add cryptobox method to generate dkim keypairs * [Feature] Add fast hashes to lua cryptobox hash * [Feature] Add least passthrough results * [Feature] Allow oversign if exists mode * [Feature] Clickhouse: Modernise table initial schema * [Feature] Implement IUF interface for specific fast hashes * [Feature] Lua_util: Allow to obfuscate different fields * [Feature] Tune memory management in Rspamd and Lua * [Fix] Avoid buffer overflow when printing long lua strings * [Fix] Change the default oversigning headers to a more sane list * [Fix] Clickhouse: Do not store digest as it is not needed now * [Fix] Clickhouse: Fix lots of storage issues * [Fix] Clickhouse: Support custom actions * [Fix] Deny URLs where hostname is bogus * [Fix] Do not blacklist mail by SPF/DMARC for local/authed users * [Fix] Fix DoS caused by bug in glib * [Fix] Fix UCL parsing of the multiline strings * [Fix] Fix buffer overflow when printing small floats * [Fix] Fix init code for servers keypairs cache * [Fix] Fix issue with urls with no tld (e.g. IP) * [Fix] Fix memory in arc signing logic * [Fix] Fix memory leak in language detector during reloads * [Fix] Fix mixed case content type processing * [Fix] Fix processing of the ip urls in file * [Fix] Fix use after free * [Fix] HTML: Fix `size` attribute processing * [Fix] Hum, it seems that 99ff1c8 was not correct * [Fix] Lua_task: Fix task:get_from method * [Fix] Preserve fd when mapping file to scan * [Fix] Re-use milter_headers settings when doing arc signing * [Fix] Set dmarc force action as least action * [Fix] Switch to GMT * [Fix] allow PKCS7 signatures to be text/plain, too * [Project] Add initial version of the vault management tool * [Project] Add vault support for DKIM and ARC signing * [Project] Implement keys rotation in the vault * [Project] Improve dkim keys generation for vault * [Project] Improve keys creation in rspamadm vault * [Rework] Move lua_worker to a dedicated unit * [WebUI] Add URL fragments (#) support * [WebUI] Fix AJAX request URL Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	7 years ago
Vsevolod Stakhov	fb577bd5a3	Release 1.9.2 * [Conf] Allow to load users plugins from plugins.d * [Conf] oversign openpgp and autocrypt headers * [Feature] Add SPF FFI library for Lua * [Feature] Add more verbosity for SPF caching * [Feature] Antivirus: Handle encrypted files specially * [Feature] Clickhouse: Slashing - add new fields to CH * [Feature] Dkim_signing: Add OpenDKIM like signing_table and key_table * [Feature] Dkim_signing: Allow to use new options as maps * [Feature] Import fpconv library * [Feature] Lua_maps: Allow static regexp and glob maps * [Feature] Parse ical files * [Feature] Rspamadm: Add dns_tool utility * [Feature] Store SPF records digests * [Feature] Use fpconv girsu2 implementation for printing floats * [Fix] Clickhouse: Use integer seconds when inserting rows * [Fix] Fix floating point printing * [Fix] Fix processing of embedded urls * [Fix] Lua_clickhouse: Fix CH errors processing * [Fix] Make spf digest stable * [Fix] Properly detect encrypted files in zip archives * [Fix] Slashing: Store times in GMT timezone in ClickHouse * [Rules] Add additional conditions to perform BTC checks * [Rules] Fix pay-to-hash addresses validation Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	7 years ago
Vsevolod Stakhov	38ca7b5e10	Release 1.9.1 * [Conf] Add vendor groups for symbols * [Feature] Add `rspamadm template` command * [Feature] Allow to add messages from settings * [Feature] Allow unconnected DNS servers operations * [Feature] Check limits after being set, migrate to uint64 * [Feature] Greylist: Allow to disable greylisting depending on symbols * [Feature] Improve lua binary strings output * [Feature] Mime_types: Implement user configurable extension filters * [Feature] Mime_types: When no extension defined, detect it by content * [Feature] Preprocess config files using jinja templates * [Feature] Replies: Filter replies sender to limit whitelisting to direct messages * [Feature] Treat all tags with HREF as a potential hyperlinks * [Feature] Validate BTC addresses in LEAKED_PASSWORD_SCAM * [Fix] Add crash safety for HTTP async routines * [Fix] Another fix for Redis sentinel * [Fix] Clickhouse: Fix table schema upload * [Fix] Core: Fix squeezed dependencies handling for virtual symbols * [Fix] Finally fix default parameters parsing in actions section * [Fix] Fix ES sending logic (restore from coroutines mess) * [Fix] Fix finishing script for clickhouse collection * [Fix] Fix priority for regexp symbols registriation * [Fix] Fix various issues found by PVS Studio * [Fix] Initialize lua debugging earlier * [Fix] Neural: Fix training * [Fix] Rework cached Redis logic to avoid sentinels breaking * [Fix] SURBL: Fix regression in surbl module * [Fix] Fix double signing in the milter * [Project] Add support of HTTP proxy in requests * [Rework] Change lua global variables registration * [Rework] Rework HTML content urls extraction * [Rework] Start rework of aliasing in Rspamd * [WebUI] Combine Scan and Learning into one tab * [WebUI] Fix symbol score input type * [WebUI] Show grayed out pie * [WebUI] Update Throughput summary values dynamically Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	7 years ago
Vsevolod Stakhov	0e2437493e	Release 1.9.0 * [Conf] Add missing includes * [Conf] Move to options * [Conf] Rbl: DWL is actually special whitelist * [Conf] Relax some uribl rules * [Conf] Remove abuse.ch * [CritFix] Html: Entities are not valid within tag params values * [Feature] Add `rspamadm mime sign` tool * [Feature] Add configgraph utility * [Feature] Add dedicated ZW spaces detection for URLs * [Feature] Add flag to url object when visible part is url_like * [Feature] Add method task:lookup_words * [Feature] Add pyzor support (by crosenberg) * [Feature] Allow to add upstream watchers to Lua API * [Feature] Allow to set rewrite subject pattern from settings * [Feature] Better escaping of unicode * [Feature] Clickhouse: Allow to store subject in Clickhouse * [Feature] Core: Add QP encoding utility * [Feature] Core: Add libmagic detection for all parts * [Feature] Core: Add support for gzip archives * [Feature] Core: Allow to construct scan tasks from raw data * [Feature] Core: Detect charset in archived files * [Feature] Core: Ignore and mark invisible spaces * [Feature] Core: Normalise zero-width spaces in urls * [Feature] Core: Process data urls for images * [Feature] Core: Relax quoted-printable encoding * [Feature] Core: Support RFC2231 encoding in headers * [Feature] Core: Support telephone URLs * [Feature] Core: allow to emit soft reject on task timeout * [Feature] DCC: Add bulkness and reputation checks to dcc * [Feature] Elastic: Modernize plugin * [Feature] Export visible part of url to lua * [Feature] Fuzzy_storage: add preliminary support of rate limits * [Feature] HTML: Specially treat data urls in HTML * [Feature] Implement event watchers for upstreams * [Feature] Implement includes tracing in Lua * [Feature] Improve dkim part in configwizard * [Feature] Lua_scanners: Add VadeSecure engine support * [Feature] Lua_task: Add flexible method to get specific urls * [Feature] Mime_types: Add MIME_BAD_UNICODE rule * [Feature] Mime_types: Use detected content type as well * [Feature] Plugins: Add preliminary version of the external services plugin * [Feature] Query sentinel on master errors * [Feature] Regexp: Allow local lua functions in Rspamd regexp module * [Feature] Rspamadm: Allow to append footers to plain messages * [Feature] Rspamadm: Allow to rewrite headers in messages * [Feature] Selectors: Add `ipmask` processor * [Feature] Settings: Allow hostname match * [Feature] Settings: Allow local when selecting settings * [Feature] Settings: Allow multiple selectors * [Feature] Settings: Allow to inverse conditions * [Feature] Support User-Agent in HTTP requests * [Feature] Support ed25519 dkim keys generation * [Feature] Try to filter bad unicode types during normalisation * [Feature] external_services - oletools (olefy) support * [Feature] lua_scanners - icap protocol support * [Feature] lua_scanners - spamassassin spam scanner * [Fix] Add filter for absurdic URLs * [Fix] Add some more cases for Received header * [Fix] Allow to disable/enable composite symbols * [Fix] Arc: Use a separated list of headers for arc signing * [Fix] Archive: Final fixes for 7z archives * [Fix] Clickhouse: Fix database usage * [Fix] Controller: Make save stats timer persistent * [Fix] Core: Detect encrypted rarv5 archives * [Fix] Core: Don't detect language twice * [Fix] Core: Fix address rotation bug * [Fix] Core: Fix content calculations for message parts * [Fix] Core: Fix emails comments parsing and other issues * [Fix] Core: Fix etags support * [Fix] Core: Fix headers folding on the last token * [Fix] Core: Fix iso-8859-16 encoding * [Fix] Core: Fix log_urls flag (and encrypted logging) * [Fix] Core: Fix part length when dealing with boundaries * [Fix] Core: Fix parts distance calculations * [Fix] Core: Fix processing of NDNs of certain type * [Fix] Core: Implement logic to find some bad characters in URLs * [Fix] Core: treat nodes with ttl properly in lru cache * [Fix] Fix Content-Type parsing * [Fix] Fix HTTP headers signing case * [Fix] Fix control interface * [Fix] Fix deletion of the duplicate headers * [Fix] Fix emails filtering in emails module * [Fix] Fix greylisting log message and logic * [Fix] Fix issues with storing of the accepted addr in rspamd control * [Fix] Fix maps object update race condition * [Fix] Fix memor leaks and whitespace processing * [Fix] Fix processing of null bytes in headers * [Fix] Fix rcpt_mime and from_mime in user settings * [Fix] Fix rfc2047 decoding for CD headers * [Fix] Fix rfc2231 for Content-Disposition header * [Fix] Fix setting of the subject pattern in config * [Fix] Greylist: fix records checking * [Fix] HTML: Another HTML comments exception fix * [Fix] HTML: Another entities decoding logic fix * [Fix] HTML: Fix HTML comments with many dashes * [Fix] HTML: Fix entities in HTML attributes * [Fix] HTML: Fix some more SGML tags issues * [Fix] Ignore whitespaces at the end of value in DKIM records * [Fix] MID module: Fix DKIM domain matching * [Fix] Milter_headers: Fix remove_upstream_spam_flag and modernise config * [Fix] Mime_parser: Fix issue with parsing of the trailing garbadge * [Fix] Mime_parser: Fix parsing of mime parts without closing boundary * [Fix] Multimap: Fix operating with userdata * [Fix] Process orphaned `symbols` section * [Fix] Rdns: Fix multiple replies in fake replies * [Fix] Rework groups scores definitions * [Fix] Set proper element when reading data from Sentinel * [Fix] Set rspamd user to initialise supplementary groups on reload * [Fix] Settings: Fix selectors usage * [Fix] Sort data received from Sentinel to avoid constant replacing * [Fix] groups.conf - filename typo * [Fix] lua_scanner - oletools typos, logging * [Fix] lua_scanners - actions and symbol_fail * [Fix] lua_scanners - fix luacheck * [Fix] lua_scanners - kaspersky - response with fname * [Fix] lua_scanners - savapi redis prefix * [Fix] tests - antivirus - fprot symbols * [Project] Add concept of flexible actions * [Project] Add heuristical from parser to received parser * [Project] Add new flags to clickhouse, redis and elastic exporters * [Project] Attach new received parser * [Project] Fallback to callbacks from coroutines * [Project] Implement keep-alive support in lua_http * [Project] Lua_udp: Implement fully functional client * [Project] Plug keepalive knobs into http connection handling * [Project] Rspamadm: Add `modify` tool * [Rework] Convert rspamd-server to a shared library * [Rework] Dcc: Rework DCC plugin * [Rework] Enable explicit coroutines symbols * [Rework] Rework telephone urls parsing logic * [Rework] Rewrite RBL module * [Rework] Settings: Rework settings check * [Rework] Slashing: Distinguish lualibdir, pluginsdir and sharedir * [Rework] Unify task_timeout * [Rework] Use VEX instructions in assembly, relocate * [WebUI] Notify user if uploaded data was not learned * [WebUI] Remove redundant condition Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	8 years ago
Vsevolod Stakhov	1b9f2f3026	Release 1.8.3 * [CritFix] Make flags mutually exclusive for mime parts * [CritFix] Strictly deny unencoded bad utf8 sequences in headers * [Feature] Add Kaspersky antivirus support * [Feature] Add method to get dkim results * [Feature] Add more words regexp classes * [Feature] Allow to choose words format in `rspamadm mime` * [Feature] Allow to get all types of words from Lua * [Feature] Allow to get task flags in C expressions * [Feature] Allow to require encryption when accepting connections * [Feature] Ignore bogus whitespaces in the words * [Feature] Implement more strict configuration tests * [Feature] Improve SPF results in Authentication-Results * [Feature] Support ClickHouse database * [Fix] Add failsafety for utf8 regexps * [Fix] Do not trigger BROKEN_CONTENT_TYPE on innocent text parts * [Fix] Emit error if connection has been terminated with no stop pattern * [Fix] Fix boundaries checks in embedded messages * [Fix] Fix double free * [Fix] Perform policy downgrade on sample out, add tests * [Fix] Properly escape utf8 regexps in hyperscan mode * [Fix] Selectors - attachments args condition * [Fix] Some fixes for raw parts * [Fix] Treat learning errors as non-fatal * [Fix] Use tld when looking for DKIM domains * [Project] Words unicode structure rework * [Project] Add preliminary Redis Sentinel support * [Project] Improve Authentication-Results header * [Project] Rework DKIM checks results * [Project] Use more generalised API to produce meta words Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	8 years ago
Vsevolod Stakhov	dcab7362e0	Release 1.8.2 * [Conf] Add DWL support in the default configuration * [Conf] Disable rspamd_update by default (again) * [Conf] Fix configuration sample for ratelimit * [CritFix] Disable broken url tags by default * [CritFix] Fix \0 processing when doing RSA sign * [CritFix] Fix adding symbols to their primary groups * [Feature] Add `rspamadm cookie` utility * [Feature] Add specialised functions for generating encrypted cookies * [Feature] Add support of cookies in replies module * [Feature] Add support of words regexps * [Feature] Allow to add 3rd party clang plugins * [Feature] Allow to create lua regexps from glob or plain patterns * [Feature] Allow to set custom limits for upstream lists * [Feature] Detect orphaned parts and attach them to message * [Feature] Filter tokens in bayes * [Feature] Fold b= value when doing arc sealing * [Feature] Ignore cookies in the future and too old in the past * [Feature] Skip stop words in statistics * [Feature] Store stop words and allow to query them * [Feature] Support query arguments in controller's custom commands * [Feature] Tune upstream limits in Rspamd proxy * [Feature] Use different callback symbols for different uribls * [Feature] Write DKIM selector in dkim allow/reject symbols * [Fix] Add obs_fws state support to eoh state machine * [Fix] Add sanity check when applying mime boundaries heuristic * [Fix] Antivirus - virus names with 0 were recognized as tables * [Fix] Disable headernames in bayes temporarily * [Fix] Do not allow syntax errors in include files... * [Fix] Do not allow to merge an object with an array (or vice versa) * [Fix] Don't perform forged recipients check for missing recipients * [Fix] Fix DKIM based RBLs * [Fix] Fix actrie implementation (sync from upstream), fixed OOB read * [Fix] Fix explicit methods call in selectors * [Fix] Fix extraction of additional parts * [Fix] Fix finalization for internal plugins * [Fix] Fix override_defaults function * [Fix] Fix squeezed symbols when using settings * [Fix] Fix urls insertion in Clickhouse module * [Fix] Furhter fixes to ratelimits logic * [Fix] Ignore signatures when looking for boundaries * [Fix] Properly set learned count * [Fix] Really fix ratelimits configuration and work * [Fix] Remove ambigious format flag from printf * [Fix] Restore URLs exporting in ClickHouse plugin * [Fix] Rework bayes calculations... * [Fix] Switch from chi-square to naive for large Fisher value * [Fix] Treat normal password as enable password if there is no enable password * [Fix] Use proper syntax for making DNS requests * [Fix] Various fixes in embedded plugins * [Project] Change fuzzy check selection logic to lua_fuzzy library * [Project] Rework async events and symbols * [Project] Move all metatokens in Bayes to lua_stat from C * [WebUI] Add history rows per page control Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	8 years ago
Vsevolod Stakhov	6fe627f31f	Release 1.8.1 * [CritFix] Fix options insertion * [CritFix] Fix words decay one more time (affects long messages) * [CritFix] Increase default words_decay * [CritFix] Plug memory leak in redis pool * [Feature] Add `check_violation` feature to DKIM/ARC signing * [Feature] Add only unique elements to Clickhouse url arrays * [Feature] Allow `g+:` and `g-:` composite atoms * [Feature] Allow dkim domains check in surbl * [Feature] Allow maps with HTTP auth * [Feature] Allow to disable actions by users settings * [Feature] Extend whitelisting options * [Feature] Store url object in images * [Feature] Use verdict instead of the plain action in plugins * [Fix] Allow to call fstring append with NULL string * [Fix] DCC - luacheck * [Fix] Do not load torch on each rspamadm invocation * [Fix] Fix boundaries detection and rework stop words algorithm * [Fix] Fix dependencies for DNS_SIGNED symbol * [Fix] Fix errors when dealing with dynamic rates/bursts in Ratelimit * [Fix] Fix groups mess * [Fix] Fix groups mess * [Fix] Fix parsing address with comments * [Fix] Fix resolving in DMARC reports * [Fix] Fix various issues with parsing of the received headers * [Fix] Fix watchers issue in lua_tcp when doing no resolving * [Fix] Plug memory leak in language detector (affects reloads) * [Fix] Remove one letter stop words * [Fix] Slashing: backport chunk logic from libucl * [Fix] Stop libevent from using cached time in rspamadm * [Fix] Try to fix watchers chaining * [Fix] Various fixes in redis sync interface * [Fix] ip_score - respect check_authed and check_local settings from config * [Project] Rework passthrough actions * [Project] Clustering module * [Rework] Always create result for a task * [Rework] Completely rewrite DMARC checks logic * [Rework] Rework and fix whitelist plugin * [WebUI] Add symbols sorting buttons * [WebUI] Change symbols order without updating history * [WebUI] Colorize symbols * [WebUI] Do not display password form when secure_ip is set * [WebUI] Fix symbol description tooltips display * [WebUI] History: add sorting by symbol score value Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	8 years ago
Vsevolod Stakhov	f7f7588e22	Release 1.8.0 * [Feature] Add arguments schemas to processors and extractors * [Feature] Add functional selectors library * [Feature] Add generic selector to reputation module * [Feature] Add more ratelimits: by digest, by attachments data, by filenames * [Feature] Add preliminary stop words detection support * [Feature] Add pure Lua debugm function * [Feature] Add schema validation for Redis settings * [Feature] Add selectors combine function * [Feature] Add some recursion protection to lua logger * [Feature] Add support for Lua API tracing * [Feature] Allow to apply schema to arguments * [Feature] Allow to get dkim signing data directly from HTTP headers * [Feature] Allow to reuse existing authentication results * [Feature] Cache selectors results in re runtime * [Feature] Implement new text tokenizer based on libicu * [Feature] Integrate selectors framework to multimap * [Feature] Relax FORGED_RECIPIENTS * [Feature] Support (almost) all html entities * [Feature] Support adding and deletion of recipients in the milter block * [Feature] Support gathering HTTP body from fragments in lua_http * [Feature] Support multi flag in regexp and glob maps * [Feature] Support selectors in ratelimit module * [Feature] Support selectors in settings * [Feature] Use khash in HTML parser * [Feature] Use pure Lua debugm function * [Fix] Add fail-safety for destroying sessions * [Fix] Allow to add result-less fake DNS records * [Fix] Another try to fix race conditions on config unload * [Fix] Call Lua callback on DNS timeouts * [Fix] Deprecate task:inc_dns_req as it is redundant * [Fix] Do not allow events deletions on cleanup * [Fix] Do not try to process skipped messages * [Fix] Fix HTTP requests with no body * [Fix] Fix another cleanup race condition * [Fix] Fix bug in processing of pcre regexps * [Fix] Fix byte array allocation in the pool * [Fix] Fix crashes on task cleanup * [Fix] Fix dynamic buckets in ratelimits * [Fix] Fix endless loop when waiting for Rspamd to stop * [Fix] Fix lua_util.str_split in case of delimiters set * [Fix] Fix more issues with watching of async events * [Fix] Fix stop words detection and loading logic * [Fix] Fix various corner cases for language detection * [Fix] Fix watchers in lua_tcp * [Fix] Fix words decay algorithm * [Fix] Implement watchers replacement to handle nested calls * [Fix] Save faked code into fake dns record * [Fix] Show the proper frame when using lua_util.debugm * [Fix] Use fake dns records in tests * [Fix] Use unicode replacements for HTML entities * [Fix] fixed "cannot find dependency on symbol 1" issue when using replaced symbols in spamassassin rules * [Fix] partition_id is not available in old versions of CH * [Project] Add implicit conversion logic to selectors * [Project] Add initial support for selectors in regexps * [Project] Add method concept * [Project] Further changes in unicode operations * [Project] Implement Clickhouse migrations * [Project] Implement implicit conversions to userdata * [Project] Implement insert method * [Project] Implement selectors registration for regular expressions * [Project] Implement selectors support in re_cache * [Project] Improve language detector: cleanup unused files, categorize * [Project] Migrate CH data to a fat table * [Project] Rework selectors logic * [Project] Start Clickhouse utilities library * [Project] Start unicode rework * [Project] coroutine threaded model for API calls: thread pool * [Rework] Move phishtank to a DNS based service * [Rework] Rework Clickhouse plugin to use the new API * [Rework] Rework language detector * [Rework] Rework utf content processing in text parts * [WebUI] Add progress bar for AJAX requests * [WebUI] Avoid errors table reinitialization * [WebUI] Avoid history table reinitialization * [WebUI] Avoid throughput summary table reinitialization * [WebUI] Destroy summary table on disconnect * [WebUI] Fix "auth" request URL * [WebUI] Fix disabling and hiding controls on page reload * [WebUI] Fix maps loading from neighbours * [WebUI] Fix symbols sorting by score * [WebUI] Fix tables destroying * [WebUI] Fix throughput data consolidation * [WebUI] Fix upload buttons disabling * [WebUI] Notify user on module loading failure * [WebUI] Update FooTable 3.1.4 -> 3.1.6 Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	8 years ago
Vsevolod Stakhov	f63ec0e39b	Release 1.7.9 * [CritFix] Fix caseless comparison of equal length strings * [Feature] Add HTTP basic auth support to elastic and clickhouse plugins * [Feature] Add SPF selector to reputation * [Feature] Add support of the fallback backends for HTTP maps * [Feature] Allow to print full mime structure when extracting mime data * [Feature] Allow to split symbols in reputation plugin * [Feature] Check attachments only on AV scanners in attachments_only mode * [Feature] Disable all SSL checks if ssl_no_verify flag is set * [Feature] Implement parsing of scoped IPv6 addresses * [Feature] Improve `rspamc counters` output * [Fix] Add sanity checks when expanding SPF macros * [Fix] Allow to parse SA rules with no spaces around =~ (dirty hack) * [Fix] Avoid one extra byte writing * [Fix] Deal with direct hash table * [Fix] Detect empty text part as text, not HTML * [Fix] Do not reduce map watch timeout for mixed http/file maps * [Fix] Fix HTML part detection heuristic * [Fix] Fix double free in redirectors cleanup * [Fix] Fix legacy history handling in the controller * [Fix] Fix messages insertion * [Fix] Fix sending string method * [Fix] Fix statconver command line arguments * [Fix] Fixed argument checking for being null * [Fix] Fixed issues reported by luacheck * [Fix] Freeze updates queue when do actual storage update * [Fix] HTTP map hash is per-backend and not per-map * [Fix] Plug memory leak in fuzzy updates * [Fix] Prefer 'MTA-Name' when producing authentication results * [Fix] Replace bad unicode sequences instead of stopping on them * [Fix] Set classifier version on learning * [Project] Reworked ratelimits * [Project] Apply topological sorting for symbols in Rspamd * [Project] Remove global contexts from C modules * [Project] Move performance critical hash tables to khash * [WebUI] Avoid unused indexes * [WebUI] Do not execute `on_success` callback * [WebUI] Fix history reset for "All SERVERS" (#2346) * [WebUI] Fix query URL for selected server * [WebUI] Fix symbols display in legacy history, * [WebUI] Hide symbols order selector for legacy history * [WebUI] Refactor query functions into one * [WebUI] Remove previously-attached event handlers * [WebUI] Save symbols to the selected server * [WebUI] Unify arguments of query functions * [WebUI] Use common query functions to get graph data * [WebUI] Use common query functions to save symbols Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	8 years ago
Vsevolod Stakhov	4c80975059	Release 1.7.8 * [Feature] Add more extended statistics about fuzzy updates * [Feature] Add more non-conformant Received headers support * [Feature] Add preliminary function to get fuzzy hashes from text in Lua * [Feature] Allow to configure AV module rejection message * [Feature] Implement fuzzy hashes extraction in mime tool * [Feature] Improve WHITE_ON_WHITE rule * [Feature] Improve integer -> string conversion * [Feature] Reuse maps in multimap module more aggressively * [Fix] Avoid race condition in skip map as pool lifetime is not enough * [Fix] Eliminate all specific C plugins pools * [Fix] Fix DKIM check rule if DNS is unavailable * [Fix] Fix build where ucontext is defined in ucontext.h * [Fix] Fix crash in base url handling * [Fix] Fix descriptors leak in sqlite3 locking code * [Fix] Fix messages quarantine * [Fix] Fix padded numbers printing * [Fix] Fix race condition on maps reinit * [Fix] Fix regexp functions when no data is passed * [Fix] Fix specific urls extraction * [Fix] Fix styles propagation * [Fix] Improve resetting of the limit buckets * [Fix] Initialize sqlite3 properly * [Fix] Work with broken resolvers in resolv.conf * [Project] Implement HTTP maps caching * [Project] Refresh fuzzy hashes when matched * [Project] Add logic to deduplicate fuzzy updates queue * [WebUI] Add missed declarations * [WebUI] Avoid using "undefined" property * [WebUI] Do not accept passwords containing control characters * [WebUI] Do not redeclare variables * [WebUI] Enable strict mode, * [WebUI] Fix variable assignment * [WebUI] Initialize variables at declaration * [WebUI] Remove duplicated path from RequireJS config * [WebUI] Remove unused block * [WebUI] Remove unused variable * [WebUI] Remove unused variables * [WebUI] Use self-explanatory notation * [WebUI] Use type-safe equality operators Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	8 years ago
Vsevolod Stakhov	a20e33c22d	Release 1.7.7 * [CritFix] Check NM part of pubkey to match it with rotating keypairs * [CritFix] Do not overwrite PID of the main process * [CritFix] Fix maps after reload * [CritFix] Fix maps race conditions on reload * [CritFix] Fix shmem leak in encrypting proxy mode * [Feature] Add a concept of ignored symbols to avoid race conditions * [Feature] Add ability to print bayes tokens in rspamadm mime * [Feature] Add method to get statistical tokens in Lua API * [Feature] Add preliminary mime stat command * [Feature] Add rspamadm mime tool * [Feature] Add urls extraction tool * [Feature] Address ZeroFont exploit * [Feature] Allow rspamadm mime to process multiple files * [Feature] Allow to extract words in `rspamadm mime` * [Feature] Allow to print mime part data * [Feature] Allow to show HTML structure on extraction * [Feature] Distinguish IP failures from connection failures * [Feature] Improve output for mime command * [Feature] Improve styles propagation * [Feature] Main process crash will now cleanup all children * [Feature] Preload file and static maps in main process * [Feature] Print stack trace on crash * [Feature] Process font size in HTML parser * [Feature] Propagate content length of invisible tags * [Feature] Read ordinary file maps in chunks to be more safe on rewrites * [Feature] Support base tag in HTML * [Feature] Support more size suffixes when parsing HTML styles * [Feature] Support opacity style * [Fix] Another fix for nested composites * [Fix] Fill nm id in keypairs cache code * [Fix] Fix colors alpha channel handling * [Fix] Fix destruction logic * [Fix] Fix double free * [Fix] Fix maps preload logic * [Fix] Fix nested composites process * [Fix] Fix proxying of Exim connections * [Fix] Fix reload crash * [Fix] Fix rspamadm -l command * [Fix] Update ed25519 signing schema * [WebUI] Stop using "const" declaration * [WebUI] Update RequireJS to 2.3.5 Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	8 years ago
Vsevolod Stakhov	d9c84b4bd0	Release 1.7.6 * [CritFix] Fix multiple neural networks support * [Feature] Add decryption function to keypair command * [Feature] Add gzip compression for HTTP requests in elastic module * [Feature] Add gzip methods to lua util * [Feature] Add maps based on Top Level Domains * [Feature] Add pubkey checks for dkim_signing * [Feature] Add support of fake DNS records * [Feature] Add tool to encrypt files * [Feature] Allow to add symbols using settings directly * [Feature] Allow to match private and public keys for DKIM signatures * [Feature] Allow to set task flags via settings * [Feature] Allow to specify fake DNS address from the config * [Feature] Implement signatures verification using rspamadm keypair * [Feature] Implement signing using `rspamadm keypair` * [Feature] Improve error reporting for DKIM key access issues * [Feature] Provide $HOSTNAME variable in UCL * [Feature] Rework levenshtein distance computation * [Feature] Split message parsing and processing * [Feature] Support ED25519 DKIM signatures * [Feature] Support encrypted configs in UCL * [Feature] Suppress duplicate warning on very large radix tries * [Feature] Use OSB to combine header names * [Fix] Cleanup maps data on shutdown * [Fix] Fix '~' behaviour in composites * [Fix] Fix HTTP maps updates * [Fix] Fix NIST signatures * [Fix] Fix RFC822 comments when processing a mime address * [Fix] Fix double free * [Fix] Fix dynamic settings application * [Fix] Fix for CommuniGate Pro maillist * [Fix] Fix keypair creation method to actually create keypair... * [Fix] Fix matching patterns with no paths * [Fix] Fix memory leak in parsing comments * [Fix] Fix parsing of urls with numeric password * [Fix] Fix plugins intialisation in configwizard * [Fix] Fix potential crash on reload * [Fix] Fix potential race condition for a finished HTTP connections * [Fix] Fix race-condition leak on processes reload * [Fix] Fix signing in openssl mode * [Fix] Free language detector structures * [Fix] Relax alignment requirements * [Fix] Send DMARC reports compressed * [Fix] Try to fix leak in dmarc module * [Fix] Try to plug memory leak in metric exporter * [Project] Convert rspamadm subcommands to Lua * [WebUI] Display smtp sender/recipient in history * [WebUI] Fix elements disabling in "Symbols" tab * [WebUI] Limit recipients list in history column to 3 * [WebUI] Match envelope and mime addresses following in arbitrary order * [WebUI] Update column header * [WebUI] Wrap addresses in history Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	8 years ago
Vsevolod Stakhov	341222861d	Release 1.7.5 * [Conf] Add MSBL proposed return codes * [Conf] Add additional groups for policies * [CritFix] Do not use volatile Lua strings as UCL keys * [Feature] Add ability to add fuzzy hashes to headers * [Feature] Add function to extract most meaningful urls * [Feature] Add rule to block mixed text and encrypted parts * [Feature] Allow multiple groups for symbols * [Feature] Allow to disable lua squeezing logic * [Feature] Allow to get multipart children in Lua * [Feature] Allow to insert multiple headers from milter headers * [Feature] Allow to print scores in subject and further extensions * [Feature] Be more error-prone in squeezed rules * [Feature] Support multiple return codes in emails module * [Feature] Use EMA for calculating averages * [Feature] Use common jit cache for all regexps * [Feature] support for CommuniGate Pro self-generated messages * [Fix] Allow to have multiple values for headers as arrays * [Fix] Do not open sockets for disabled workers * [Fix] Fix AuthservId * [Fix] Fix base64 folding in Lua API * [Fix] Fix build on non-x86 platforms * [Fix] Fix cached maps logic * [Fix] Fix compatibility with old maps query logic * [Fix] Fix crash if skip_map is used * [Fix] Fix importing static maps from UCL * [Fix] Fix parsing of unix sockets * [Fix] Fix raw_mime regexp on HTML part with no text content * [Fix] Fix tables logging * [Fix] Fix vertical tab handling in libucl * [Fix] Try to fix frequency counters * [Fix] Use better sharding for ip_score * [Fix] Use multiple results from SURBL DNS reply * [Fix] When doing AV scan select a different server for retransmit Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	8 years ago
Vsevolod Stakhov	72b3e29432	Release 1.7.4 * [Conf] Major stock config updates: - Workers are now specified in a new format worker "type" { ... } - Enable fuzzy worker to simplify local fuzzy storages configuration - Bind all workers to localhost by default to avoid security flaws * [Conf] Make more sane fuzzy_check default settings * [CritFix] Fix ucl escape for bad symbols * [Feature] Add failure symbol for AV module * [Feature] Add lazy expiration mode for new classifier schema * [Feature] Add preliminary version of maps stats plugin * [Feature] Allow to block fuzzy requests from specific networks * [Feature] Allow to change `expire` of live statistics * [Feature] Distinguish AV failure from clean result * [Feature] Further improvements of language detector by using khash * [Feature] Further optimization of the lang_detection * [Feature] Implement cluster-aware bayes expiry * [Feature] Implement exclude patterns in rspamc * [Feature] Implement glob maps in addition to regexp maps * [Feature] Implement map statistics function for lua API * [Feature] Implement stop symbols for Clickhouse collection * [Feature] Support recipients separated by commas * [Feature] Try harder to upload scripts to the Redis server * [Feature] Upgrade t1ha distribution * [Feature] use_domain_sign_inbound * [Feature] Use scores from maps if `symbols_set` is not defined * [Fix] Add resolving version of radix map helper * [Fix] Check URL before adding implicit prefix * [Fix] Do not check pid/state when using PRNG * [Fix] Fix CentOS logrotate script for systemd * [Fix] Fix slash + dot in urls * [Fix] Fix systemd version of the logrotate script * [Fix] Propagate key when import implicit array from Lua * [Fix] Strip spaces from map keys and values * [Fix] Try to fix a specific case when processing milter protocol * [Fix] Try to fix crash when a tcp connection cannot be set * [Fix] Typo use_domain_local --> use_domain_sign_local * [Fix] Various fixes to once_received module * [Project] Store hits counters for map elements Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	8 years ago
Vsevolod Stakhov	4f481f93bc	Release 1.7.3 * [CritFix] Plug bad memory leak in protocol reply * [Feature] Add avx2 codec for base64 * [Feature] Add method to receive all URL flags from Lua API * [Feature] Allow to fold headers on stop characters * [Feature] Allow to set lua_cpath from options * [Feature] Allow to specify custom rejection message in milter * [Feature] Deal with unnormalised Unicode obfuscation * [Feature] Do not detect language twice for relative parts * [Feature] Implement oversigning feature * [Feature] Implement silent logging level to minimize noise in logs * [Feature] Improve URL_IN_SUBJECT rule * [Feature] Use hashing to reduce redis attack surface * [Fix] Add oversigning for the most important headers * [Fix] add 'rewrite subject' to History dropdown * [Fix] Another fix in folding algorithm * [Fix] Do not call multimap addr for parts of addr if filter is presented * [Fix] Do not clean hostname on generic reset * [Fix] Do not create pid file in no-fork mode * [Fix] Fix fold_after case to preserve multiple spaces * [Fix] Fix folding and folding tests * [Fix] Fix hostname usage in milter mode * [Fix] Fix lua RSA verify and its tests * [Fix] Fix metadata exporter send_mail backend (#2124) * [Fix] Fix processing of '\v' in libucl * [Fix] Fix shemaless URLs detection * [Fix] Fix support of multiple headers in sign_header * [Fix] Fix usage of util.parse_mail_address * [Fix] Fix weights of dynamic squeezed rules * [Fix] Leak from bucket before checking the burst * [Fix] Stop using own localtime as DST could be messy in many cases * [Fix] Treat unnormalised URLs as obscured * [Rework] Restore leaky bucket model in ratelimit plugin * [WebUI] Add messages total to throughput summary * [WebUI] Add symbols order selector to history * [WebUI] Config: Load list on demand * [WebUI] Fix modalBody for maps that appear more than once * [WebUI] History: Fix Tooltips on paging, filtering and sorting * [WebUI] Remove a previously-attached event handler * [WebUI] Update D3 to v5.0.0 and jQuery to v3.3.1 Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	8 years ago
Vsevolod Stakhov	8073aece67	Release 1.7.2 * [Feature] Store emails in Clickhouse * [Feature] Support single quotes in config * [Feature] Use templates when publishing CH schema * [Feature] Improve Docker image * [Fix] Add rounding when printing a lot of FP variables * [Fix] Allow to disable certain actions by assigning null to them * [Fix] Disable results caching * [Fix] Fix disabling of squeezed symbols * [Fix] Fix scan time set * [Fix] Rework logic of actions setting * [Fix] Try to fix various Lua stack issues * [WebUI] Add link tag for favicon.ico * [WebUI] Display hostname:port/path in the page title Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	8 years ago
Vsevolod Stakhov	3ba34325ab	Release 1.7.1 * [CritFix] Fix lowercase comparison * [CritFix] Timezone defines seconds WEST UTC not East * [Feature] Add filename to log format * [Feature] Add lua rules squeezing * [Feature] Add related symbols analysis to rspamd_stats * [Feature] Remove upstream `X-Spam: Yes` header by default * [Feature] rspamd_stats: Output progress info on STDERR * [Feature] Whitelist for emails module * [Fix] Do not allow dependencies on self * [Fix] Do not cache metric result * [Fix] Do not trust all issuers as a client certificate * [Fix] Fix dependencies in lua squeeze * [Fix] Fix enabling/disabling squeezed rules * [Fix] Fix enabling/disabling symbols * [Fix] Fix external dependencies * [Fix] Fix processing of a single compressed file * [Fix] Fix some typos * [Fix] Fix various modules in case of empty message * [Fix] Handle callbacks that returns table of options * [Fix] Improve cached action interaction * [Fix] Make dynamic conf more NaN aware * [Fix] Never hide actions from WebUI `configuration` tab * [Project] Implementation of Lua rules squeezing Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	8 years ago
Vsevolod Stakhov	720ae30a31	Release 1.7.0 * [Conf] Add bayes_expiry as explicit module * [Conf] Adjust names and weights for neural network plugin * [Conf] Change updates url * [Conf] Default statistics is stored in Redis now * [Conf] Disable fann_redis module by default * [Conf] Fix default elastic configuration * [Conf] Fix double quote position * [Conf] Massive config rework for new structure of symbols and scores * [Conf] Rename Rambler BLs as they are now Rspamd's ones * [Conf] Use dedicated rspamd.com subdomains * [Conf] Use more data from rspamd.com fuzzy storage * [CritFix] Add sanity guards for badly broken HTML * [CritFix] Another errors path handling fix * [CritFix] Another portion of tokenization fixes * [CritFix] Do not send reject messages after set reply * [CritFix] Fix ARC chain verification * [CritFix] Fix crash in milter errors handler * [CritFix] Fix memory leak in spf caching logic * [CritFix] Fix milter commands pipelining * [CritFix] Fix newlines detection * [CritFix] Fix semicolons parsing in the content type * [CritFix] Plug memory leak in zstd protocol compression * [Feature] Add ability to match score in force_actions module * [Feature] Add aes-rng PRF to libottery * [Feature] Add 'composites' debug module * [Feature] Add concept of experimental modules * [Feature] Add DKIM trace symbol * [Feature] Add EBL to the default config * [Feature] Add expected ip check for emails plugin * [Feature] Add framework to manage Redis scripts * [Feature] Add framing for the new reputation generic plugin * [Feature] Add function to show plugins stat * [Feature] Add gzip compression support for clickhouse module * [Feature] Add gzip compression support for rspamd controller * [Feature] Add gzip support when sending lua http requests * [Feature] Add json output for rspamd_stats * [Feature] Add method to do a synchronous Redis connection * [Feature] Add method to get all content-type attributes in Lua * [Feature] Add `-m` flag to configdump to show modules states * [Feature] Add mime types to extensions map * [Feature] Add more features to rescore utility * [Feature] Add more gtube like patterns to test other spam actions * [Feature] Add more metafunctions, improve logging * [Feature] Add more text attributes * [Feature] Add new configwizard command to rspamadm * [Feature] Add new tooling for stats conversation * [Feature] Add old groups migration tool * [Feature] Add plugins state variable * [Feature] Add preliminary ecdsa keys support in DKIM * [Feature] Add preliminary support of idempotent symbols * [Feature] Add Redis server wizard * [Feature] Add routine to convert old style stats to a new one * [Feature] Add some sanity checks for actions and controller * [Feature] Add statistic convertation module to configwizard * [Feature] Add sugestions logic to mempool allocator * [Feature] Add support of config transform in Lua * [Feature] Add timeout to rspamc when doing corpus test * [Feature] Add tooling to convert bayes schemas * [Feature] Add torch conditional to configuration * [Feature] Add torch-decisiontree package * [Feature] Add torch-optim contrib package * [Feature] Add TTL autodetection * [Feature] Add urls reputation to the reputation framework * [Feature] Allow floating and negative values in expressions limits * [Feature] Allow multiple CTs in full extensions map * [Feature] Allow multiple fann rules * [Feature] Allow randomly select User-Agent from a list * [Feature] Allow rspamadm commands to export methods in Lua * [Feature] Allow rule specific min_bytes in fuzzy check * [Feature] Allow to adjust symbols scores from Lua * [Feature] Allow to attach stat signature to messages * [Feature] Allow to change SMTP from via milter headers * [Feature] Allow to configure monitored * [Feature] Allow to create directories in Lua API * [Feature] Allow to disable torch and skip train samples for ANN * [Feature] Allow to discard messages dynamically * [Feature] Allow to enable/disable languages from the detector * [Feature] Allow to generate DKIM keys from rspamadm API * [Feature] Allow to get CPU flags from Lua * [Feature] Allow to have high precision timestamps in logs * [Feature] Allow to insert headers into specific position * [Feature] Allow to limit redirector requests per task * [Feature] Allow to load and use dynamic ANNs with torch * [Feature] Allow to quarantine rejected messages using milter interface * [Feature] Allow to receive signing keys from mempool vars * [Feature] Allow to reserve elements in libucl * [Feature] Allow to reuse signal handlers chains * [Feature] Allow to set custom mempool variables from settings * [Feature] Allow to set headers from settings * [Feature] Allow to set Settings-Id for all connections * [Feature] Allow to skip real action and add a header instead * [Feature] Allow to skip specific hashes in fuzzy storage * [Feature] Allow to spawn asynchronous processes from Lua * [Feature] Allow to specify number of threads for ANN learning * [Feature] Allow to use global lua maps in settings * [Feature] Allow to use postfilters in composites * [Feature] Allow to verify signatures from HTTP headers in maps * [Feature] Antivirus: ordered pattern matches * [Feature] Authentication-Results: support hiding usernames * [Feature] Automatically create tables in clickhouse * [Feature] Catch next-to-last bad extension * [Feature] Check cached maps more frequently * [Feature] Check groups sanity * [Feature] Deal with obscured URLs with @ symbols * [Feature] Enhance task:store_in_file method * [Feature] Export password encryption routines to Redis * [Feature] Filter nan and inf when adding scores * [Feature] Finalize 7zip files support * [Feature] Further improvements in language detection * [Feature] Further improvements in language detection algorithm * [Feature] Generic key name expansion for Redis keys * [Feature] Hash whitelist for fuzzy_check * [Feature] Implement bayes signatures storage * [Feature] Implement buckets for Redis backend * [Feature] Implement DKIM reputation adjustments * [Feature] Implement forked workers children monitoring * [Feature] Implement headers flags in mime parser * [Feature] Implement l1/l2 regularization against the current weights * [Feature] Implement manual ANN train mode * [Feature] Implement per-user ANN support * [Feature] Implement torch based ANN learning * [Feature] Implement upstreams logic for clickhouse exporter * [Feature] Import torch to Rspamd... * [Feature] Improve allocation policy when interacting with Lua * [Feature] Improve Lua/C interaction in history_redis * [Feature] Improve multiple fuzzy results combining * [Feature] Improve parsing of DKIM keys: parse algorithm * [Feature] Improve subprocesses termination handle * [Feature] Improve symbol type parsing in Lua API * [Feature] Metadata Exporter: e-Mail Alerts: support multiple recipients; alerting senders/recipients/users (#1600) * [Feature] Milter headers: support adding/removing arbitrary headers from config * [Feature] More metatokens * [Feature] Multimap: checking of symbol options * [Feature] Multimap: template URL filter * [Feature] New bayes expiry plugin * [Feature] Periodically save rspamd stats to disk * [Feature] Preliminary import of the elasticsearch module * [Feature] Ratelimit: allow full addresses in whitelisted_rcpts * [Feature] Ratelimit: support fetching limits from Redis * [Feature] RBL: received: filtering by position & flags * [Feature] Read global maps for lua * [Feature] Redis settings: support checking multiple keys * [Feature] Rework fann plugin to be a normal post-filter * [Feature] Rework logging configuration for rspamadm case * [Feature] Rework short hashes generation to avoid FP * [Feature] Save real ucl types when exporting to Lua * [Feature] Set TCP_NODELAY for milter sockets * [Feature] Setup DKIM signing from configwizard * [Feature] Skip certain symbols from ANN classify * [Feature] Store plugins state * [Feature] Support etag for HTTP maps * [Feature] Support Expires header when using HTTP maps * [Feature] Support sending given header multiple times in lua_http * [Feature] Support sha512 in DKIM signatures * [Feature] Try to detect HTML messages better * [Feature] Use array instead of queue to reduce memory fragmentation * [Feature] Use controller port by default when connecting to local IP * [Feature] Use rdtsc where possible * [Fix] Actively load skip hashes map in fuzzy storage * [Fix] Add another workaround to display history properly * [Fix] Add definition for old glib compatibility method * [Fix] Add missing rspamadm control options to help * [Fix] Add workaround for IPv6 in sendmail * [Fix] Add workaround for system with non-XSI compatible tzset * [Fix] Allow oversigning in DKIM signatures * [Fix] Allow to check negative scores in force_actions * [Fix] Allow to have negative actions limits * [Fix] Allow to set any layers number for fann rules * [Fix] Another fix for rdtcs * [Fix] Another fix to lua xmlrpc * [Fix] Another try to deal with #1998 * [Fix] Another try to fix #1998 * [Fix] Another try to fix threading in torch * [Fix] Apply language detection when adding fuzzy hashes * [Fix] ARC: Fix Lua 5.3 compatibility; timestamp should be integer * [Fix] Authentication Results: Fix SPF smtp.mail_from * [Fix] Auth-Results: Multiple DKIM signatures * [Fix] Avoid changing content-transfer-encoding header's value * [Fix] Better handling of the legacy protocol * [Fix] Check decoded headers sanity (e.g. by excluding \0) * [Fix] Check for magic when checking for an archive * [Fix] Cleanup mess with groups * [Fix] Clickhouse: Insertion in the symbols table * [Fix] Crash in URL processing * [Fix] Deal with another case when processing exceptions * [Fix] Deal with deeply nested messages more aggressively * [Fix] Deal with nan and inf encoding in json/ucl * [Fix] Deal with non-key arguments in lua_redis.exec_script * [Fix] Deal with unknown weight * [Fix] Deal with URLs with no slashes after protocol * [Fix] Deal with URLs wrapped in [] in text parts * [Fix] Deal with zero scores symbols * [Fix] Default monitoring domain for surbl plugin * [Fix] Delay upstream re-resolving when one upstream is defined * [Fix] Detection of maillist optimized and fixed * [Fix] DKIM signing: allow for auth_only to be false * [Fix] DMARC: require report_settings for sending reports only * [Fix] Do not allow garbadge when checking url domain * [Fix] Do not cache SPF records with PTR elements * [Fix] Do not constantly re-resolve failed upstreams with a single element * [Fix] Do not crash if no words defined * [Fix] Do not crash on empty subtype * [Fix] Do not expose spamtrap messages to SMTP reply * [Fix] Do not fail rbl plugin when there are no received or emails * [Fix] Do not ignore short words * [Fix] Do not include idempotent/nostat symbols to checksum * [Fix] Do not override groups when converting metrics * [Fix] Do not override unix socket group when group comes before owner * [Fix] Do not skip the last character * [Fix] Do not spawn too many workers by default * [Fix] Do not stop monitored on dns errors * [Fix] Do not stop parsing headers on bad IP header * [Fix] Do not strip last character in the last word * [Fix] Do not treat script content as text * [Fix] Do not try to connect to non-supported addresses * [Fix] Do not try to dereference last character * [Fix] Do not try to sign unknown domains * [Fix] Don't use whitelist/greylist maps as regexp, but as map * [Fix] Erase unknown HTML entities * [Fix] Exim Received header protocol parsing * [Fix] First load selector_map and path_map. And only return false when domain not found if try_fallback is false * [Fix] Fix a lot of FP in chartable in mixed languages * [Fix] Fix ANN checks * [Fix] Fix ANN loading logic * [Fix] Fix another tokenization issue * [Fix] Fix autolearn parameters reading * [Fix] Fix bad archive characters stripping * [Fix] Fix bad extension check * [Fix] Fix bayes schema conversion * [Fix] Fix blacklists and DMARC in whitelist * [Fix] Fix brain-damaged torch build system * [Fix] Fix build on FreeBSD * [Fix] Fix clickhouse exporter * [Fix] Fix clickhouse schema * [Fix] Fix comparision * [Fix] Fix composites processing * [Fix] Fix connecting to a unix socket in rspamadm statconvert * [Fix] Fix couple of warnings * [Fix] Fix crashes in the rspamd_control path * [Fix] Fix deletion from hash * [Fix] Fix DKIM forgeries via multiple headers * [Fix] FIx dynamic conf plugin * [Fix] Fix emails detection * [Fix] Fix empty headers simple canonicalization * [Fix] Fix empty threshold check in greylisting module * [Fix] Fix encrypted legacy reply in fuzzy storage * [Fix] Fix enormous scores for R_WHITE_ON_WHITE * [Fix] Fix exceptions list in surbl * [Fix] Fix *_EXCESS_BASE64 rules * [Fix] Fix expire rounding * [Fix] Fix extra hits in PCRE mode for regular expressions * [Fix] Fix format strings * [Fix] Fix get_content method * [Fix] Fix groups override when defining symbols * [Fix] Fix learned count in new schema * [Fix] Fix learn errors propagation * [Fix] Fix loading of per-user redis backend for statistics * [Fix] Fix logging buffer corruption in case of repeated messages * [Fix] Fix lua cached elements invalidation * [Fix] Fix merging of the implicit arrays * [Fix] Fix mime_types scoring * [Fix] Fix multiple headers in DKIM headers list * [Fix] Fix null callee case in clang plugin * [Fix] Fix obscured url in format user@@example.com * [Fix] Fix parsing of the per-user script * [Fix] Fix priorities in rspamd_update, disable rules execution * [Fix] Fix processing of closed tags * [Fix] Fix processing of idempotent rules when autolearn fails * [Fix] Fix processing of multipart parts with no headers * [Fix] Fix processing of skip-hashes in fuzzy storage * [Fix] Fix PTR processing in SPF * [Fix] Fix pushing country to clickhouse asn table * [Fix] Fix random forests module * [Fix] Fix real IP parsing for some strange Exim received * [Fix] Fix Redis timeout setup * [Fix] Fix reload crash when hyperscan is enabled * [Fix] Fix reusing of redis connection after exec * [Fix] Fix sanity checks on macro value * [Fix] Fix setting of path and cpath for Lua * [Fix] Fix setting of signals when spawning a thread * [Fix] Fix text splitting: stack overflow (too many captures) * [Fix] Fix ticks processing * [Fix] Fix upstream addrs updating * [Fix] Fix urls/emails distinguishing found in queries * [Fix] Fix user settings check * [Fix] Fix variable increment * [Fix] Fix various issues in stat_convert * [Fix] F-PROT Antivirus infection string for all known occurences * [Fix] F-PROT Antivirus: only check return code to determine infection * [Fix] Further fixes around floating point expressions * [Fix] Further fixes to ANN module * [Fix] Further fixes to rescore tool * [Fix] Further fixes to support ES 6 * [Fix] Further tokenization fixes * [Fix] Greylisting set phase is not idempotent * [Fix] Handle proxy copy errors * [Fix] Header checks: Fix get_raw_header method * [Fix] Header checks: REPLYTO_UNPARSEABLE rule * [Fix] Kill spawned processes on termination * [Fix] Load skip map from all processes as shared cache is unavailable * [Fix] Lowercase HTTP headers to make them searchable from Lua * [Fix] Lowercase words * [Fix] Lua_http: freeing * [Fix] Lua: lpeg to be loaded with rspamd_lua_add_preload, to avoid "rspamd_config_read: rcl parse error: cannot init lua file […] module 'lpeg' not found" * [Fix] Map absence is not an error * [Fix] Metadata exporter: check IP sanity * [Fix] Milter headers: custom headers: removing headers * [Fix] Milter headers: skip_local / skip_authenticated settings * [Fix] Milter headers: X-Spamd-Result header if X-Virus ran first * [Fix] mime_types: fix next-to-last extension length check * [Fix] More hacks to deal with old configs * [Fix] Move composites second pass to the dedicated stage * [Fix] Multimap: received: filtering of artificial header * [Fix] Multiple fixes in torch based ANN plugins * [Fix] Once more (#1879) fix bad extension check * [Fix] Optimize rspamd_fstring_t reallocations * [Fix] options.local_networks setting * [Fix] Parse HREF urls without explicit prefix * [Fix] Plan new event on HTTP errors * [Fix] Plug another possible memory leak * [Fix] Plug memory leak * [Fix] Plug memory leak in lua_tcp * [Fix] Plug memory leak when setting email addresses from Lua * [Fix] Propagate learn/stat errors more precisely * [Fix] Ratelimit: fix whitelisted_rcpts matching * [Fix] Ratelimit: lowercase email addresses * [Fix] RBL: received: deal with missing data (#1965) * [Fix] Rebalance and slightly rework MX check plugin * [Fix] Redis key expansion: EVAL: deal with strings * [Fix] Redis script loading in DMARC; URL tags; URL reputation * [Fix] Reject invalid bh for DKIM signatures earlier * [Fix] Relax pem signature detection * [Fix] Relax unicode properties requirements for chartable module * [Fix] Remove extra noise from dkim and arc signing * [Fix] Remove hop-by-hop headers in proxy * [Fix] Remove incorrect method `task:set_metric_subject` * [Fix] Replace space like characters in headers with plain space * [Fix] Restore old style ratelimits support * [Fix] Rework elasticsearch plugin * [Fix] Rewriting subjects via force actions module * [Fix] RPM postinstall * [Fix] Sanitize IP in history redis * [Fix] Select the correct signature when doing simple canon * [Fix] Set CLOEXEC flag on files opened * [Fix] Setting check_local / check_authed in plugins (#1954) * [Fix] Settings: avoid checking invalid IP (#1981) * [Fix] Settings: header: deal with multiple settings (#1988) * [Fix] Skip checks if both extensions are not bad * [Fix] Skip nostat tokens when get number of tokens * [Fix] Some more fixes towards emails detection * [Fix] SpamAssassin: Fail check_freemail_header if regexp didn't match * [Fix] Stop using of g_slice... * [Fix] Switch rspamadm logging to message level * [Fix] Symbol 'FANNR_SPAM' has its score defined.. * [Fix] Table parameter for rspamd_config:add_doc() * [Fix] Treat 'rewrite subject' as spam action * [Fix] Try harder in passing IPv6 addresses * [Fix] Try harder to find rfc822 notifications * [Fix] Try harder to find urls * [Fix] Use decoded values when parsing mime addresses * [Fix] Use full URL when making an HTTP request * [Fix] Use greylisting threshold in greylisting module * [Fix] Use n_words attribute from ngramms * [Fix] Use raw urls when sending requests to redirector * [Fix] Use the right boolean operator on error check * [Fix] Use weight from map for fuzzy scoring * [Fix] Various fixes to elastic plugin * [Fix] Various fixes to fann_redis instantiation * [Fix] Various improvements in language detection * [Fix] Virus infection string for F-PROT Antivirus * [Fix] Virus infetction string for F-PROT Antivirus * [Fix] WebUI: use relative path for savemap (#1943) * [Fix] WHITE_ON_WHITE: Ensure score is matched to part that fired the rule * [Fix] Write configuration changes as UCL config * [Project] Add detection logic for words * [Project] Add fast debug logging infrastructure * [Project] Add more flags to languages * [Project] Add n-gramms data files * [Project] Add ngramms frequencies detector * [Project] Add random words selection logic * [Project] Add unigramms to language detection as well * [Project] Convert all C modules to fast debug infrastructure * [Project] Detect some languages based on unicode script * [Project] Enable fast debug lookup for some modules * [Project] Enable language detector init in scanner workers * [Project] Further improvements to language detector * [Project] Implement logic of ngramms application * [Project] Improve weighting in lang_detection * [Project] Initialize language detector * [Project] Preliminary version of ngramms based language detector * [Project] Preliminary version of the new stat_convert * [Project] Remove old language detector * [Project] Rework language detection ngramms structure * [Project] Start language detection project * [Project] Start rework of language detection to improve quality * [Project] Use fast debug logging check * [Rework] Add frame for new reputation based IP score module * [Rework] Continue stat_convert rework task * [Rework] Implement new version of fuzzy replies * [Rework] Improve readability of xmlrpc API * [Rework] Kill metrics!11 * [Rework] Ratelimit module * [Rework] Rename fann_redis to neural plugin * [Rework] Reorganize mime_types module * [Rework] Rework rescore utility * [Rework] Rewrite model and learning logic for rescore * [Rework] Run post-loads when all initialization is completed * [Rework] Simplify lua path initialization * [Rework] Start major stat_convert rework * [Rework] Start mempool fragmentation reduce project * [Rework] Start moving of fann redis to torch * [Rework] Stop embedding rspamadm scripts into C * [Rework] Use floating point arithmetics in Rspamd expressions * [Rework] Use frequencies distribution in language detector * [Rules] Penalise R_BAD_CTE_7BIT for utf8 messages * [WebUI] Compact graph selectors * [WebUI] Escape strings inside HTML in history * [WebUI] Fix message count in throughput summary (#1724) * [WebUI] Fix NaNs display on Throughput graph * [WebUI] Migrate widgets to D3 v4 * [WebUI] Restore passwordless login support (#2003) * [WebUI] Show symbol descriptions as tooltips in history * [WebUI] Stop using commas in pie chart tooltips * [WebUI] Update D3 and jQuery * [WebUI] Update D3Evolution 1.0.0 -> 1.1.0 Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	9 years ago
Vsevolod Stakhov	8fb0ef19f4	Release 1.6.6 * [CritFix] Add sanity guards for badly broken HTML * [CritFix] Another errors path handling fix * [CritFix] Fix ARC chain verification * [CritFix] Fix crash in milter errors handler * [Feature] Allow to insert headers into specific position * [Feature] Allow to receive signing keys from mempool vars * [Feature] Authentication-Results: support hiding usernames * [Fix] Another try to deal with #1998 * [Fix] Another try to fix #1998 * [Fix] Better handling of the legacy protocol * [Fix] Check decoded headers sanity (e.g. by excluding \0) * [Fix] Deal with nan and inf encoding in json/ucl * [Fix] Deal with URLs wrapped in [] in text parts * [Fix] DKIM signing: allow for auth_only to be false * [Fix] Do not crash on empty subtype * [Fix] Do not fail rbl plugin when there are no received or emails * [Fix] Do not skip the last character * [Fix] Do not try to dereference last character * [Fix] Do not try to sign unknown domains * [Fix] Exim Received header protocol parsing * [Fix] First load selector_map and path_map. And only return false when domain not found if try_fallback is false * [Fix] Fix bad archive characters stripping * [Fix] Fix comparision * [Fix] Fix connecting to a unix socket in rspamadm statconvert * [Fix] Fix empty headers simple canonicalization * [Fix] Fix extra hits in PCRE mode for regular expressions * [Fix] Fix parsing of the per-user script * [Fix] Fix processing of skip-hashes in fuzzy storage * [Fix] Fix Redis timeout setup * [Fix] Fix sanity checks on macro value * [Fix] Fix text splitting: stack overflow (too many captures) * [Fix] Fix urls/emails distinguishing found in queries * [Fix] F-PROT Antivirus: only check return code to determine infection * [Fix] Metadata exporter: check IP sanity * [Fix] Multimap: received: filtering of artificial header * [Fix] Plan new event on HTTP errors * [Fix] Plug another possible memory leak * [Fix] Remove hop-by-hop headers in proxy * [Fix] Sanitize IP in history redis * [Fix] Setting check_local / check_authed in plugins (#1954) * [Fix] Settings: avoid checking invalid IP (#1981) * [Fix] Try harder in passing IPv6 addresses * [Fix] WebUI: use relative path for savemap (#1943) * [WebUI] Fix message count in throughput summary (#1724) * [WebUI] Fix NaNs display on Throughput graph * [WebUI] Restore passwordless login support (#2003) Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	9 years ago
Viktor Szépe	b3e626e260	Typo: comparision	9 years ago
Vsevolod Stakhov	111dfde1ad	[Minor] Sync ChangeLog	9 years ago
Vsevolod Stakhov	fa9b2fc2d8	Release 1.6.3 * [CritFix] Fix semicolons parsing in the content type * [Feature] Add EBL to the default config * [Feature] Allow to configure monitored * [Feature] Allow to skip specific hashes in fuzzy storage * [Feature] Multimap: checking of symbol options * [Feature] Redis settings: support checking multiple keys * [Fix] ARC: Fix Lua 5.3 compatibility; timestamp should be integer * [Fix] Avoid changing content-transfer-encoding header's value * [Fix] Don't use whitelist/greylist maps as regexp, but as map * [Fix] Fix get_content method * [Fix] Header checks: Fix get_raw_header method * [Fix] Header checks: REPLYTO_UNPARSEABLE rule * [Fix] Lua_http: freeing * [Fix] Milter headers: custom headers: removing headers * [Fix] Parse HREF urls without explicit prefix * [Fix] WHITE_ON_WHITE: Ensure score is matched to part that fired the rule * [WebUI] Escape strings inside HTML in history Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	9 years ago
Vsevolod Stakhov	e5cefe177e	Release 1.6.2 * [Conf] Remove Rambler email bl for now * [Conf] Switch RAMBLER_URIBL to a locally managed source * [CritFix] Switch from ragel to C for Content-Type parsing * [Feature] Add `-e` option for lua_repl * [Feature] Add per-domain emails normalisation rules * [Feature] Add sessions cache to debug dangling sessions * [Feature] Add short_text_direct_hash for fuzzy check module * [Feature] Add text_part:get_stats function * [Feature] Allow to add custom processing script for surbl * [Feature] Allow to check reply-to email * [Feature] Allow to customize spam header, remove existing spam headers * [Feature] Allow to disable specific workers in the config * [Feature] Allow to discard messages instead of rejection * [Feature] Allow to specify custom delimiter in emails plugin * [Feature] Allow to specify custom User-Agent for rspamc * [Feature] Allow to store symbols data in Clickhouse * [Feature] Allow to use HTTPS when connecting to Clickhouse * [Feature] Enable sessions cache tracking for milter connections * [Feature] Implement per-line mode in lua_repl (like `perl -p`) * [Feature] Implement rdns-curve plugin based on rspamd cryptobox * [Feature] Improve maps cached data lifetime * [Feature] Improve maps checking frequency * [Feature] Improve monitored timeouts logic * [Feature] milter_headers: add `extended_headers_rcpt` option * [Feature] Milter headers: Add X-Spam-Flag to rmilter-compatibility headers * [Feature] Milter headers: remove-header routine * [Feature] Multimap: received filters for extracting TLDs from hostnames * [Feature] Normalize email aliases in emails module * [Feature] Re-add rambler email bl (as hashed list) * [Feature] Reload file maps more frequently * [Feature] Rework newlines strip parser one more time * [Feature] Skip updates for messages scanned via controller * [Feature] Split long DKIM public keys * [Feature] Store more data when stripping newlines * [Feature] Support SPF macros transformations * [Feature] Support suppressing DMARC reports for some domains * [Fix] Add missing `break` statement * [Fix] Allow modifiers in SPF macros * [Fix] DKIM sign tools: edge-cases around use_esld * [Fix] Do not cache SPF records with macros * [Fix] Do not overwrite score when setting pre-action * [Fix] Fix comparision logic * [Fix] Fix DKIM base64 folding for milter flagged messages * [Fix] Fix emails module configuration * [Fix] Fix folding for arc headers when milter interface is used * [Fix] Fix gmail dots removal * [Fix] Fix rspamc detection in greylist module * [Fix] Fix some more issues with HTTP maps * [Fix] Milter sessions can live forever * [Fix] Normalize fuzzy probability better * [Fix] Plug memory leak * [Fix] RBL: Fixed hashed email address lookups * [Fix] Try to deal with brain-damaged milter behaviour * [Fix] Use `\n` to fold headers for milter * [Rework] Allow to use custom callback for monitored checks * [Rework] Further steps towards one process monitoring * [Rework] Send health checks from a single worker * [WebUI] Round-up throughput summary values Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	9 years ago
Alexander Moisseev	bb5a9f8e45	[Minor] Massive spelling correction by a bot https://github.com/ka7/misspell_fixer	9 years ago
Vsevolod Stakhov	8895fc6fab	Release 1.6.1 * [Fix] Allow to init resolver without rspamd_config * [Fix] Do not crash when resolver failed to initialize * [Fix] Fix abstract context layout * [Fix] Fix CGP helper reply parsing * [Fix] Fix crashes when socket write errors occur * [Fix] Fix parsing IPv6 nameservers in resolv.conf * [Fix] Milter: Don't defer on "greylist" action Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	9 years ago
Vsevolod Stakhov	9c0fd5e439	Release 1.6.0 * [Conf] Add rspamd_proxy to the default configuration set * [Conf] Add sample arc module config * [Conf] Do away with systemd specifics completely * [Conf] Increase min_bytes to avoid FP * [Conf] Remove ratelimits from default configuration * [CritFix] Fix accepting on IPv6 sockets * [CritFix] Fix corruption when multiple fuzzy are defined * [CritFix] Fix learn condition in fuzzy check * [CritFix] Fix memory leak in fuzzy check * [CritFix] Fix memory leak in maps scheduling * [CritFix] Paese the last character in DKIM signature correctly * [CritFix] Zero fill sockaddr_un * [Feature] Add ability to add doc strings by example * [Feature] Add API to verify DKIM (and ARC) signatures * [Feature] Add compression/decompression to proxy * [Feature] Add count to url structure * [Feature] Add initial support of the new protocol reply * [Feature] Add Lua plugin spamtrap * [Feature] Add `monitored_address` for rbls * [Feature] Add new schema for bayes tokens * [Feature] Add preliminary ARC support to dkim code * [Feature] Add preliminary support of ARC signing * [Feature] Add rules to detect bad 8bit characters in From and To * [Feature] Add scanning support for milter protocol * [Feature] Add support for bidirectional symbols in rspamd_stats * [Feature] Add support for static maps * [Feature] Add support of maps with multiple regexps matches * [Feature] Add `text_multiplier` param * [Feature] Add the preliminary ARC plugin * [Feature] Add top redirector targets rank * [Feature] Allow async events to be registered from LUA rules * [Feature] Allow storing bayes tokens in Redis * [Feature] Allow to exclude specific domains from mx check * [Feature] Allow to have a stack of watcher finalisers * [Feature] Allow to pass hostname to `-i` flag in Rspamc * [Feature] Allow to set custom user agent in url redirector * [Feature] Allow to use custom callback when parsing resolv.conf * [Feature] Allow to use domain from authenticated user * [Feature] Bayes expiry plugin * [Feature] Check dkim sign keys for modifications * [Feature] DKIM signing: sign_networks/local address specific use_domain settings * [Feature] DMARC: Support excluding domains from sampling * [Feature] Expire processing items for URL redirector aggressively * [Feature] Fix surbl monitored for IP lists, add `monitored_domain` option * [Feature] Implement caching for dkim body hashes * [Feature] Implement milter protocol scan reply * [Feature] Improve omograph phishing detection * [Feature] Initial support of self-scan in Rspamd proxy * [Feature] Keep track of headers in milter interface * [Feature] Milter headers: better controls for local/authenticated * [Feature] Multimap: email:domain:tld filter * [Feature] Preliminary DMARC reporting implementation * [Feature] Reuse stemmers in the cache * [Feature] Rework confighelp to load Lua plugins * [Feature] Rework hfilter to use hyperscan if possible * [Feature] Rework lua RSA API * [Feature] Rmilter_headers: approximate rmilter's extended_spam_headers * [Feature] Start integration of milter support in proxy * [Feature] Store average words length and short words count * [Feature] Store hash of headers order and names * [Feature] Support MTA name header * [Feature] Support multiple types of dkim signing in Lua * [Feature] Support numeric arguments for Redis requests * [Feature] Use headers hash in bayes metatokens * [Feature] Use normal resolv.conf rules of rotation in Rspamd * [Feature] Use version 2 proto for checking messages * [Fix] Allow to follow symlinks when safe * [Fix] Append MX name for authentication results as required * [Fix] Change default text multiplier from 0.5 to 2.0 * [Fix] Check min_bytes for images as well * [Fix] Deal with 7bit charsets properly * [Fix] Deal with 8bit characters in email addresses * [Fix] Deal with unpaired <a> tags * [Fix] Detect confighelp in plugins initialisation * [Fix] Disable certain checks for utf spoof detection * [Fix] DKIM Signing: avoid nil index when From header is missing * [Fix] Do not add exact hashes from different parts * [Fix] Do not check DMARC if SPF or DKIM were not checked * [Fix] Do not check URLs that are resolved to be redirected * [Fix] Do not set bayes probability if we don't use it * [Fix] Do not stop on illegal unicode points - replace them * [Fix] Fix another race condition in arc checks * [Fix] Fix arc count logic * [Fix] Fix ARC signing * [Fix] Fix brain-damaged spamc protocol for now * [Fix] Fix calling for peak functions * [Fix] Fix couple of issues in FORWARDED rule * [Fix] Fix CTE propagation from parent containers to children parts * [Fix] Fix errors processing in the controller * [Fix] Fix format string in milter * [Fix] Fix issues in SPF macros parsing * [Fix] Fix logging format string * [Fix] Fix logic of cached passwords check * [Fix] Fix lowercasing of stemmed words * [Fix] Fix LRU elements removal * [Fix] Fix memory leak when accepting from unix sockets * [Fix] Fix milter connections persistence * [Fix] Fix objects merging in UCL * [Fix] Fix order of operations to avoid race condition * [Fix] Fix parsing of long regexp types * [Fix] Fix passing data to log helper when many symbols defined * [Fix] Fix pools management for milter session * [Fix] Fix processing of the watchers * [Fix] Fix queue id macro in milter * [Fix] Fix R_BAD_CTE_7BIT rule * [Fix] Fix Redis timeout set * [Fix] Fix REPLYTO_UNPARSEABLE rule * [Fix] Fix setting of email address * [Fix] Fix some more issues about duplicated fuzzy requests * [Fix] Fix spamc support in rspamd proxy * [Fix] Fix syntax error in spamtrap plugin * [Fix] Fix url counts for href urls * [Fix] Fix url handling in the protocol * [Fix] Multimap: Received IP filters with Redis * [Fix] Oops, fix d9d0fa5e86 * [Fix] Parse rgb[a](x,x,x[,x]) css colors * [Fix] Phishing: strict_domains * [Fix] Reduce maps aggressiveness * [Fix] Reresolve upstreams even if there is a single server there * [Fix] Rspamadm grep: Disable Lua patterns in string search by default * [Fix] Skip text parts when checking binary parts in fuzzy check * [Fix] Support v2 checks in controller * [Fix] Treat empty address as valid * [Fix] Try harder to detect CTE * [Fix] Try to deal with v4 mapped to v6 addresses on accept * [Fix] Use dkim signing callback properly * [Fix] Use non-volatile memory for storing data * [Fix] Use static maps instead of ugly hack for radix_from_config * [Fix] Use the same pool for related sessions * [Rework] Continue modularisation for lua library * [Rework] Initial milter protocol support * [Rework] Make log pipes worker agnostic, add scanners API * [Rework] Move authentication results generation to a separate routine * [Rework] Move common DKIM functions to a separate lua module * [Rework] Move global functions to a separate directory * [Rework] Prepare dkim module for ARC checks * [Rework] Propagate ucl variables from the command line * [Rework] Remove multiple metrics support from Rspamd * [Rework] Stop using name 'rmilter' for the modern protocol * [Rework] Use LFU algorithm in LRU cache * [Rules] Fix received TLS rules * [Rules] Improve URL_COUNT_ODD rule * [WebUI] Fix add header filter in history * [WebUI] Use modern protocol for checking messages Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	9 years ago
Vsevolod Stakhov	a92e55249f	[Minor] Update ChangeLog from 1.5 branch	9 years ago
Sebastien Badia	dd64721ae6	[Minor] debian: Update BD and refresh packaging	9 years ago
Vsevolod Stakhov	27ea426985	Release 1.5.6 * [Feature] Add unigramms support in bayes * [Feature] Allow configurable sign headers for DKIM * [Feature] Allow to add unigramm metatokens from Lua * [Feature] DKIM Signing: envelope match exception for local IPs * [Feature] UCL: register parser variables from Lua * [Fix] Always try to adjust filename * [Fix] Do extra copy to ensure that original content is never touched * [Fix] Fix SPOOF_REPLYTO rule * [Fix] Ignore Rmilter added Received * [Fix] More fixes for hashed email dnsbls * [Fix] Plug memory leak in chartable module * [WebUI] Display multiple alerts at once Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	9 years ago
Vsevolod Stakhov	939ab1342e	Release 1.5.5 * [CritFix] Fix classifier learning with Redis backend * [CritFix] Fix issue when parsing encoded rfc822/messages * [Feature] Add escaped version of lua_ucl import * [Feature] Add task:headers_foreach function * [Feature] Allow to process filenames from content type * [Feature] Allow to query hashed emails * [Feature] Ignore bayes with mostly metatokens or with too few text * [Feature] Probabilistically skip metatokens * [Feature] Retrieve all virus names from SAVAPI * [Feature] Rework classifiers lua metatokens * [Feature] Store headers order * [Feature] Store text tokens inside bayes tokens * [Feature] Use cached shingles keys * [Fix] Add missing score normalisation for HFILTER_URL_ONLY * [Fix] Avoid lookup in absent hash * [Fix] Check return values from Lua functions called from C * [Fix] Do not count sending and loading time in rspamc * [Fix] Escape json strings for controller rejplies from Lua * [Fix] Fix archive scans for savapi * [Fix] Fix domain_only emails RBL * [Fix] Fix ip_score map configuration * [Fix] Fix JSON output for history_redis * [Fix] Fix one character length substrings search * [Fix] Fix parsing of non-RFC compatible Exim received * [Fix] Fix parsing of options for workers with the same type * [Fix] Fix processing of small tokens vectors * [Fix] Fix rfc2047 tokenization * [Fix] Fix typo * [Fix] More fixes for inplace decoding * [Fix] Try to avoid modifications of the original data * [Fix] URL redirector: Fix call to is_redirector * [Rework] Set token data as uint64_t instead of chars array * [WebUI] Check if neighbours' history backend versions match * [WebUI] Disable phrase connectors replacement in history filtering * [WebUI] Disable phrase connectors replacement in symbols filtering * [WebUI] Do not hide messages with bad subject, just replace it with '???' * [WebUI] Fix error message * [WebUI] Fix history v2 display * [WebUI] Fix legacy history * [WebUI] history: break To address lists on commas * [WebUI] Increase default timeout to 20 seconds * [WebUI] Save some history table space Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	9 years ago
Vsevolod Stakhov	0839f87a25	Release 1.5.4 * [Conf] Add history_redis default configuration * [Feature] Add spoofed rules * [Feature] Add URL_IN_SUBJECT rule * [Feature] Allow to get task's subject * [Feature] Allow to specify maximum number of shots for symbols * [Feature] Distinguish URLs found in Subject * [Feature] Memoize LPEG grammars * [Feature] Parse else parts in SA rules * [Feature] Process subject for mixed characters * [Feature] Resolve url chains in url_redirector module * [Feature] Stat greylisted messages as greylisted not soft-rejected * [Feature] Support checking for redirector in Lua SURBL * [Feature] Support tag_exists SA function * [Feature] Work with broken rfc2047 tokens * [Fix] Check all watcher's dependencies * [Fix] Do not compile hyperscan with no SSSE3 support * [Fix] Do not crash if cannot decode qp encoded part * [Fix] Fix dependencies of DKIM when multiple signatures are found * [Fix] Fix lists in whitelist plugin * [Fix] Fix one-shot symbols weight calculations * [Fix] Fix options and shots match * [Fix] Fix order of symbol options * [Fix] Fix parsing of dot at the end of the address * [Fix] Fix parsing of lua table arguments * [Fix] Fix processing of subject words * [Fix] Fix string split memoization * [Fix] Fix templates grammar usage * [Fix] Fix various issues related to Lua stack manipulation * [Fix] Force actions: Use postfilter if we have honor_action / require_action * [Fix] Further fixes to avoid PHISHING FP * [Fix] Preserve order of options in symbols * [Fix] Rspamadm grep: deal with unusually-formatted logs * [Fix] Use hostname suffix when dealing with history * [Rework] Remove outdated SA rules * [WebUI] Add flexible columns * [WebUI] Add footable * [WebUI] Add sender, recipients and subject columns * [WebUI] Allow message-id break * [WebUI] Fix history clustering * [WebUI] Fix history display * [WebUI] Fix sorting * [WebUI] Humanize sizes * [WebUI] Initial move towards footable * [WebUI] Remove datatables * [WebUI] Replace `.values` method with `.map` * [WebUI] Rework v2 symbols display * [WebUI] Try to normalize frequencies * [WebUI] Unbreak WebUI * [WebUI] Use Footable to draw Throughput summary table Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	9 years ago
Vsevolod Stakhov	ca1a968cf5	Release 1.5.3 * [Conf] Add composite for hacked wordpress phishing * [CritFix] Fix base64 decoding when there are unparseable characters * [Feature] Additional symbol metadata in metadata exporter * [Feature] Add method to get protocol reply from Lua * [Feature] Add symbols when tagged rcpt/sender are normalised * [Feature] Add task:get_symbols_all() function * [Feature] Allow multiple formats of DKIM signing key * [Feature] Allow to cache and use flexible protocol reply * [Feature] Allow to set one_shot flag from register_symbol * [Feature] Allow to skip certain types of hashes when learning fuzzy * [Feature] Cache and insert scan time into the protocol * [Feature] Detect newlines in rspamc --mime * [Feature] DKIM signing: support use of maps * [Feature] Greylist: Support excluding low-scoring messages from greylisting * [Feature] Implement lua history in controller * [Feature] Implement redis history querying * [Feature] Preliminary implementation of redis history plugin * [Feature] Support using request headers in settings * [Fix] Change default template to deal with non-ASCII characters * [Fix] Deal with lists of maps in whitelist module * [Fix] DKIM signing: use domain-specific signing key * [Fix] Do not reallocate completed zstd buffer * [Fix] Do not use local_addrs in proxy * [Fix] Fix crash when resolver is undefined * [Fix] Fix double free when closing lua_tcp connections * [Fix] Fix for lua 5.3 * [Fix] Fix freeing of arrays iterators * [Fix] Fix issue with task:get_symbol and symbols with no metric * [Fix] Fix log line duplication in `rspamadm grep` * [Fix] Fix memory corruption on termination * [Fix] Fix out-of-bound access in base64 decode * [Fix] Fix ratelimit + greylisting * [Fix] Fix subject rewriting * [Fix] Fix task:set_recipients function * [Fix] Fix URI_COUNT_ODD rule * [Fix] Follow the traditional symbols conventions in RCPT_COUNT rule * [Fix] Greylist: Suppress greylist action for whitelisted hosts too * [Fix] Metadata exporter: use rule-specific settings for emails * [Fix] Properly set missing fields in exporter * [Fix] Proxy: max_retries option * [Fix] RCPT_COUNT fixes * [Fix] Rework HAS_X_PRIO rule to match symbols conventions * [Fix] Update issues in ac-trie * [Fix] Use optimised base64 decoding in DKIM * [WebUI] Add preliminary v2 history parser * [WebUI] Allow different history parsers * [WebUI] Display symbols * [WebUI] Rework history v2 function Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	10 years ago
Vsevolod Stakhov	898c9c7b5f	Release 1.5.2 * [Conf] Add default config for spamassasssin plugin * [Conf] Add default configuration for antivirus module * [Conf] Add dkim signing docs * [Conf] Add mx_check default config * [Conf] Add replies config * [Conf] Add trie default config * [Feature] Add heuristic to find text parts in files * [Feature] Add rule to detect broken content type * [Feature] Allow to extract CTE in Lua API * [Feature] Allow to set from address for a lua_task * [Feature] Allow to set recipients of a task from Lua * [Feature] Enchance text_part:get_content method * [Feature] Remove + aliases from emails * [Feature] Support rmilter block and dkim signature in CGP helper * [Feature] Support running event loop from Lua * [Fix] Antivirus: use scanner-specific redis prefix * [Fix] Couple of fixes for DKIM signing module * [Fix] Distinguish missing and broken mandatory headers * [Fix] Do more heuristical detection for missing CTE * [Fix] Do not resort cache on each check * [Fix] Fix CGP escaping * [Fix] Fix MISSING_MIME_VERSION rule for plain messages * [Fix] Fix parsing of cte in expressions * [Fix] Fix partial matches in rspamadm grep * [Fix] Fix setting class on style field * [WebUI] Auto-switch Throughput units to `msg/min` for very low rate * [WebUI] Update D3Evolution to 0.0.2 Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	10 years ago
Vsevolod Stakhov	f6c7e98b74	Release 1.5.1 * [CritFix] Fix processing of stop_patterns with `\0` character * [CritFix] Fix setting of raw key for signing * [Fix] Fix lua exports from plugins during reload * [Fix] Fix prefilters action scores * [Fix] Fix symbols processing order * [Minor] Help cmake find gthread * [Minor] Some cmake fixes Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	10 years ago
Vsevolod Stakhov	9d2e7e0e2c	Release 1.5.0 * [Conf] Add configurations for asn, clickhouse and dcc * [Conf] Add default config for url redirector plugin * [Conf] Add the default config for greylist module * [Conf] Allow to edit all local maps from WebUI by default * [CritFix] Deal with absent headers in DKIM * [CritFix] Do not trust remote shingles count * [CritFix] Fix bad memory leak in TLS certificates validation * [CritFix] Fix critical memory issues with radix maps * [CritFix] Fix descriptors leak on reload * [CritFix] Fix headers selection in DKIM verification * [CritFix] Fix parsing of boundaries that end with `--` * [CritFix] Repair PTR_ARRAY_FOREACH macro * [Feature] Add CORS support to the controller * [Feature] Add FROM_NAME_EXCESS_SPACE rule * [Feature] Add REPLYTO_EMAIL_HAS_TITLE rule * [Feature] Add `caseless_hash` method to `lua_util` * [Feature] Add `rip` keyword to ratelimit module * [Feature] Add a simple benchmark for content type parsing * [Feature] Add boundaries parsing in content type * [Feature] Add charset detection for text parts * [Feature] Add content disposition parser * [Feature] Add fallback if too many updates are failing * [Feature] Add function to convert struct tm to time using timezone * [Feature] Add function to normalize HTTP paths * [Feature] Add fuzzy collection plugin * [Feature] Add fuzzy logic for images * [Feature] Add gmime parser to mime_tool * [Feature] Add heuristic to detect broken messages * [Feature] Add heuristic to find displayed URLs * [Feature] Add heuristic to process broken email addresses * [Feature] Add images normalization * [Feature] Add mechanism for disabling composites (Fixes #1270) * [Feature] Add method to create regexp from a glob pattern * [Feature] Add mime encoding manipulation routines * [Feature] Add mime tool to explore messages * [Feature] Add more meta tokens from received headers * [Feature] Add neighbours option to support Rspamd cluster in WebUI * [Feature] Add new function to parse mime addresses * [Feature] Add new methods for lua_tcp * [Feature] Add own headers decoding routine * [Feature] Add own routine to generate a message id * [Feature] Add parser for SMTP date * [Feature] Add per-task lua cache to reuse 'heavy' objects * [Feature] Add plugins list path in WebUI * [Feature] Add preliminary multipart support * [Feature] Add preliminary version of DKIM signing module * [Feature] Add profiling support in client output * [Feature] Add rfc2047 grammar * [Feature] Add rfc2047 variant for QP decoding * [Feature] Add rmilter_headers module (Fixes #1227) * [Feature] Add sse42 version of base64 decoding * [Feature] Add ssse3 and avx2 base64 decoders * [Feature] Add support of libgd * [Feature] Add the preliminary version of redirects resolver in Lua * [Feature] Add ucl_object_iterate_full function * [Feature] Add url encoding function * [Feature] Allow SOA requests in lua dns * [Feature] Allow custom parse types in lua ucl * [Feature] Allow plugins to register webui handlers * [Feature] Allow to add options explicitly to symbols * [Feature] Allow to call a callback when symbol frequency is on peak * [Feature] Allow to call redirector script from SURBL * [Feature] Allow to create variable length dkim keys * [Feature] Allow to have module specific options for Redis in plugins * [Feature] Allow to pass sign key directly from Lua * [Feature] Allow to register configuration docs from Lua API * [Feature] Allow to return options as a table * [Feature] Allow to set peak callbacks from Lua * [Feature] Allow to specify custom method for a message * [Feature] Allow to store dkim keys in Redis * [Feature] Allow to store messages in files * [Feature] Apply DCT using AAN for fuzzy signature * [Feature] Avira SAVAPI support * [Feature] Cache and simplify DCT and jpeg decode * [Feature] Cache libicu convertors * [Feature] Detect URLs with suspicious omographs * [Feature] Do not increase score for duplicate options * [Feature] Do not trust CTE, check base64 and qp strictly * [Feature] Dynamic reputation in URL reputation plugin * [Feature] Extend redis lock when learning spawned * [Feature] Filter non-utf chars from all decoded headers * [Feature] Fix phishing detection for IDNA urls * [Feature] Ignore bad symbols on base64 decoding * [Feature] Ignore too wide elements in SPF * [Feature] Implement fuzzy collection mode * [Feature] Implement helo maps in multimap * [Feature] Implement human readable buckets configuration * [Feature] Implement min-hash shingles for DCT data from images * [Feature] Implement new algorithm for fuzzy hashes of images * [Feature] Implement new unicode normalizer * [Feature] Implement quoted printable decoding * [Feature] Implement received headers flags * [Feature] Implement rspamdgrep tool * [Feature] Implement sane checksum for config file * [Feature] Implement url tags concept * [Feature] Improve detection of omographs using libicu * [Feature] Improve url redirector module * [Feature] Multimap: Received header processing * [Feature] Multiple improvements in the maps * [Feature] New URL filters in multimap * [Feature] Plugin to force actions on selected symbols * [Feature] RBL module: support hashing for emails and helo RBL * [Feature] Reuse URL tags in SURBL module * [Feature] Rework RRD ds count, add conversion path * [Feature] Rework surbl module to avoid extra redirector calls * [Feature] Send config id to the WebUI * [Feature] Simplify HTTPCrypt client support * [Feature] Skip processing for large images * [Feature] Start collection only mode implementation for fuzzy storage * [Feature] Start import of the optimized base64 decode * [Feature] Store all received headers in lua * [Feature] Store relational order of all headers in a message * [Feature] Support DKIM signing in Lua plugins * [Feature] Support HTTPCrypt client in lua_http * [Feature] Support setting SMTP message in multimap * [Feature] Support setting metric subject from Lua * [Feature] Support setting subject in force actions module * [Feature] Treat v6 mapped addresses as v4 addresses * [Feature] URL reputation plugin * [Feature] Use Redis instead of memcached in URLs redirector * [Feature] Use Rspamd rfc2047 decoder instead of gmime one * [Feature] Use a different normalization for fuzzy images * [Feature] Use normalized images in fuzzy hashes * [Feature] Use own code for parsing of date * [Feature] Use shingles for images fuzzying * [Feature] Use t1ha for hashes, allow inlining * [Feature] Use t1ha instead of metrohash and xxhash32 * [Feature] Various new features in metadata exporter module * [Feature] rmilter_headers: authentication-results (#78) * [Fix] Add additional check to mark redis connection inactive * [Fix] Add packed attribute for protocol structure * [Fix] Adopt OMOGRAPH_URL rule * [Fix] Allow static maps * [Fix] Allow to disable classifiers checks using settings and conditions * [Fix] Another try to fix 0 length maps * [Fix] Another try to fix corruption during maps reload * [Fix] Another try to fix descriptors leak * [Fix] Another try to fix reload and logger * [Fix] Antivirus module: register virtual symbols for patterns * [Fix] Avoid extensive reallocs * [Fix] Avoid mempool leak in SA plugin on reload * [Fix] Avoid race condition on saving cache and reload * [Fix] Avoid reusing g_error (Fixes #1262) * [Fix] Break pool connection on fatal redis errors * [Fix] Check for NaN properly * [Fix] Couple of fixes for date parsing * [Fix] Date header timezone adjustments (#1279) * [Fix] Deal with EOF properly * [Fix] Decode filename in content disposition * [Fix] Disable fuzzy images by default * [Fix] Disable zero-copy mode for text parts to avoid crashes * [Fix] Do not destroy session when not all finish scripts are done * [Fix] Do not greyscale images * [Fix] Do not leave parent-less workers processes on fatal errors * [Fix] Do not lowercase Content-Disposition to perform decoding * [Fix] Do not penalize characters just after numeric prefix * [Fix] Do not refork workers that are intended to die * [Fix] Do not set pre-result and update records for no `Queue-ID` messages * [Fix] Do not skip post-filters when pre-filters have set some results * [Fix] Do not stop symbols planning if async events are pending * [Fix] Do not try to set keys for unencrypted requests in proxy * [Fix] Encode URLs according to rfc3986 * [Fix] Encode URLs before sending them to the protocol * [Fix] Filter bad characters from message id * [Fix] Fix CTE detection heuristic * [Fix] Fix Content-Type in HTTP requests * [Fix] Fix IDN eslds phishing checks * [Fix] Fix adding maps from config in Lua * [Fix] Fix another reload memory issue * [Fix] Fix argument returned on redis backend errors * [Fix] Fix assertion in graph handling * [Fix] Fix body trie matching * [Fix] Fix build * [Fix] Fix byte array expansion during toutf8 conversion * [Fix] Fix charset normalisation * [Fix] Fix checking of DKIM bodies that needs just `\n` to be added * [Fix] Fix couple of cornercases with email addresses * [Fix] Fix couple of issues * [Fix] Fix dependencies tracking for callback symbols * [Fix] Fix detection of jpeg size * [Fix] Fix errors handling in fuzzy backend initialization * [Fix] Fix fuzzy hashes count * [Fix] Fix globbing and convert lists to arrays in fuzzy_check * [Fix] Fix heuristical CTE detection for QP encoding * [Fix] Fix ignoring of bad text parts * [Fix] Fix indexes in array access, interleave loop * [Fix] Fix int64 -> double conversion * [Fix] Fix invalid memory access on reload * [Fix] Fix issues with empty updates * [Fix] Fix issues with quoted-printable encoding * [Fix] Fix keys names * [Fix] Fix lots of issues in mime parser code * [Fix] Fix lua maps load * [Fix] Fix macro name * [Fix] Fix mas group score calculations * [Fix] Fix matching of the same patterns from different tries * [Fix] Fix memory corruprtion and leak * [Fix] Fix memory leak in HTTP maps * [Fix] Fix memory leak in expression destroying * [Fix] Fix memory leak in parsing of mime names * [Fix] Fix memory leak in safe ucl iterators * [Fix] Fix memory leak on reload in plugins * [Fix] Fix modules reconfigure on reload * [Fix] Fix monitored setup fro URLBLs with IP addresses * [Fix] Fix name of var * [Fix] Fix new rrd updates * [Fix] Fix out of bounds access * [Fix] Fix parsing messages with no body * [Fix] Fix parsing of '=' character in headers * [Fix] Fix parsing of messages with no content type * [Fix] Fix plugins callbacks in webui * [Fix] Fix possible memory corruption in redis pool * [Fix] Fix probability calculations for fuzzy redis backend * [Fix] Fix processing errors in lua_tcp * [Fix] Fix processing of emails with name only * [Fix] Fix processing of non-multipart messages * [Fix] Fix processing of parts with no valid content type * [Fix] Fix race condition in SIGUSR2 handler * [Fix] Fix redis options parsing when no redis servers are defined * [Fix] Fix reload and hyperscan ready event * [Fix] Fix reload memory issue * [Fix] Fix rra_ptr conversion * [Fix] Fix rrd file conversion * [Fix] Fix setting of content-type attributes * [Fix] Fix signing headers creation in DKIM * [Fix] Fix stddev calculations * [Fix] Fix surbl plugin to work with composite maps * [Fix] Fix timezones parsing * [Fix] Fix tokens usage * [Fix] Fix urls and emails hashes * [Fix] Fix usage of unsafe ucl iterators * [Fix] Fix work with broken utf8 tokens * [Fix] Fix writing of user to roll history * [Fix] Forgotten worker * [Fix] Further memory leaks fixes * [Fix] Ignore lua metatokens in bayes for now * [Fix] Improve OMOGRAPH_URL rule * [Fix] Lua IP from string should be invalid if parsing failed * [Fix] Miltiple fixes to new lua_tcp, add debugging * [Fix] More fixes for iterators cleanup * [Fix] More fixes to logger initialization * [Fix] More heuristic fixes for phishing detection * [Fix] More leaks eliminated * [Fix] More leaks... * [Fix] More random fixes for reload... * [Fix] Multimap: Fixes for email filters * [Fix] Multiple fixes for fann module * [Fix] Multiple memory corruption fixes * [Fix] Normalize path in HTTP router * [Fix] Plug memory leak * [Fix] Plug memory leak in adding radix trees * [Fix] Plug memory leak in configuration parser * [Fix] Plug memory leak in expressions parsing during reload * [Fix] Plug memory leak in learning fuzzy storage * [Fix] Plug memory leak in lua_tcp * [Fix] Plug reload leaks * [Fix] Plug termination memory leaks * [Fix] Really increase lock lifetime * [Fix] Replies module: fix symbol weight * [Fix] Restore content type params related functions * [Fix] Set task's subject from mime subject * [Fix] Sigh, one more reload leak * [Fix] Simplify images shingles * [Fix] Some more memory issues are fixed * [Fix] Stop hardcoding of lua in C * [Fix] Stop processing of bad parts as text parts * [Fix] Strictly filter bad characters when emittin json * [Fix] Strings returned from lua are ephemeral * [Fix] Support unix sockets for lua redis * [Fix] Try to fix issues with reloading config * [Fix] Try to fix race condition in redis_pool * [Fix] Use checksum to avoid intersection between different ANNs * [Fix] Use rspamd hashes in embedded ucl * [Fix] Use sane default rewrite subject (*** SPAM *** %s) * [Fix] Various collection mode fixes * [Fix] Various fixes to mime parser * [Fix] Various reload leak fixing * [Fix] Whitelist certain extensions from archive checks * [Rework] Add preliminary implementation of the mime parser * [Rework] Adopt code for the new options * [Rework] Change logger setup interface * [Rework] Composite configuration (#1270) * [Rework] Finally remove gmime dependency from Rspamd * [Rework] Further fixes to symbols frequencies * [Rework] Implement content type parser for mime * [Rework] Kill all InternetAddressList usages * [Rework] Multiple fixes for symbols cache statistics * [Rework] Refactor struct names * [Rework] Rework images fuzzy hashes algorithm * [Rework] Rework lua_tcp to allow TCP dialog * [Rework] Start massive rework to get rid of gmime * [Rework] Start new approach for multiparts parsing * [Rework] Start rework of mime addresses * [Rework] Start rework of symbols cache updates * [Rework] Start switching to libicu * [Rework] Use a special structure for stats tokens * [Rework] Use hash tables for symbols options * [Rework] Use libicu instead of iconv for conversions * [Rework] Use new scheme to parse mime parts * [WebUI] Add Access-Control-Allow-Origin for cluster management * [WebUI] Add Throughput graph autorefreshing (#820) * [WebUI] Add Visibility.js library * [WebUI] Add basic cluster support to Throughput tab * [WebUI] Add graph legend entries for new DSes * [WebUI] Add graph tab * [WebUI] Add neighbours RRD data consolidation * [WebUI] Add preliminary save symbols clustering * [WebUI] Add server selector to navbar * [WebUI] Add soft reject to auth stats * [WebUI] Add summary to the Throughput tab * [WebUI] Allow to save maps on the cluster * [WebUI] Avoid extra graph redraw and alerts glitching * [WebUI] Be more generous with AJAX timeout * [WebUI] Disable error ring loading in `read only` mode * [WebUI] Enclose table header cells with `tr`s * [WebUI] Finish interface rework * [WebUI] Fix RRD summary pie chart position * [WebUI] Fix `All SERVERS` graph fot just one available server * [WebUI] Fix case when no cluster is defined * [WebUI] Fix compatibility with non-ES6 compliant browsers * [WebUI] Fix config ID * [WebUI] Fix configuration page partially * [WebUI] Fix disabled state * [WebUI] Fix graph dataset selector initialization * [WebUI] Fix graph selectors state resetting * [WebUI] Fix mouse events on throughput summary table area * [WebUI] Fix multiple JS issues * [WebUI] Fix pie chart displaying * [WebUI] Fix read only * [WebUI] Fix read only2 * [WebUI] Fix retarded datatables * [WebUI] Fix soft reject in pie chart * [WebUI] Fix stat widgets timers multiplication on `Refresh` click * [WebUI] Fix symbols config * [WebUI] Fix various errors with login form * [WebUI] Further fixes * [WebUI] Hide learning tab in read-only mode * [WebUI] Initial clusters support * [WebUI] Make legend entry colours more contrast * [WebUI] Move configuration tab to a separate module * [WebUI] Move history tab * [WebUI] Move symbols config as well * [WebUI] New sec to time function * [WebUI] Prevent multiple clicks on `Refresh` * [WebUI] RRD summary: Hide inner labels of tiny pie sectors * [WebUI] RRD summary: Respect undefined values * [WebUI] Reduce font size of graph's legend * [WebUI] Remove orphaned font duplicates * [WebUI] Remove unused code * [WebUI] Replace spinner with animated glyphicon * [WebUI] Reset refresh timer on server switching * [WebUI] Rework interface to use requirejs * [WebUI] Rework neighbours query function * [WebUI] Separate attributes by space * [WebUI] Set focus to password field (#1230) * [WebUI] Simplify neighbours table populating * [WebUI] Start rework of modules * [WebUI] Stop stats refreshing if the page is hidden * [WebUI] Turn d3pie's stuff into a reusable function, * [WebUI] Unify send data functions * [WebUI] Update D3Evolution to 0.0.1 * [WebUI] Update d3.js * [WebUI] Update datatables to work with the requirejs * [WebUI] Use unified tab click event handler, * [WebUI] clusters for the chart * [WebUI] fix uptime Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	10 years ago
Vsevolod Stakhov	37573488e2	Release 1.4.2 * [CritFix] Deal with absent headers in DKIM * [CritFix] Do not trust remote shingles count * [CritFix] Fix headers selection in DKIM verification * [Feature] Add EXT_CSS rule * [Feature] Add toggle for disabling SURBLs * [Feature] Extend redis lock when learning spawned * [Feature] Parse <link> HTML tags * [Fix] Avoid reusing g_error (Fixes #1262) * [Fix] Do not reset loaded ANN when learning is requested * [Fix] Fix another issue with external deps in SA * [Fix] Fix body trie matching * [Fix] Fix checking of DKIM bodies that needs just `\n` to be added * [Fix] Fix fuzzy hashes count * [Fix] Fix keys names * [Fix] Fix length calculations for url encoded urls * [Fix] Fix matching of the same patterns from different tries * [Fix] Fix name of var * [Fix] Fix parsing of URLs with spaces and other bad chars * [Fix] Fix probability calculations for fuzzy redis backend * [Fix] Fix signing headers creation in DKIM * [Fix] Plug memory leak * [Fix] Really fix chained SA dependencies * [Fix] Really increase lock lifetime * [Fix] Use checksum to avoid intersection between different ANNs * [Fix] Use rspamd hashes in embedded ucl * [Fix] Yet another change for testing external deps Signed-off-by: Vsevolod Stakhov <vsevolod@highsecure.ru>	10 years ago