Mirrors
/
data-importer
mirror of https://github.com/firefly-iii/data-importer.git
3
0
Fork 0
Code Releases Activity
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
2657 Commits
6 Branches
299 Tags
28 MiB
 
 
 
 
 
Tree: d2148da489
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'd2148da489'
${ noResults }
data-importer/changelog.md

39 KiB
Raw Blame History

Change Log

All notable changes to this project will be documented in this file. This project adheres to Semantic Versioning.

v1.7.9 - 2025-07-31

Added

Fixed

v1.7.8 - 2025-07-20

⚠️ Some changes in this release may unexpectedly lead to duplicate transactions. This is caused by changes in the data handling routines. This is unfortunate, but a result of new insights, changed APIs and other minor fixes. My apologies for any inconvenience. I try to avoid these kinds of changes, but it can't always be helped.

Fixed

v1.7.7 - 2025-07-16

Changed

  • Various SimpleFIN optimizations, drops the "CORS URL" from the required fields.

Fixed

v1.7.6 - 2025-07-15

⚠️ Some changes in this release may unexpectedly lead to duplicate transactions. This is caused by changes in the data handling routines. This is unfortunate, but a result of new insights, changed APIs and other minor fixes. My apologies for any inconvenience. I try to avoid these kinds of changes, but it can't always be helped.

Added

  • Log entries will occasionally include the data importer version, so that you can see which version was used to import data.
  • Issue 8296 (GoCardless Revolute currency exchange) reported by @TomVarga

Changed

  • Issue 10556 (Save the connection string for subsequent SimpleFIN imports) reported by @Gsyltc

Fixed

v1.7.5 - 2025-07-06

Fixed

  • Improved duplicate detection for SimpleFIN imports.

v1.7.4 - 2025-07-06

Fixed

v1.7.3 - 2025-07-04

Fixed

  • Restore duplicate detection options
  • SimpleFIN import will not always try to create accounts

v1.7.2 - 2025-07-03

Fixed

  • Issue 10541 (Type error in str_replace function, breaks GoCardless import during configuration) reported by @alexmaisel

v1.7.1 - 2025-07-02

⚠️ Some changes in this release may unexpectedly lead to duplicate transactions. This is caused by changes in the data handling routines. This is unfortunate, but a result of new insights, changed APIs and other minor fixes. My apologies for any inconvenience. I try to avoid these kinds of changes, but it can't always be helped.

Changed

  • For GoCardless imports: opposing account names and other reference data is no longer added as tags, but as text in the notes. This may lead to duplicates.
  • GoCardless: better handling of rate limits.

Fixed

  • Issue 10508 (Firefly Version 1.7.0 uses SimpleFIN for GoCardless and crashes when importing) reported by @L3tum
  • Improved parsing of dates.

v1.7.0 - 2025-06-26

Added

  • Support for SimpleFIN, developed by user 'skell'

Fixed

  • Better check on Authorization header for users who use Basic Auth.
  • Issue 5396 (Data importer: SimpleFIN) reported by @marcusds, with special thanks again to skell, and all the beta testers who helped out!

v1.6.3 - 2025-06-08

⚠️ Some changes in this release may unexpectedly lead to duplicate transactions. This is caused by changes in the data handling routines. This is unfortunate, but a result of new insights, changed APIs and other minor fixes. My apologies for any inconvenience. I try to avoid these kinds of changes, but it can't always be helped.

Added

  • Add some more debug logging.
  • Issue 8994 (Creditor Reference Information) reported by @fschirinzi
  • Issue 9533 (Extension of GoCardess imports with tags from custom transaction fields) reported by @kondas

Changed

  • Data Importer generated transaction IDs are prefixed with ff3-.
  • Account numbers have their spaces removed, for easier matching in Firefly III.

Fixed

  • Discussion 10162 (Reverse proxy and X-Forwarded-Prefix header) started by @frenchu
  • Issue 10387 (Failing to import inverted transactions when there is no account name and the account number does not exist) reported by @oddstr13
  • PR 802 Ensure all token acquisition methods are the same by @grgar
  • PR 803 Ignore exception opening counter-main.txt by @grgar
  • PR 814 Consider account number uniquely identifying by @oddstr13

v1.6.2 - 2025-04-21

Added

Fixed

1.6.1 - 2025-02-19

⚠️⚠️ This release required PHP 8.4.0 and will not work on earlier releases of PHP ⚠️⚠️

⚠️ Some changes in this release may unexpectedly lead to duplicate transactions. This is caused by changes in the data handling routines. This is unfortunate, but a result of new insights, changed APIs and other minor fixes. My apologies for any inconvenience. I try to avoid these kinds of changes, but it can't always be helped.

Fixed

1.6.0 - 2025-01-31

⚠️⚠️ This release required PHP 8.4.0 and will not work on earlier releases of PHP ⚠️⚠️

⚠️ Some changes in this release may unexpectedly lead to duplicate transactions. This is caused by changes in the data handling routines. This is unfortunate, but a result of new insights, changed APIs and other minor fixes. My apologies for any inconvenience. I try to avoid these kinds of changes, but it can't always be helped.

Changed

Due to changes in the Docker image, the commands that use WEB_SERVER=false to automatically import data need another docker container. You can read more about it in the documentation.

  • Better rate limit reporting for Nordigen
  • Better exit codes
  • Data is cached, should make the importer slightly faster

Fixed

Added

  • Support for PHP 8.4
  • Discussion 9626 (Allow newlines in description/Add hook points to Data importer process) started by @Sobuno

Removed

  • Support for < PHP 8.4

v1.5.7 - 2024-11-10

Added

  • Add debug information for rate limits from GoCardless
  • Pull request 686 (Add DB as Debit indicator) by @akhy

Changed

Fixed

  • Better error catching for rate limits from GoCardless
  • Fix link to documentation
  • Issue 9180 (Data importer - gocardless EUA is expired - print clear message instead of 500 server error) reported by @Ovenlade
  • Issue 9340 (Default transaction import without year to current year) reported by @mokkun
  • Issue 9347 (Importing transaction with huge external id throws error) reported by @diogotcorreia
  • Issue 9345 (data importer - error lines don't match to actual lines in csv) reported by @rel-thanu

1.5.6 - 2024-09-15

Changed

  • Improved GoCardless rate limiting handling

Fixed

  • GoCardless rate limit could report negative time.
  • Documentation links

1.5.5 - 2024-09-05

Changed

  • Better GoCardless rate limiting reporting

Fixed

  • Issue 9194 (data importer via gocardless EU - details of the account is not imported anymore after last two versions) reported by @fate8383

1.5.4 - 2024-08-22

Added

  • Issue 9139 (Implement rate limiting for GoCardless) reported by @JC5

1.5.3 - 2024-07-28

Added

  • Support for another bank's debit indicator, by @0xjams

Fixed

  • Case-sensitive matching on bank debit indicator, by @rajabilal555
  • Discussion 9067 (Unable to automate: Data-importer doesn't remember which accounts spectre accounts map to.) started by @demiGod095
  • Issue 9044 (Minor issue in AutoImport.php - does not catch a 'bad path' error) reported by @Terry-JF

1.5.2 - 2024-04-24

Added

Fixed

1.5.1 - 2024-04-18

Fixed

  • Issue 8798 (Undefined variable $version) reported by @JC5
  • Fix an issue with default currency selection.

1.5.0 - 2024-04-18

Added

  • The data importer has a release flow just like Firefly III, dropping generated JS files and other dependencies from the repository.
  • Issue 8665 (API exception if transaction is deleted by rule) reported by @Medformatik
  • Issue 8691 (Import gocardless "merchantCategoryCode") reported by @tboby

Changed

  • Rewrote front-end layout to AlpineJS + Laravel Blade, moving away from Twig + Vue.
  • Some rules in the PR template

Fixed

1.4.5 - 2024-03-21

Added

Changed

Fixed

v1.4.4 - 2024-02-18

Fixed

1.4.3 - 2024-02-05

Changed

  • The data importer can connect to nightly builds of Firefly III, which self-report as version develop/YYYY-MM-DD.

Fixed

v1.4.2 - 2024-01-22

Added

Fixed

  • Issue 8367 Bad type error, reported by @quotary
  • Issue 8389 (Data Importer stops Autoimport of multiple Nordigen/GoCardless files if one has zero transactions) reported by @rerobade
  • Issue 8404 (Data importer confuses thousands separator for decimal separator when decimal separator is missing.) reported by @nonsavant
  • Issue 8408 (Importing camt.053 file fails during conversion step) reported by @ltouroumov
  • Issue 8414 (Getting Unexpected data found. trying to import via GoCardless/Nordigen) reported by @dabeeeenster

v1.4.1 - 2024-01-15

Added

  • Nordigen supports "bookingDateTime"

Fixed

  • Issue 8267 Fix reference to variable
  • Issue 8294 issue 8355 Null pointer fixed
  • Issue 8293 Don't stop importing when one connection has no transactions
  • Issue 8344 The UI will report progress again
  • Issue 8355 Foreign amounts would not be imported
  • Fix log levels in papertrail configuration
  • The UI will email reports again

1.4.0 - 2023-12-18

⚠️⚠️ This release required PHP 8.3.0 and will not work on earlier releases of PHP ⚠️⚠️

⚠️ Some changes in this release may unexpectedly lead to duplicate transactions. This is caused by changes in the data handling routines. This is unfortunate, but a result of new insights, changed APIs and other minor fixes. My apologies for any inconvenience. I try to avoid these kinds of changes, but it can't always be helped.

Added

  • GoCardless-users are forwarded to refresh their agreement if necessary.

Changed

  • ⚠️⚠️ Added support for PHP 8.3 ⚠️⚠️
  • Requires Firefly III v6.1.0 or higher

Removed

  • ⚠️⚠️ Removed support for PHP 8.2 ⚠️⚠️

Fixed

  • Issue 8000 Inform user when there are no transactions found by the Nordigen Data Importer
  • Issue 6500 Transactions with 0.0 amount are not imported
  • Issue 6470 use the "ultimateCreditor"-field for Nordigen transactions
  • Issue 5460 Add balance info in configuration screen
  • Access token from Authorization-header is forwarded to Firefly III if necessary.
  • All documentation links are fixed again.
  • Improve text in configuration screen.
  • Expand notifications so logging can be more easily parsed.
  • Nordigen / GoCardless now saves the booked / pending tags as well.

1.3.12 - 2023-11-23

Added

1.3.11 - 2023-11-23

Changed

  • Add some more explanation text to the date input

Fixed

1.3.10 - 2023-11-21

⚠️ Some changes in this release may unexpectedly lead to duplicate transactions. This is caused by changes in the data handling routines. This is unfortunate, but a result of new insights, changed APIs and other minor fixes. My apologies for any inconvenience. I try to avoid these kinds of changes, but it can't always be helped.

Added

Changed

  • Renamed Nordigen to GoCardless

Fixed

1.3.9 - 2023-11-04

⚠️ Some changes in this release may unexpectedly lead to duplicate transactions. This is caused by changes in the data handling routines. This is unfortunate, but a result of new insights, changed APIs and other minor fixes. My apologies for any inconvenience. I try to avoid these kinds of changes, but it can't always be helped.

Added

  • Fancy new security.txt file.

Changed

  • Better safety catches for Nordigen and Spectre, in case they accidentally try to create deposits or withdrawals that should be transfers.

Fixed

  • Issue 7948 Null pointer exception in CAMT processing.
  • Issue 8005 Sometimes the importer does not recognize the importer account
  • Issue 8045 Bad caching leads to unfixable authentication errors
  • Issue 8052 Credit card transactions after a certain date are not imported while all other transactions are
  • Issue 8207 Data Importer ignoring relevant Nordigen error.

1.3.8 - 2023-09-02

Fixed

  • Issue 7918 Make sure XML file are recognized by the POST importer

1.3.7 - 2023-09-02

Fixed

  • The POST import command and the CLI import routine will also stop working if the connected Firefly III is too old.

1.3.6 - 2023-09-02

⚠️ This release may create duplicate transactions. Don't import large batches!

Fixed

  • Issue 7891 Importing from Spectre has improved, taking into account new API information. This may change the look and feel of your transactions.
  • Some banks would return amount in scientific notation.
  • Several CAMT fixes when the file is missing data.
  • CAMT fix to verify the transaction type.
  • Better flow for the onboarding of new users. Updated the documentation to match.

1.3.5 - 2023-08-23

Fixed

  • Error in CSV import would break imports for many users, thanks for reporting!
  • Many null pointers removed from CAMT.053 import.

1.3.4 - 2023-08-21

⚠️ This release may create duplicate transactions. Don't import large batches!

Added

  • Unmapped fields are now sorted at the top for CSV files.
  • Spectre import will include IBAN and name, not just "name or IBAN".

Fixed

  • Notes are properly trimmed for import
  • Account search will report on its results in the logs
  • Fixed several camt.053 unstructured data import issues.
  • Issue 7816 Importer would break when there was no currency information in a camt file.

1.3.3 - 2021-07-17

Fixed

  • Formatting issue in debug log message broke the CAMT import

1.3.2 - 2023-07-16

Added

  • Better error handling for CAMT files.

Changed

  • New way of saving user info, should make it easier to change variables without running into old values.

Fixed

  • Issue 7729 Make sure account type detection can't be overruled by later attempts.
  • Issue 7747 Nullpointer in CAMT import

1.3.1 - 2021-06-27=8

⚠️ This release may create duplicate transactions. Don't import large batches!

Changed

  • This version of the data importer requires Firefly III v6.0.16 or higher.

Fixed

  • Many small and large fixes in the new CAMT.053 import routine. Thanks again, @beatbesmer and bug testers!
  • Issue 7643 Bad redirect when trying to import using a config file.

v1.3.0 - 2023-06-05

  • ⚠️ This release may create duplicate transactions. Don't import large batches!

CAMT.053 support

GitHub user @beatbesmer has been working hard to make the data importer support CAMT.053 files. You can now upload these files like you would a CSV file and have them imported into Firefly III. Although this new feature has been thorougly tested by Beat and myself, CAMT.053's XML standards are a very loose standard so expect bugs, issues and crashes. We welcome your feedback in the form of bug reports

Added

  • Issue 6108 You can customize the import tag used by the data importer

Fixed

  • Issue 5996 Cannot send email using self-signed certificates
  • Issue 7512 Broken links in app and documentation
  • Issue 7264 Data Importer crashes when mail server auth fails

v1.2.2 - 2023-03-19

  • ⚠️ This release may create duplicate transactions. Don't import large batches!

Changed

  • Drop accuracy to 12 decimals. This fixes a lot of rounding issues.

Fixed

  • Issue 7225 Fix email error when no vanity url is set

1.2.1 - 2023-03-13

Fixed

  • Issue 7214 Previous issue with account currency matching needed a Data Importer fix as well.

Changed

  • This release will only work with Firefly III v6.0.4

1.2.0 - 2023-03-13

Fixed

Changed

  • This release will only work with Firefly III v6.0.3

1.1.0 - 2023-03-12

Added

  • Warnings when you use bad configuration values.
  • IMPORT_DIR_WHITELIST is now IMPORT_DIR_ALLOWLIST

Changed

  • Improve logging when reporting on possible duplicates.

1.0.2 - 2022-01-27

Fixed

  • Missing array key breaks Nordigen and Spectre imports.

1.0.1 - 2022-01-26

Fixed

  • Fixed missing options when importing CSV file.
  • Remove APP_KEY generation since the data importer doesn't use one.

1.0.0 - 2022-01-23

  • ⚠️ This release requires PHP 8.2.
  • ⚠️ This release may create duplicate transactions. Don't import large batches!

Added

Changed

  • ⚠️ Use Nordigen's internal transaction ID as the transaction ID.
  • Switch to Mastodon in the readme.
  • Fix the link to the PHP docs, thanks @sa7mon!
  • Small fixes to the .env.example file, thanks @axelsimon!

Fixed

  • Issue 6518 Issue with bad account listings
  • Fix bad method call
  • Make sure CSV import values are trimmed (prevents account names like PAYPAL )

0.9.17 - 2021-10-24

⚠️ This release WILL create duplicate transactions. Don't import large batches!

Added

  • Better error handling.

Fixed

0.9.16 - 2022-09-12

Added

  • Ability to recognize 'Charge' as a debit transaction.
  • Expand error messages with URL.
  • Log more errors.

Fixed

  • Issue 6354 Path fix for subdirectory installations.
  • Issue 6377 Switch issue were transaction amounts were reversed (and thus also the types)
  • Issue 6412 Update for command line import.
  • Issue 6419 Time out and SSL check wasn't consistent.

Security

  • Update packages

0.9.15 - 2022-07-25

Fixed

  • Issue 6259 Fixed an issue where deposits would not be given a source account.

0.9.14 - 2022-07-21

Added

  • Fallback to additional information field if description is empty, thanks @martinslota

Fixed

  • Issue 6242 Bad string conversion when dates are empty.
  • Issue 6244 Rename field from 'uri' to 'url'.
  • Fix missing field data check for spectre imports.

0.9.13 - 2022-07-19

Fixed

  • Fixed an issue where the importer would break when importing mapped transactions.

0.9.12 - 2022-07-16

Fixed

  • Make sure packages match PHP 8.0

0.9.11 - 2022-07-16

Fixed

  • Issue where config files with empty strings would break the date selection.
  • Issue 6146 Fix issue where missing Nordigen data would break the import.
  • Issue 6233 Fix issue where mapping an expense account to a revenue spot would break the import.
  • Fix issue where the importer would not detect empty descriptions. Thanks @BerghsJelmer

0.9.10 - 2022-05-03

Changed

  • Require the most recent version of Firefly III

0.9.9 - 2022-05-03

Changed

  • Made the texts more generic to accommodate future changes
  • Catch deleted transactions
  • Require most recent Firefly III

Fixed

0.9.8 - 2022-04-13

Fixed

  • Fixes an issue with an underlying package.

0.9.7 - 2022-04-11

Fixed

  • Fixes an issue with an underlying package.

0.9.6 - 2022-04-11

Fixed

  • Fixes an issue with an underlying package.

0.9.5 - 2022-04-10

Fixed

  • Fixes an issue with an underlying package.

0.9.4 - 2022-04-10

Fixed

  • Fixes another issue with detecting duplicate transactions.

0.9.3 - 2022-04-05

Fixed

  • A small change should make it easier for the data importer to detect failed transaction creation attempts.
  • Issue 5964 Updated libraries to fix issue.

0.9.2 - 2022-04-02

Fixed

  • Removed some overly chatty status messages.

0.9.1 - 2022-04-02

Added

  • There is an option to set the maximum days to import from Nordigen, thanks @krehl!
  • Importer will now report expired Nordigen User Agreements, thanks @MacPaille!

Fixed

0.9.0 - 2022-02-22

  • ⚠️ This release WILL create duplicate transactions. Don't import large batches!

Added

  • Add command to upgrade import configurations.

Changed

  • Stop logging stack traces to prevent logs from filling up.
  • ⚠️ Don't submit unused amount_modifier field

Fixed

  • Better error catching in configuration controller and other places.
  • Sanity check to prevent the importer from treating JSON files as CSV files.
  • Issue 5731 Could not import transfers from Spectre
  • Issue 5735 Better support for old import configuration files

0.8.0 - 2022-02-07

  • ⚠️ Using Nordigen? This release WILL create duplicate transactions. Don't import large batches.

Added

  • Collect more Nordigen information for future debug.
  • Collect IBAN from secondary source if possible.
  • Add value date as "payment date" whenever possible.

Fixed

  • Issue 5590 500 error when attempting to add bank account from Nordigen
  • Issue 5650 Better error catching for timeouts
  • Issue 5700 Missing route for Nordigen (experimental fix)
  • Clean up some logging.
  • Better check on session states
  • Better check on messages to report
  • Will not email if not configured to

0.7.0 - 2022-01-22

  • ⚠️ Using Nordigen? This release WILL create duplicate transactions. Don't import large batches.

A special thanks to the excellent folks over at @nordigen for some quick debugging and fixing.

Added

  • A special 500 page so you can see what's happening.
  • Nordigen will now import transaction details in notes.
  • If you have no Firefly III accounts, the importer will warn you.
  • Extra account details debug information.

Fixed

  • A missing method broke navigation.
  • Sometimes session details would get lost in translation.

0.6.5 - 2022-01-15

Fixed

  • Catch error in balanceAfterTransaction field.

0.6.4 - 2022-01-15

Fixed

  • ownerAddressUnstructured could unexpectedly be unset.

0.6.3 - 2022-01-12

Fixed

  • Nordigen reports the ownerAddressUnstructured both as string and array, thanks @dawid-czarnecki

0.6.2 - 2022-01-12

Fixed

  • Issue 5507 ownerAddressUnstructured field is of the wrong data type.
  • Version 0.6.1 also reports as 0.6.0.

0.6.1 - 2022-01-08

Fixed

  • Missing continue statement in transaction processor

0.6.0 - 2022-01-08

  • ⚠️ Duplicate detection could be broken, due to changes in the way transactions are handled. Be careful importing large batches.
  • 💡 Some people have reported running into loops when trying to start importing CSV files. Please open an issue if this happens to you.

Added

  • The Spectre import checks more fields for payee information, thanks @ddelbondio!

Changed

  • The importer can stop warning you about duplicate transactions, making for a cleaner import. Check out .env.example.
  • If there is nothing to report, there will be no email message.
  • The import tag will not be created until the first transaction is successfully imported.
  • The configuration file export will mention the data importer version.

Fixed

  • Various issues where people would end up in a redirect loop.
  • In some cases, the "mapping" feature was hidden.
  • The debug page could add newlines, breaking the table.
  • The autoupload endpoint would require CSV files, even when using Nordigen or Spectre.
  • Issue 5502 Sloppy copy/paste bug leads to confusion between the base URL and the vanity URL.

0.5.0 - 2022-01-01

  • ⚠️ Duplicate detection could be broken, due to changes in the way transactions are handled. Be careful importing large batches.
  • 💡 Some people have reported running into loops when trying to start importing CSV files. Please open an issue if this happens to you.

Added

Changed

  • Issue 5453 Different text for button
  • Importer will complain about bad environment variables.
  • Only create the import tag when necessary.

Fixed

  • Issue 5354 Fix edge case when importing CSV files.
  • Issue 5440 Can now handle amounts formatted 0,xxxxx
  • Issue 5452 Bad vanity URL in reports
  • Issue 5459 Fix issue when skipping configuration page.
  • Filter spaces from IBANs

0.4.1 - 2021-12-23

  • ⚠️ Duplicate detection could be broken, due to changes in the way transactions are handled. Be careful importing large batches.
  • 💡 Some people have reported running into loops when trying to start importing CSV files. Please open an issue if this happens to you.

Added

  • Dark mode. Responds to your browser or OS.

Fixed

  • Issue 5416 Mismatch in function name breaks Nordigen.

0.4.0 - 2021-12-22

Known issues

  • ⚠️ Duplicate detection could be broken, due to changes in the way transactions are handled. Be careful importing large batches.
  • 💡 Some people have reported running into loops when trying to start importing CSV files. Please open an issue if this happens to you.

Changed

  • The importer will search for, and match account numbers as well as IBANs.
  • The importer will prefer 'bookDate' over 'valueDate'

Added

  • POST import and import via upload is now possible. Several mandatory security measures are listed in .env.example.

Fixed

0.3.0 - 2021-12-11

Added

  • Issue 5342 Can import "external URL" field.
  • Will now send report messages over mail if you want.

Fixed

  • Remove TODO's from README file.
  • Respect date range when importing.
  • Nordigen will pre-select country + bank
  • Nordigen will recycle requisition
  • Mapping works for Spectre + Nordigen
  • Better secret management for Nordigen + Spectre
  • Issue 5351 Fix URL's
  • Fix duplicate detection.
  • Fix issue with transfers being imported as deposits.
  • New debit indicator thanks to @reeperbahnause

0.2.0 - 2021-12-04

Added

  • There is a /debug page if you need it.
  • Auto-import works over the command line.
  • Importing works over the command line.
  • Full documentation in place.

Changed

  • More consistent layout

Fixed

0.1.0 - 2021-11-27

First release of the data importer.

v0.0.0 - 2021-xx-xx

Added

  • Initial release.

Changed

  • Initial release.

Deprecated

  • Initial release.

Removed

  • Initial release.

Fixed

  • Initial release.

Security

  • Initial release.