Initial commit

Add README, LICENSE, requirements file and helper scripts, minor code
cleanup

.gitignore

@@ -0,0 +1,31 @@
+### OS-specific
+
+# Linux
+*~
+*.swp
+.fuse_hidden*
+.directory
+
+# Windows
+Thumbs.db
+ehthumbs.db
+ehthumbs_vista.db
+*.stackdump
+[Dd]esktop.ini
+$RECYCLE.BIN/
+*.lnk
+
+### Python
+__pycache__/
+*.py[cod]
+*$py.class
+.env
+.venv
+env/
+venv/
+ENV/
+*.egg
+*.egg-info/
+
+### Pycharm
+.idea/

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2020 Vladislav Glinsky
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,45 @@
+# Trello-to-Kanboard migration
+Command-line scripts written to automate migration from Trello boards to Kanboard projects.
+
+## Setup
+
+Install dependencies with `pip install -r requirements.txt`.
+
+## Usage
+
+To get started you'll need to:
+
+1. Get an Trello API key and API secret from https://trello.com/app-key.
+2. Obtain an Trello OAuth token, there are at least two ways:
+    - generate it at https://trello.com/app-key, this token doesn't need token secret;
+    - run py-trello package from command line with `TRELLO_API_KEY` and `TRELLO_API_SECRET` environment variables set accordingly:
+    
+        ```
+        python -m trello oauth
+        ```
+
+3. Generate Kanboard User API access token (My profile → API).
+
+
+### Board-to-project migration
+
+For migration you'll have to specify ids of existing Kanboard project and Trello board (you can get them with helper scripts). You should add member users to Kanboard project beforehand.
+
+Migration logic is implemented via `Migrator` class, it accepts instances Kanboard and Trello clients, board and project ids. There's a `migration_example.py` [script](migration_example.py) that you can use as a starting point for your migration process.
+ 
+Trello labels assigned to cards can be converted to Kanboard task priority, complexity and tags according to their text. `Migrator` will try to preserve link between cards and will create "related to" link between corresponding tasks. Files attached to cards will be uploaded to Kanboard instance if they are less than configured size limit (set to 3 MB by default) otherwise they will be added as external link.
+
+`Migrator` will try to match Kanboard project user to Trello board member by name if no mapping for that user was specified explicitly. This can be done by adding `Trello member name`-`Kanboard user id` pairs to `users_map`. Card with multiple assigned users will be migrated to the task with tag "migration: more than one member", one of the users will be assigned to the task and other users will be mentioned at the end of the task's description. There's [Group_assign](https://github.com/creecros/Group_assign) Kanboard plugin that should provide functionality similar to Trello's multiple assignment but it didn't exist at the time when I was writing migration script.
+
+To migrate dates correctly you have to specify Kanboard instance timezone (due to issues [3653](https://github.com/kanboard/kanboard/issues/3653) and [3654](https://github.com/kanboard/kanboard/issues/3654)).
+
+### Helper scripts
+
+There are two simple scripts for retrieving lists of Trello boards and Kanboard projects that the user is a member of: `list_trello_boards.py` and `list_kanboard_projects.py`. Usage details for them are shown below:
+
+
+    python list_kanboard_projects.py your_kanboard_instance_url
+User is defined by `KANBOARD_USERNAME` and `KANBOARD_TOKEN` environment variables.
+
+    python list_trello_boards.py show
+User is defined by `TRELLO_API_KEY`, `TRELLO_API_SECRET`, `TRELLO_TOKEN` and `TRELLO_TOKEN_SECRET` environment variables accordingly.

list_kanboard_projects.py

@@ -0,0 +1,34 @@
+#!/usr/bin/env python3
+import os
+import sys
+from urllib.parse import urljoin
+from kanboard import Client as KanboardClient, ClientError
+
+
+def usage():
+    print("""Usage: list_kanboard_projects Kanboard_instance_URL
+
+    Shows ids and names of Kanboard projects that the user specified with
+    KANBOARD_USERNAME and KANBOARD_TOKEN environment variables is a member of.""")
+
+
+def main():
+    if len(sys.argv) < 2:
+        usage()
+        return
+    instance_url = sys.argv[1]
+    api_url = urljoin(instance_url, 'jsonrpc.php')
+    api_user = os.getenv('KANBOARD_USERNAME')
+    api_token = os.getenv('KANBOARD_TOKEN')
+    client = KanboardClient(url=api_url, username=api_user, password=api_token)
+    try:
+        projects = client.get_my_projects()
+    except ClientError as e:
+        print('Fail:', e)
+        return
+    for p in projects:
+        print(p['id'], p['name'])
+
+
+if __name__ == '__main__':
+    main()

list_trello_boards.py

@@ -0,0 +1,40 @@
+#!/usr/bin/env python3
+import os
+import sys
+
+from trello import TrelloClient, ResourceUnavailable
+
+
+def usage():
+    print("""Usage: list_trello_boards show
+
+    Shows ids and names of Trello boards that the user specified with
+    TRELLO_API_KEY, TRELLO_API_SECRET, TRELLO_TOKEN and TRELLO_TOKEN_SECRET
+    environment variables is a member of.""")
+
+
+def main():
+    if len(sys.argv) < 2 or sys.argv[1].lower() != 'show':
+        usage()
+        return
+    api_key = os.getenv('TRELLO_API_KEY')
+    api_secret = os.getenv('TRELLO_API_SECRET')
+    api_token = os.getenv('TRELLO_TOKEN')
+    api_token_secret = os.getenv('TRELLO_TOKEN_SECRET')
+    client = TrelloClient(
+        api_key=api_key,
+        api_secret=api_secret,
+        token=api_token,
+        token_secret=api_token_secret
+    )
+    try:
+        boards = client.list_boards()
+    except ResourceUnavailable as e:
+        print('Fail:', e)
+        return
+    for b in boards:
+        print(b.id, b.name)
+
+
+if __name__ == '__main__':
+    main()

migration_example.py

@@ -0,0 +1,64 @@
+#!/usr/bin/env python3
+#
+# This is an example script for migrating single Trello board to already existing
+# Kanboard project (you
+# NB: for the sake of example all provided identifiers/tokens/keys here are randomly generated
+# and you have to provide your own to perform migration. Please consult README for details.
+#
+
+import re
+
+import pytz
+from kanboard import Client as KanboardClient
+from trello import TrelloClient
+
+from migrator import Migrator
+
+
+def _add_label_mappings(registry, pairs):
+    for regex, value in pairs:
+        registry[re.compile(regex, re.I)] = value
+
+
+class MigrateFactory:
+    def __init__(self):
+        self._trello_client = TrelloClient(
+            api_key='73256545e41ez1101a7x1c2280yf4600',
+            api_secret='fdfba7d4287279b43f7x60261702c2c19340c6y18178589372d1d69661z6a3c3',
+            token='1dd336d2e571ae0d9506620be008xy7543a6360443ff28f0ece033cc7345625b',
+            token_secret='f07b4c4eaedda3e3168050xyz9c4eae1'
+        )
+        self._kanboard_client = KanboardClient(
+            url='https://localhost/jsonrpc.php',
+            username='user',
+            password='4a24bc173c208a6f5512637320309wxf2e03af69eb765ec6016a8d7e9f7f'
+        )
+
+    def migrate(self, board_id, project_id):
+        m = Migrator(self._trello_client, self._kanboard_client, board_id, project_id)
+        # Customize Trello labels handling: use them to determine task priority and complexity
+        _add_label_mappings(
+            m.complexity_map,
+            [('complexity: low', 2), ('complexity: medium', 5), ('complexity: high', 7)]
+        )
+        _add_label_mappings(
+            m.priority_map,
+            [('priority: low', 1), ('priority: medium', 3), ('priority: high', 5)]
+        )
+        # Explicitly map Trello usernames to Kanboard user ids instead of relying
+        # on Migrator's attempts to match user by names
+        # In any case users have to be added to the project beforehand
+        m.users_map.update({'root': 0, 'admin': 1})
+        # Workaround for kanboard issues #3653, #3654
+        # (https://github.com/kanboard/kanboard)
+        m.kanboard_timezome = pytz.timezone('Europe/Kiev')
+        # Override attachment size limit
+        m.attachment_max_size = 2 * 1024 * 1024
+        m.run()
+
+
+if __name__ == '__main__':
+    MigrateFactory().migrate(
+        board_id='e787e73314d22x516bf115cb',
+        project_id=3
+    )

migrator.py

@@ -0,0 +1,353 @@
+import base64
+from datetime import datetime
+from urllib.parse import urlparse
+
+import pytz
+import requests
+
+
+class Migrator:
+    kanboard_datetime_format = '%Y-%m-%d %H:%M'
+    trello_datetime_format = '%Y-%m-%dT%H:%M:%S.%fZ'
+
+    def __init__(self, trello_client, kanboard_client, board_id, project_id):
+        self._kanboard = kanboard_client
+        self._trello = trello_client
+
+        self.users_map = {}
+        self.complexity_map = {}
+        self.priority_map = {}
+        self.attachment_max_size = 3 * 1024 * 1024  # bytes
+
+        # Workaround for kanboard issues #3653, #3654
+        # (https://github.com/kanboard/kanboard)
+        self.kanboard_timezome = pytz.UTC
+
+        self.project_id = project_id
+        self.board = self._trello.get_board(board_id)
+        self.lists = self.board.get_lists('all')
+
+        self._members2users = {}
+        self._migrated_cards = {}
+        self._unmigrated_relations = set()
+
+    def run(self):
+        lists2columns = self._create_columns_from_lists()
+        project_users = self._kanboard.get_project_users(project_id=self.project_id)
+        members = self.board.all_members()
+        self._members2users = self._map_members2users(members, project_users)
+
+        for list_id, column in lists2columns.items():
+            column_id = column['id']
+            board_list = column['origin']
+            self._migrate_column(board_list, column_id)
+
+        for a, b in self._unmigrated_relations:
+            a_id = self._migrated_cards.get(a)
+            b_id = self._migrated_cards.get(b)
+            if not a_id or not b_id:
+                print('Skip relation for tasks {} {} - some of them are not migrated'.format(a, b))
+                continue
+
+            self._add_task_relation(a_id, b_id)
+
+    def _create_columns_from_lists(self):
+        columns = self._kanboard.get_columns(project_id=self.project_id)
+        column_names = {c['title']: c for c in columns}
+        mapped = {}
+
+        print('Migrate lists to columns:')
+        for board_list in self.lists:
+            status = 'done'
+            column = None
+            name = board_list.name
+            if name not in column_names:
+                column_id = self._kanboard.add_column(
+                    project_id=self.project_id,
+                    title=name
+                )
+
+                if type(column_id) is int:
+                    column = {
+                        'id': column_id,
+                        'title': name,
+                        'origin': board_list
+                    }
+                else:
+                    status = 'failed'
+            else:
+                status = 'already exists'
+                column = column_names[name]
+                column['origin'] = board_list
+
+            if column:
+                mapped[board_list.id] = column
+
+            print('  "{name}" [{status}]'.format(name=name, status=status))
+
+        print('Removing extra columns...')
+        for c in columns:
+            if c.get('origin'):
+                continue
+
+            success = self._kanboard.remove_column(column_id=c['id'])
+            print('  {id} "{title}" - {status}'.format(
+                **c,
+                status='done' if success else 'failed'
+            ))
+        print('complete')
+
+        return mapped
+
+    def _map_members2users(self, members, project_users):
+        mapped = {}
+        for m in members:
+            user_id = self.users_map.get(m.username)
+            if user_id is None:
+                match_names = lambda i, n: m.username == n or m.full_name == n
+                matched = list(filter(match_names, project_users.values()))
+
+                if not matched:
+                    print(' No match found for member "{}"'.format(m.username))
+                    continue
+
+                if len(matched) > 1:
+                    print(' More than one match found for member "{}": {}'.format(
+                        m.username,
+                        ', '.join(n for _, n in matched)
+                    ))
+                    continue
+                user_id, _ = matched
+            mapped[m.id] = {'id': user_id, 'username': m.username}
+        return mapped
+
+    def _migrate_card_members(self, member_ids):
+        if not member_ids:
+            return None, []
+
+        owner_id = None
+        mentions = []
+        for i in member_ids:
+            matched = self._members2users.get(i)
+            if not matched:
+                mentions.append(i)
+                continue
+
+            if owner_id:
+                mentions.append('@{}'.format(matched['username']))
+            else:
+                owner_id = matched['id']
+        return owner_id, mentions
+
+    def _add_task_relation(self, a_id, b_id):
+        task_link_id = self._kanboard.create_task_link(
+            task_id=a_id,
+            opposite_task_id=b_id,
+            link_id=1  # generic "related to", don't blame me for hardcoding this
+        )
+        if _operation_failed(task_link_id):
+            print(' Failed to add related task {} link for task {}'.format(
+                a_id, b_id
+            ))
+            return None
+        return task_link_id
+
+    def _migrate_column(self, board_list, column_id):
+        cards = board_list.list_cards('all')
+        for card in cards:
+            card.fetch(eager=False)
+            task_id = self._migrate_card(column_id, card)
+
+            if task_id is None:
+                print('Failed to create task "{}" in column "{}"'.format(
+                    card.name,
+                    board_list.name
+                ))
+                continue
+
+            if card.is_due_complete or card.closed:
+                self._kanboard.close_task(task_id=task_id)
+
+            self._migrated_cards[card.id] = task_id
+            self._migrate_comments(task_id, card.comments)
+            self._migrate_checklists(task_id, card.checklists)
+            attachments = card.get_attachments()
+            self._migrate_attachments(card.id, task_id, attachments)
+
+    def _migrate_checklists(self, task_id, checklists):
+        status_map = {True: 2, False: 0}
+        for checklist in checklists:
+            for item in checklist.items:
+                subtask_id = self._kanboard.create_subtask(
+                    task_id=task_id,
+                    title=item['name'],
+                    status=status_map[item['checked']]
+                )
+                if type(subtask_id) is bool:
+                    print('  Failed to migrate checklist "{}" item "{}"'.format(
+                        checklist.name, item.name
+                    ))
+
+    def _migrate_card(self, column_id, card):
+        priority, score, tags = self._map_labels(card.labels)
+        description = card.description
+        owner_id, mentions = self._migrate_card_members(card.member_id)
+        if mentions:
+            description += '\n\nOther members of original card: ' + ' '.join(mentions)
+            tags.append('migration: more than one member')
+
+            print(' More than one member on "{}" card'.format(card.name))
+
+        date_due = card.due
+        if date_due:
+            date_due = _convert_datetimes(
+                card.due, Migrator.trello_datetime_format, self.kanboard_timezome
+            ).strftime(Migrator.kanboard_datetime_format)
+
+        task_id = self._kanboard.create_task(
+            title=card.name,
+            project_id=self.project_id,
+            column_id=column_id,
+            owner_id=owner_id,
+            date_due=date_due,
+            description=description,
+            score=score,
+            priority=priority,
+            tags=tags
+        )
+
+        return None if _operation_failed(task_id) else task_id
+
+    def _migrate_comments(self, task_id, comments_json):
+        for c in comments_json:
+            author_id = c['idMemberCreator']
+            author_id = self._members2users[author_id]['id']
+            text = c['data']['text']
+            comment_id = self._kanboard.create_comment(
+                task_id=task_id,
+                user_id=author_id,
+                content=text
+            )
+            if _operation_failed(comment_id):
+                print('Failed to migrate comment "{}" by {}'.format(
+                    text, c['idMemberCreator']
+                ))
+
+    def _get_related_card_id(self, url):
+        url_parts = urlparse(url)
+        if url_parts.netloc != 'trello.com':
+            return None
+
+        path_components = _extract_path_components(url_parts.path)
+        if len(path_components) < 2 or path_components[0] != 'c':
+            return None
+
+        related_card = self._trello.get_card(card_id=path_components[1])
+        if related_card.idBoard == self.board.id:
+            return related_card.id
+
+        return None
+
+    def _migrate_attachments(self, card_id, task_id, attachments):
+        related_links_set = set()
+        for attachment in attachments:
+            if attachment.is_upload:
+                if not self._try_reupload_attachment(attachment, task_id):
+                    self._add_external_link_to_task(attachment, task_id)
+                continue
+
+            # handle related cards separately
+            related_card_id = self._get_related_card_id(attachment.url)
+            if related_card_id:
+                if related_card_id == card_id or related_card_id in related_links_set:
+                    continue
+
+                related_task = self._migrated_cards.get(related_card_id, None)
+                if not related_task:
+                    self._unmigrated_relations.add((card_id, related_card_id))
+                    continue
+
+                task_link_id = self._add_task_relation(
+                    task_id,
+                    related_task
+                )
+                if task_link_id is not None:
+                    self._unmigrated_relations.discard((related_card_id, card_id))
+                    related_links_set.add(related_card_id)
+                continue
+
+            self._add_external_link_to_task(attachment, task_id)
+
+    def _try_reupload_attachment(self, attachment, task_id):
+        if self.attachment_max_size < attachment.bytes:
+            print(' Attachment "{}" @{} is too big({}), migrated as url'.format(
+                attachment.name, task_id, attachment.bytes
+            ))
+            return False
+
+        r = requests.get(attachment.url)
+        file_id = self._kanboard.create_task_file(
+            project_id=self.project_id,
+            task_id=task_id,
+            filename=attachment.name,
+            blob=base64.b64encode(r.content).decode('utf-8')
+        )
+        if _operation_failed(file_id):
+            print(' Failed to add file "{}", migrated as url'.format(
+                attachment.name
+            ))
+            return False
+        return True
+
+    def _add_external_link_to_task(self, attachment, task_id):
+        link_id = self._kanboard.create_external_task_link(
+            task_id=task_id,
+            url=attachment.url,
+            dependency='related',
+            title=attachment.name
+        )
+        if _operation_failed(link_id):
+            print(' Failed to add link "{}" for attachment "{}"'.format(
+                attachment.url,
+                attachment.name
+            ))
+
+    def _map_labels(self, labels):
+        priority = None
+        score = None
+        tags = []
+        for label in labels:
+            if not label.name:  # just label with color, ignore for now
+                continue
+
+            name = label.name
+            v = _match_name(name, self.complexity_map)
+            if v:
+                score = v
+                continue
+            v = _match_name(name, self.priority_map)
+            if v:
+                priority = v
+                continue
+            tags.append(name)
+        return priority, score, tags
+
+
+def _extract_path_components(path):
+    return list(filter(None, path.split('/')))
+
+
+def _operation_failed(id_):
+    return type(id_) is bool
+
+
+def _convert_datetimes(datetime_utc_str, utc_format, timezone):
+    d = datetime.strptime(datetime_utc_str, utc_format)
+    return d.replace(tzinfo=pytz.UTC).astimezone(timezone)
+
+
+def _match_name(name, mapping):
+    for regex, value in mapping:
+        if regex.match(name):
+            return value
+    return None

requirements.txt

@@ -0,0 +1,4 @@
+kanboard>=1.1.2
+py-trello>=0.16.0
+requests>=2.24.0
+pytz>=2020.1