75 Commits (d83be23689c50f81d2df040cd047e9100e004c3f)

Author SHA1 Message Date
Brett Cannon 45b96d373e Merged revisions 88498 via svnmerge from 15 years ago
Alexander Belopolsky b9d10d08c4 Issue #10386: Added __all__ to token module; this simplifies importing 15 years ago
Victor Stinner 58c0752a33 Issue #10335: Add tokenize.open(), detect the file encoding using 15 years ago
Raymond Hettinger a0e79408bc A little bit more readable repr method. 16 years ago
Raymond Hettinger 3fb79c747b Experiment: Let collections.namedtuple() do the work. This should work now that _collections is pre-built. The buildbots will tell us shortly. 16 years ago
Raymond Hettinger 6c60d099e5 Improve the repr for the TokenInfo named tuple. 16 years ago
Florent Xicluna 43e4ea1b17 Remove unused import, fix typo and rewrap docstrings. 16 years ago
Benjamin Peterson 66428b2e5d Merged revisions 84364 via svnmerge from 16 years ago
Benjamin Peterson 33856de84d handle names starting with non-ascii characters correctly #9712 16 years ago
Benjamin Peterson 1613ed8108 fix for files with coding cookies and BOMs 16 years ago
Benjamin Peterson 689a558098 in tokenize.detect_encoding(), return utf-8-sig when a BOM is found 16 years ago
Benjamin Peterson 81dd8b9594 use some more itertools magic to make '' be yielded after readline is done 16 years ago
Benjamin Peterson 21db77e396 simply by using itertools.chain() 16 years ago
Benjamin Peterson 8f6713f46d Merged revisions 76235 via svnmerge from 16 years ago
Benjamin Peterson a0dfa82eca Merged revisions 75149,75260-75263,75265-75267,75292,75300,75376,75405,75429-75433,75437,75445,75501,75551,75572,75589-75591,75657,75742,75868,75952-75957,76057,76105,76139,76143,76162,76223 via svnmerge from 16 years ago
Benjamin Peterson 0c7f9c96f5 Merged revisions 75299 via svnmerge from 16 years ago
Benjamin Peterson d3afadaa49 normalize latin-1 and utf-8 variant encodings like the builtin tokenizer does 16 years ago
Raymond Hettinger aa17a7fc98 Remove dependency on the collections module. 17 years ago
Raymond Hettinger a48db39992 Issue #5857: tokenize.tokenize() now returns named tuples. 17 years ago
Benjamin Peterson 9b8d24b17d reuse tokenize.detect_encoding in linecache instead of a custom solution 17 years ago
Benjamin Peterson 433f32c3be raise a SyntaxError in detect_encoding() when a codec lookup fails like the builtin parser #4021 17 years ago
Antoine Pitrou fd036451bf #2834: Change re module semantics, so that str and bytes mixing is forbidden, 18 years ago
Benjamin Peterson 0fe14383a8 use the more idomatic (and Py3k faster) while True 18 years ago
Christian Heimes ba4af493a5 Merged revisions 61964-61979 via svnmerge from 18 years ago
Trent Nelson 428de65ca9 - Issue #719888: Updated tokenize to use a bytes API. generate_tokens has been 18 years ago
Georg Brandl fceab5a385 Merged revisions 60080-60089,60091-60093 via svnmerge from 18 years ago
Guido van Rossum 4fe72f9b03 Patch 1420 by Ron Adam. 18 years ago
Collin Winter ce36ad8a46 Raise statement normalization in Lib/. 19 years ago
Guido van Rossum cd16bf6404 Merged revisions 55817-55961 via svnmerge from 19 years ago
Guido van Rossum 1bc535dc78 Merged revisions 55328-55341 via svnmerge from 19 years ago
Georg Brandl a18af4e7a2 PEP 3114: rename .next() to .__next__() and add next() builtin. 19 years ago
Nick Coghlan 650f0d06d3 Hide list comp variables and support set comprehensions 19 years ago
Georg Brandl dde002899d Make ELLIPSIS a separate token. This makes it a syntax error to write ". . ." for Ellipsis. 19 years ago
Guido van Rossum be19ed77dd Fix most trivially-findable print statements. 19 years ago
Neal Norwitz c150536b5e PEP 3107 - Function Annotations thanks to Tony Lownds 19 years ago
Thomas Wouters 89f507fe8c Four months of trunk changes (including a few releases...) 19 years ago
Brett Cannon cf588f6448 Remove support for backticks from the grammar and compiler. 20 years ago
Guido van Rossum b053cd8f40 Killed the <> operator. You must now use !=. 20 years ago
Thomas Wouters 00ee7baf49 Merge current trunk into p3yk. This includes the PyNumber_Index API change, 20 years ago
Thomas Wouters 49fd7fa443 Merge p3yk branch with the trunk up to revision 45595. This breaks a fair 20 years ago
Raymond Hettinger da99d1cbfe SF bug #1224621: tokenize module does not detect inconsistent dedents 21 years ago
Raymond Hettinger 68c0453418 Add untokenize() function to allow full round-trip tokenization. 21 years ago
Anthony Baxter c2a5a63654 PEP-0318, @decorator-style. In Guido's words: 22 years ago
Guido van Rossum 68468eba63 Get rid of many apply() calls. 23 years ago
Raymond Hettinger 78a7aeeb1a SF 633560: tokenize.__all__ needs "generate_tokens" 24 years ago
Guido van Rossum 9d6897accc Speed up the most egregious "if token in (long tuple)" cases by using 24 years ago
Tim Peters 8ac1495a6a Whitespace normalization. 24 years ago
Raymond Hettinger d1fa3db52d Added docstrings excerpted from Python Library Reference. 24 years ago
Tim Peters 496563a514 Remove some now-obsolete generator future statements. 24 years ago
Neal Norwitz e98d16e8a4 Cleanup x so it is not left in module 24 years ago