py3to2 2009.01.03
run python3.0 & 2to3 generated scripts w/ 2.6 extension support
License: BSD

this project is being superseded by asciiporn at http://code.google.com/p/asciiporn/

AUTHOR
  kai zhu
  kaizhu256@gmail.com

REQUIREMENTS
  - posix/unix os (Windows currently unsupported)
  - w/ python2.6 & python3.0 installed

INSTALL
  $ python2.6 setup.py build
  $ python2.6 setup.py install
  $ python2.6 setup.py dev --quicktest
  $ python2.6 setup.py dev --py2to3test ## takes awhile to finish

  the above will build & install 3 files:
  - extended python2.6 interpreter: bin/py3to2
  - initialization script:          lib/python2.6/site-packages/py3to2_init.py
  - python3.0 bytecode compiler:    lib/python2.6/site-packages/py3to2.py

################################################################################
ABSTRACT

py3to2 is a python2.6 interpreter w/ extended python3.0 opcodes, allowing it to
natively run python3.0 & 2to3 generated scripts. it should b
mostly backwards-compatible w/ cpython2.6 & its extensions.

the intended purpose is to allow developers to migrate python2.6 scripts to
python3.0 while retaining backwards compatibility w/ existing extension modules.
py3to2 coexists w/ ur existing python2.6 installation (it consists of 3 files)

for a real-world py3to2 app (python3.0 script using 2.6 extension modules),
checkout asciiporn: http://pypi.python.org/pypi/asciiporn

MECHANISM

py3to2 has 3 components:
- py3to2
  python interpreter. can evaluate python2.6 bytecode containing additional
  python3.0 opcode instructions

- py3to2_init.py
  initialization script.  sets up import hook for recognizing python3.0 scripts

- py3to2.py
  bytecode compiler. the compile process takes 2 steps:
  - a persistent python3.0 process is created for compiling scripts into
    python3.0 code
  - py3to2.py then converts the code from python3.0 to python2.6 format

MANIFEST

./patch/ - patched files
./py3to2.diff - summary of patches (maybe out-of-date)

################################################################################
MAGIC
  simply add the MAGIC LINE:

    from __future__ import py3k_syntax

  to make py3to2 aware that a script is using python3.0 syntax

PSEUDOMETHOD
  py3to2 supports ".." syntax notation for pseudomethods
  goto: http://pypi.python.org/pypi/pseudomethod
  for more details about this feature

2TO3
  py3to2 includes convenience functions for automatically generating &
  testing scripts using 2to3:
  - class py2to3:
    - __call__ - overwrites file w/ one generated by 2to3
    - overwrite_and_load_module - overwrites file & then attempt to import it
    - test_stdlib - given a directory containing a copy of python2.6's
                    standard library, it will overwrite them using 2to3 & then
                    attempts to load each file

API: try help(py3to2)  ^_-

  py3to2 module:
  - class codetree - mutable codeobj & disassembler/assembler/debugger
  - class compiler - compiling tools
  - python3.0 wrappers:
    - py3k_compile() - compile python3.0 src
    - py3k_eval() - eval py3thon3.0 src
    - py3k_exec() - exec python3.0 src

USAGE
  start up the py3to2 interpreter by typing "py3to2" in ur shell:
    $ py3to2

    Python 2.6.py3to2 (r26:66714, Nov 18 2008, 00:56:43)
    [GCC 3.4.6 20060404 (Red Hat 3.4.6-10)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>>

  try out this simple python3.0 script:
    ################################################################
    ## PEP3132  Extended Iterable Unpacking
    ## copy this to file test_pep3132.py

    from __future__ import py3k_syntax

    a,*b = 1,2,3
    assert a == 1 and b == [2,3]
    print(a,b)
    ################################################################
    >>>
    >>> import test_pep3132
    created...
    py3k server starting...
    ...py3k server started w/...

    1 [2, 3]

  here's another python3.0 script using scipy (python2.6) extension module:
    ################################################################
    ## u must have scipy installed for this script to work
    ## copy this to file test_pep3132_scipy.py

    from __future__ import py3k_syntax

    import scipy
    a,*b = scipy.array([1,2,3])
    assert a == 1 and b == [2,3]
    print(a,b)
    ################################################################
    >>>
    >>> import test_pep3132_scipy
    1 [2, 3]

  another simple, but more thorough test script, test_py3k,
  is included w/ this distribution:
    >>>
    >>> import test_py3k
    testing PEP3102  Keyword-Only Arguments
    testing PEP3104  Access to Names in Outer Scopes
    testing PEP3105  Make print a function
    testing PEP3107  Function Annotations
    testing PEP3112  Bytes literals in Python 3000
    testing PEP3113  Removal of Tuple Parameter Unpacking
    testing PEP3114  Renaming iterator.next() to .__next__()
    testing PEP3115  Metaclasses in Python 3000
    testing PEP3120  Using UTF-8 as the default source encoding
    testing PEP3127  Integer Literal Support and Syntax
    testing PEP3129  Class Decorators
    testing PEP3131  Supporting Non-ASCII Identifiers
    testing PEP3132  Extended Iterable Unpacking
    testing PEP3135  New Super
    testing pseudomethod example 0
    testing pseudomethod example 1
    testing pseudomethod example 2
    testing pseudomethod example 3
    testing numpy example

FEATURES
  PEP3102  Keyword-Only Arguments
  PEP3104  Access to Names in Outer Scopes
  PEP3105  Make print a function
  PEP3107  Function Annotations
  PEP3111  Simple input built-in in Python 3000
  PEP3112  Bytes literals in Python 3000
  PEP3113  Removal of Tuple Parameter Unpacking
  PEP3114  Renaming iterator.next() to .__next__()
  PEP3115  Metaclasses in Python 3000
  PEP3120  Using UTF-8 as the default source encoding
  PEP3127  Integer Literal Support and Syntax
  PEP3129  Class Decorators
  PEP3131  Supporting Non-ASCII Identifiers
  PEP3132  Extended Iterable Unpacking
  PEP3135  New Super

UNICODE SUPPORT
  py3to2 will only load ascii & utf8-encoded scripts
  (utf8 is the default encoding in python3.0).

  although they're illegal in python3.0, for backwards-compatibility sake,
  py3to2 supports unicode literals for explicit <unicode> obj creation:

    u"\u1234" is an explicit <unicode> obj (note unicode literal in front)
     "\u1234" is NOT a <unicode> obj when converted back to python2.6

  note also the following r equivalent under python2.6:

    u"\u1234"  <==>  "\u1234".decode("raw_unicode_escape")

  so u MUST do either u"..." or "...".decode("raw_unicode_escape")
  to create explicit <unicode> obj in py3to2.

LIMITATIONS (FEATURES NOT FULLY SUPPORTED)
  except for the aforementioned unicode issue, from a migration standpoint,
  py3to2 is mostly feature complete in terms of python3.0's language syntax,

  language issue aside, python3.0 scripts will still behave differently b/c of
  internal differences between python2.6 & python3.0:
  - exception handling.  py3to2 implements python3.0 syntax for raising &
    catching exceptions.  but the underlying behavior is still python2.6
  - builtin functions / types.  a few of these have become different beasts
    under python3.0

################################################################################
PYTHON2.6 COMPATIBILITY TEST
  $ python setup.py dev --maketest
  ...
  test_sys
  test test_sys failed -- Traceback (most recent call last):
    File ".../Python-2.6/Lib/test/test_sys.py", line 487, in test_objecttypes
      check(get_cell().func_code, size(h + '4i8Pi2P'))
    File ".../Python-2.6/Lib/test/test_sys.py", line 423, in check_sizeof
      self.assertEqual(result, size, msg)
  AssertionError: wrong size for <type 'code'>: got 128, expected 120
  ...
  325 tests OK.
  1 test failed:
      test_sys
  35 tests skipped:
      test_aepack test_al test_applesingle test_bsddb185 test_bsddb3
      test_cd test_cl test_codecmaps_cn test_codecmaps_hk
      test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_curses
      test_gdbm test_gl test_imgfile test_kqueue test_linuxaudiodev
      test_macos test_macostools test_normalization test_ossaudiodev
      test_pep277 test_py3kwarn test_scriptpackages test_socketserver
      test_startfile test_sunaudiodev test_tcl test_timeout
      test_urllib2net test_urllibnet test_winreg test_winsound
      test_zipfile64
  2 skips unexpected on linux2:
      test_tcl test_gdbm

2TO3 COMPATIBILITY TEST
  $ python setup.py dev --py2to3test
  ...
  tested 200 2to3 generated scripts from 2.6.1.py3to2 (r261:67515, Jan  4 2009,
01:28:21)
  [GCC 4.2.3 20071123 (prerelease) (Debian 4.2.2-4)] standard library

  0 skipped:


  28 couldn't import required modules:
  BaseHTTPServer CGIHTTPServer cgi cookielib copy DocXMLRPCServer
dummy_threading HTMLParser httplib _LWPCookieJar macurl2path mimetools mimetypes
_MozillaCookieJar os pdb pickle pydoc re robotparser sgmllib SimpleHTTPServer
SimpleXMLRPCServer _strptime tempfile threading urllib2 urllib

  6 were non-utf8 compliant scripts:
  base64 getopt heapq shlex smtpd tarfile

  8 failed import due to other reasons:
  anydbm dbhash doctest imputil sets trace UserList UserString

  159 passed import:
  _abcoll abc aifc ast asynchat asyncore atexit audiodev Bastion bdb binhex
bisect calendar cgitb chunk cmd codecs codeop code collections colorsys commands
compileall ConfigParser contextlib Cookie copy_reg cProfile csv decimal difflib
dircache dis dumbdbm dummy_thread filecmp fileinput fnmatch formatter fpformat
fractions ftplib functools __future__ genericpath getpass gettext glob gzip
hashlib hmac htmlentitydefs htmllib ihooks imaplib imghdr inspect io keyword
linecache locale macpath mailbox mailcap markupbase md5 mhlib MimeWriter mimify
modulefinder multifile mutex netrc new nntplib ntpath nturl2path numbers opcode
optparse os2emxpath __phello__.foo pickletools pipes pkgutil platform plistlib
popen2 poplib posixfile posixpath pprint profile pstats pty pyclbr py_compile
pydoc_topics Queue quopri random repr rexec rfc822 rlcompleter runpy sched sha
shelve shutil site smtplib sndhdr socket SocketServer sre_compile sre_constants
sre_parse sre ssl stat statvfs StringIO stringold stringprep string struct
subprocess sunaudio sunau symbol symtable tabnanny telnetlib textwrap this
_threading_local timeit toaiff tokenize token traceback tty types unittest
urlparse UserDict user uuid uu warnings wave weakref webbrowser whichdb xdrlib
xmllib xmlrpclib zipfile

################################################################################
RECENT CHANGES:
20090103 updated to python 2.6.1
20081129 - major revision
  PyCodeObject now has kwonlyargcount attr
  - breaks one regression test but greatly simplifies patch & prevents many
    future bugs
  - fixes pydoc bug
  PyFunctionObject now has __kwdefaults__ & __annotations__ attr
  ceval.c re-patched in light of above changes (much simpler)
20081128
  more documentation
  added 2to3 convenience functions
  added unicode utf-8 support
20081123
  moved pseudomethod syntax handling to py3k server
  added more checks during setup
  added more documentation
  backported patch r67299 fixing an issue w/ super()
  cleaned up py3to2.compiler class
20081120
  fixed package importing bug - py3to2 failed to import foo.bar
20081119
  created self-installing distutils distribution
20081019
  ported to python-2.6
  consolidate & simplify patches to 1 file: ceval.c
  created extension module builtins_py3k
  revamped import hook again
  removed unicode support & restrict source code to ascii-only
20080727
  revampled import hook
20080911
  consolidate patches to 2 files: bltinmodule.c & ceval.c
20080828
  add kwonlyargcount 'attr' to codeobj
  add __annotations__ & __kwdefaults__ attr to funcobj
  add __pseudomethod__ feature to baseobj
20080819
  pure python import hook - removed magic comment & use magic path instead
  revamped str & bytes handling
  revamped py3k .pyc file handling
20080802
  pep3135  New Super
20080717
  pep3107  Function Annotations
  pep3120  Using UTF-8 as the default source encoding
  pep3131  Supporting Non-ASCII Identifiers
20080713
  import / reload works transparently on py3k scripts using a magic comment
  added pep3102  Keyword-Only Arguments
20080709 added a py3k preparser
20080702
  rewrote py3k server's pipe io.  implemented partial bytearray & bytes class.
  wrote a few simple tests
20080630
  __build_class__ function to bltmodule.c.  tested class decorators to b 
working.
################################################################################