Friday, April 28, 2017

My Python Rumspringa, or There and Back Again

tl;dr In 2014 I started an unexpected journey into a foreign land fraught with peril.  Now I'm back and have a thing or two to share about both Go and Python.

    The Journey

        Building A Home: My Python History
        A New Adventure: Go
        Coming Home
    The Spoils (What I Brought Back)
        The Good

        The Bad
        The Ugly
    Summary
        What Python Can Learn from Go
    Appendix: Raw Go Observations

Tuesday, September 6, 2016

Solving Multi-Core Python

tl;dr I proposed a solution for Python's weak multi-core story (note I didn't say "support"), but didn't have time to follow through.  I still have hope, both for the project as a whole and for several standalone parts.

[Update 8/2018] I picked this back up after a break and am aiming for Python 3.8.  See this project and PEP 554.  <fingers_crossed/>

Contents:
  Backstory
  The Proposal
  The Outcome
  The Details

Thursday, December 19, 2013

Goals for Python 3.5

Python 3.4 is in its first beta and the release is just around the corner.  Hurray!  That means we are in feature freeze.  While I am finishing up odds and ends for the PEP 451 (module specs) implementation, I'm also starting to think seriously about what I hope to accomplish for the next Python version, 3.5.

The following are lists of things I'd like to make happen for Python 3.5.  Most of them are things I wanted to do for 3.4 but ran out of time (go figure).  I'll be adding to this list as I run across things I missed.  I'm sure I've missed a thing or three that I promised myself or someone else that I'd work on.

Inspired in part by Nick Coghlan, I've categorized this 3.5 wishlist in descending order of likelihood.

[WARNING: Realistically, a bunch of this will not happen.]

Must-haves


These things are my main priorities.  Barring rejection in python-dev, I will work to push these through.

1. a C implementation of OrderedDict (basically done)
2. OrderedDict as the default class definition namespace [guido]
3. a __definition_order__ attribute on classes that preserves the order of keys (or at least the keys) [me]
4. a __kwargs_order__ attribute in function locals that preserves the order of keys from **kwargs in calls (in lieu of making kwargs an OrderedDict) [link]
5. a new type that abstracts the full import system (similar to PEP 406)
    - incl. exposing suffix->loader mapping
    - context manager support
6. add statement local namespaces [PEP 3150][PEP 403][PEP 340][PEP 359][Nick]

#1-4 are all related, though 4 probably won't involve OrderedDict when all is said and done.  They shouldn't require all that much work to take across the finish line.

Helping Out


These are things that I expect others will be targeting for 3.5 and with which I plan on helping.

* refactor pythonrun.c [PEP 432]
* refactor the C extension module API to take advantage of the PEP 451 loader APIs [import-sig link][issue16421][issue13429][issue16392?]
    - incl. implement exec_module() for BuiltinImporter and ExtensionFileLoader
* a successor to PEP 395
* "speed up" interpreter start-up, import [issue16101][link][link]
* [subinterpreters] (incl. using them to circumvent GIL? [link], interpreter-per-thread)
* a pure python zip importer [link]
* a lazy importer [link]
* missing *-unpacking generalizations [PEP 448]
* add --path0/--no-path0 cmdline options (also issue15716)
* add __locallookup__ method to metaclass [PEP 447]?
* address the hacky uses of function parameter defaults [link] (Jan K.)

Smaller Tasks


These are things that I hope to get done for 3.5, but won't require a lot of work.

* a new frame attribute, f_func, which is a weakref to the function object that was actually called (basically done)
* give the sys module a custom type and add descriptors [link]
* add inspect.Signature.from_callable()
* change setup.py (the main one) to not rely on imp.load_dynamic() [imp => _imp?] (may be done already)
* update pickle to use __spec__ (in the __main__ case)

Would be Nice


Stuff that I'd like to see in 3.5 but that I likely won't have time to work on.

* interpreter startup hook (via cmdline option) [link] (for coverage.py, etc.)
allow classes to be referenced in the class body
* extension/plugin framework for unittest a la nose [link]

May Be a Good Idea...


Other things that may or may not be worth doing.

* expose _SpecMethods (or something like it) publicly
* add importlib.util.load_from_spec()?
* .ref files [link]
* find a non-hacky replacement for a module replacing itself in sys.modules [link]
* re-organize the sys module with sub-namespaces along the lines of the config categories listed in PEP 432 (similar to but more extensive than PEP 3139)
* parser hooks
* tokenizer hooks
* traceback hooks
* API for excluding functions/methods from tracebacks like importlib and unittest do [issue1705520?]
* spec.data (a wrapper for the data-related API of loaders) [issue14982?]
* add a classonlymethod decorator (not necessarily as a builtin) [link]
* support positional-only parameters (via decorator?) [link][PEP 457]
* add a listproxy type to parallel dictproxy?
* improve re-import of builtin modules, extension modules (see above in Helping Out) [issue14715?]
* generic context API (incl. Context, ThreadLocalContext), a la decimal (email policy?) (asyncio concerns [link])
* post-import hooks (a la PEP 369)
* abc.implements(obj, *<ABC>) that ensures the object has all the appropriate attributes
* ABCMeta.implements(decorated) that registers the decorated class on an ABC and wraps the decorated class's __init__() with an abc.implements() call after it runs [issue12128]
* some mechanism to inherit docstrings [issue15582][issue15731]
* expose PyArg_ParseTuple, etc. in Python
* function to resolve __qualname__ (or provide func.__origin__)

More Radical Ideas


* always import from <module.py> and set __path__ if there is a directory by the same name (obviates __init__.py) [link] (related to PEP 402 discussions) -- at least provide an import hook
* provide a safe mode for ImportSystem (see #5 above) that prevents leaking import side effects outside a context
* replace "if __name__ == '__main__'" [link][PEP 299][PEP 3122] (__spec__.is_main + builtin is_main?)
* a reverse name binding protocol (the object being bound/unbound to the name is notified) [link]
* expose key metaclass operations via special class methods (a la PEP 422)
* attribute docstrings [link]
* a hook to process a docstring after its corresponding object is created (but before decorators are applied)
* revisit PEP 224PEP 329PEP 351PEP 363PEP 377 (Kristjan)

Random Off-the-cuff Ideas


* a decorator for type hints (optional static typing)
* a decorator that supports all sorts of *args/**kwargs handling
* define a standard API for object (modules included) to "install" themselves; provide installer helpers (a la install_importhook() or an Installer base class + ImportHookInstaller base subclass)
* generic Registry class (register() method, etc.)
* spec.open() for locatable specs
* expose an easy way to make some module-global read-only (make use of loader.create_module()?) [issue15031]
* break or return in global module code to stop early (instead of delegating skipped code to submodule)
* ...

This list is as endless as the sea! :-)